1) min.insync.replicas
default 1
When a producer sets request.required.acks to -1,
min.insync.replicas specifies the minimum number of replicas that must
acknowledge a write for the write to be considered successful.
If this minimum cannot be met, then the producer will raise
an exception (either NotEnoughReplicas or NotEnoughReplicasAfterAppend).
When used together,
min.insync.replicas and request.required.acks allow you to enforce greater
durability guarantees.
A typical scenario
would be to create a topic with a replication factor of 3, set
min.insync.replicas to 2, and produce with request.required.acks of -1.
This will ensure that
the producer raises an exception if a majority of replicas do not receive a
write.
2) Q : What is the difference between an asnc
producer and a sync producer with request.required.acks = 0 ? Is there any case where an sync producer
with request.required.acks=0 is used?
A: Async producer
just wraps sync producer and handles batching. So whether
you use async producer or sync producer, it will use the
request.required.acks only for the sync producer.
3)
Async producer = sync producer + (batchsize > 1)
Sync producer = batchsize=1
durability guarantees with request.required.acks with sync
producer
0 - sync producer never waits for an acknowledge
1 - sync producer gets an acknowledge after the leader
replica has received the data
-1 - sync producer gets an acknowledge after all ISRs receive
the data.
Along with this config parameter, min.insync.replicas can
also be configured such that an error is returned if enough replicas are not
available to replicate data.
min.insync.replicas is subset of request.required.acks
4)
If you send a list of messages by sync producer (i.e. async
producer because batchsize >1 ) , those messages will bepartitioned (randomly by default) and the collated messages
will be sent out in batches to each broker. i.e., the original batch may get
split into smaller batches - each of those batches is acknowledged. So get durability guarantees can use async producer with
required.acks set to 1 (or -1) and min.insync.replicas parameters
Persistence
To compensate for this performance divergence modern operating systems have become increasingly aggressive in their use of main memory for disk caching.
A modern OS will happily divert all free memory to disk caching with little performance penalty when the memory is reclaimed.
All disk reads and writes will go through this unified cache. This feature cannot easily be turned off without using direct I/O, so even if a process maintains an in-process cache of the data, this data will likely be duplicated in OS pagecache, effectively storing everything twice.
Furthermore this cache will stay warm even if the service is restarted, whereas the in-process cache will need to be rebuilt in memory (which for a 10GB cache may take 10 minutes) or else it will need to start with a completely cold cache (which likely means terrible initial performance).
This also greatly simplifies the code as all logic for maintaining coherency between the cache and filesystem is now in the OS, which tends to do so more efficiently and more correctly than one-off in-process attempts. If your disk usage favors linear reads then read-ahead is effectively pre-populating this cache with useful data on each disk read
Intuitively a persistent queue could be built on simple reads and appends to files as is commonly the case with logging solutions.
This structure has the advantage that all operations are O(1) and reads do not block writes or each other. This has obvious performance advantages
since the performance is completely decoupled from the data size—one server can now take full advantage of a number of cheap, low-rotational speed 1+TB SATA drives.
Though they have poor seek performance, these drives have acceptable performance for large reads and writes and come at 1/3 the price and 3x the capacity.
Having access to virtually unlimited disk space without any performance penalty means that we can provide some features not usually found in a messaging system.
For example, in Kafka, instead of attempting to deleting messages as soon as they are consumed, we can retain messages for a relative long period (say a week).
This leads to a great deal of flexibility for consumers, as we will describe.
Efficiency
We discussed disk efficiency in the previous section. Once poor disk access patterns have been eliminated, there are two common causes of inefficiency in this type of system: too many small I/O operations, and excessive byte copying.
Persistence
To compensate for this performance divergence modern operating systems have become increasingly aggressive in their use of main memory for disk caching.
A modern OS will happily divert all free memory to disk caching with little performance penalty when the memory is reclaimed.
All disk reads and writes will go through this unified cache. This feature cannot easily be turned off without using direct I/O, so even if a process maintains an in-process cache of the data, this data will likely be duplicated in OS pagecache, effectively storing everything twice.
Furthermore this cache will stay warm even if the service is restarted, whereas the in-process cache will need to be rebuilt in memory (which for a 10GB cache may take 10 minutes) or else it will need to start with a completely cold cache (which likely means terrible initial performance).
This also greatly simplifies the code as all logic for maintaining coherency between the cache and filesystem is now in the OS, which tends to do so more efficiently and more correctly than one-off in-process attempts. If your disk usage favors linear reads then read-ahead is effectively pre-populating this cache with useful data on each disk read
Intuitively a persistent queue could be built on simple reads and appends to files as is commonly the case with logging solutions.
This structure has the advantage that all operations are O(1) and reads do not block writes or each other. This has obvious performance advantages
since the performance is completely decoupled from the data size—one server can now take full advantage of a number of cheap, low-rotational speed 1+TB SATA drives.
Though they have poor seek performance, these drives have acceptable performance for large reads and writes and come at 1/3 the price and 3x the capacity.
Having access to virtually unlimited disk space without any performance penalty means that we can provide some features not usually found in a messaging system.
For example, in Kafka, instead of attempting to deleting messages as soon as they are consumed, we can retain messages for a relative long period (say a week).
This leads to a great deal of flexibility for consumers, as we will describe.
Efficiency
We discussed disk efficiency in the previous section. Once poor disk access patterns have been eliminated, there are two common causes of inefficiency in this type of system: too many small I/O operations, and excessive byte copying.
No comments:
Post a Comment