The producer configurations in Kafka can play a major role in achieving certain key characteristics like message ordering, message durability, overall throughput and performance. While there are various producer configurations, we will focus on the following based on our internal testing.
Message Durability
The message durability in Kafka is controlled through the acks setting which supports three values - 0, 1 and all (-1). An ack is an acknowledgement that the producer gets from a Kafka broker to ensure that the message has been successfully committed to that broker. The config acks is the number of acknowledgements the producer needs to receive before considering a successful commit.
acks=0: You should never set acks to 0 in a production environment as the producer does not wait for a response and assumes the write is successful once the request is sent out.
acks=1: This is the default setting and the producer will wait for the ack and will consider a successful commit when it receives an ack from the leader broker but the leader doesn’t wait for confirmation from all replicas. The trade-off with this setting is to tolerate lower durability for better performance.
acks=all: This setting also makes the producer wait for the ack and in this case the leader will wait for the full set of in-sync replicas to acknowledge the message and to consider it committed. This gives the best available guarantee that the record will not be lost as long as at least one in-sync replica remains alive. The trade-off with this setting is to tolerate lower throughput/higher latency for better durability as the leader broker has to wait for acknowledgements from replicas before responding to the producer. Our recommendation is to use acks=all if you want the highest available guarantee of data.
Message Throughput
To optimize for throughput, the producers need to send as much data as they can at a faster rate. If the producers are sending several messages to the same partition in a broker, they can be sent as a batch. Generally a smaller batch leads to more requests and queuing which can result in higher latency. Increasing the batch size along with time spent waiting for the batch to fill up with messages can yield higher throughput and reduce load on producers. It also reduces the broker's CPU overhead to process each request if those were sent all the time. Following are some of the key configurations that play a vital role in achieving higher throughput.
batch.size: This allows the producer to batch the messages to the size set before sending the messages to the broker. The default batch.size is 16384. If your message size is higher than the batch.size then you should increase your batch.size to have better throughput with improved latency.
linger.ms: This allows the producer to wait until this time (in milliseconds) before sending the data to the broker. The default setting is 0 meaning no delay. The trade-off with this parameter is the added latency.
compression.type: The compression type for the data generated by the producer. The default value is none (or no compression). Valid values are none, gzip, snappy, lz4 or zstd. If compression is enabled the producer compresses the completed batch. Hence more data can be sent to the producer. Compression helps to send more payload for the same batch size which again can improve the throughput.
Use of compression comes with additional CPU cycles at the producer level and the compression ratio. gzip offers the best compression but comes with a very high CPU usage. On the other hand, lz4 uses the lowest CPU but the compression ratio is lower than every other compression type. snappy and zstd use a moderate level of CPU and provide a medium level of compression ratio compared to the other. For performance reasons we do not recommend using gzip. You can choose between lz4 and snappy based on your requirements.
To get better throughput, update both batch.size and linger.ms parameters. Increasing the batch.size without changing linger.ms from 0 might not yield the throughput you expect and at the same time might be adding more load to both producer and brokers by sending way too many smaller requests. If you want higher throughput at some added latency, we recommend that you increase batch.size and linger.ms and enable compression at the producer level.
There is no one specific setting of batch.size and linger.ms that anyone can recommend. The batching depends on various factors like the type of data, its compressibility, available CPU and network resources etc,. Hence, we recommend you to test your data to identify the right settings for these parameters to achieve higher throughput.
socket.send.buffer.bytes: The size of the socket send buffer. The default is 100KB. For higher throughput environments, this value might not be sufficient and we recommend increasing this value. You can consider increasing this to 8MB or 16MB if you have high-bandwidth networks and enough memory. If not, you can set this as low as 1 MB.
buffer.memory: This total memory (in bytes) that the producer can use to buffer records that are waiting to be sent to the broker. The default value is 33,554,432 or 32MB. If you have a lot of partitions, you might want to increase this setting.