How do we implement 'Exactly-Once Semantics' (EOS) in Kafka producer-consumer loops?

All answers to this question.

EOS in Kafka is a "Three-Part Harmony." First, "enable.idempotence=true" ensures the producer doesn't send duplicates if an ACK is lost. Second, you must use "Transactions" by providing a "transactional.id" and using the initTransactions, beginTransaction, and commitTransaction API calls. This ensures that a group of messages is either all written or none are. Finally, on the consumer side, you MUST set "isolation.level=read_committed." If you forget the consumer setting, your app will still see "Uncommitted" messages from failed transactions, defeating the whole purpose. It’s a powerful feature, but it does add about 10-20% latency to your pipeline.

   Answered 2025-01-17 by Elizabeth Martinez

If we use EOS, does it protect us from "Side Effects" like a microservice sending an email twice if the transaction retries?

   Answered 2025-01-19 by Steven Clark

• That is a critical limitation, Steven. Kafka EOS only guarantees that the "Data in Kafka" is consistent. It has no control over external systems like Email APIs or third-party databases. If your microservice crashes after sending an email but before committing the Kafka transaction, the retry will send that email again. To solve this, you need to make your external actions "Idempotent" or use a "Transactional Outbox" pattern where the email request is written to a Kafka topic first, ensuring the side effect is part of the same atomic stream.

     Commented 2025-01-21 by Anthony Lewis

Remember that EOS requires at least 3 brokers and a replication factor of 3 to be truly effective in a production-grade failover scenario.

   Answered 2025-01-23 by Barbara Robinson

• Spot on, Barbara. Infrastructure availability is the silent partner of EOS. Mark, make sure your cluster config matches the "High Availability" requirements Elizabeth mentioned.

     Commented 2025-01-25 by Elizabeth Martinez