What are the best practices for 'Compacted Topics' in Kafka?

All answers to this question.

Log compaction is great but it’s not "Instant." Kafka doesn't delete old keys as they come in; it waits for a "Segment" to be closed. If your segments are too large (default is 1GB), the cleaner won't touch them for a long time. You should tune segment.ms or segment.bytes to close segments more frequently. Also, check min.cleanable.dirty.ratio. If it's set to 0.5, the cleaner won't start until 50% of the data in the log is "Dirty" (updated). Lowering this to 0.1 will make the cleaner more aggressive, but it will consume more CPU. It’s a balance between "Disk Space" and "System Resources."

   Answered 2025-09-24 by Margaret Moore

If we use compacted topics, how do we handle "Deletions"? If the bot only keeps the latest value, how do we tell it that a key should be removed entirely?

   Answered 2025-09-26 by Brian Allen

• You use what’s called a "Tombstone." You produce a message with the Key you want to delete and a "Null" payload. When the cleaner thread sees this null value, it knows to eventually remove that key and the tombstone itself from the log during the next compaction cycle. Be aware of delete.retention.ms—this setting determines how long the tombstone stays in the log. If your consumer is offline for longer than this period, it might miss the deletion entirely and keep the old data in its local state, leading to a "Data Ghost" issue.

     Commented 2025-09-28 by Kevin Scott

Compacted topics are perfect for "KTable" lookups in Kafka Streams. It makes "State Store" recovery much faster after a pod restart.

   Answered 2025-09-30 by Nancy King

• Exactly, Nancy. That’s the most common use case. Thomas, definitely look into those segment settings Margaret mentioned; they are usually the culprit for "leaky" disk space.

     Commented 2025-10-02 by Thomas Anderson