Apache Kafka Event Streaming Architecture

A Kafka topic is divided into N partitions — each an ordered, immutable sequence of records stored on disk. Partitions enable horizontal scaling: each partition is owned by one broker and can be consumed by one consumer in a group. More partitions = more parallelism, but more overhead in ZooKeeper/KRaft metadata. A rule of thumb: target ~100MB/s throughput per partition.

Each partition has one leader and N-1 replicas. All reads and writes go through the leader; replicas pull from the leader to stay in sync. The In-Sync Replicas (ISR) list tracks which replicas are caught up. If the leader fails, Kafka elects a new leader from the ISR — typically taking under 30 seconds. Setting min.insync.replicas=2 ensures durability even if one broker fails.

Consumer groups enable parallel consumption: each partition is assigned to exactly one consumer in a group. Consumers commit their offsets (last processed message position) back to Kafka's __consumer_offsets topic. If a consumer crashes, its partitions are rebalanced to other group members. This design makes Kafka consumers stateless — any consumer can pick up where another left off.

Log compaction retains only the latest value for each message key. This transforms Kafka into a key-value store: the compacted log represents the current state of all keys. Kafka Streams and ksqlDB use compacted topics as materialized views — joining streams with state without an external database. Changelog topics (used by Kafka Streams) rely entirely on log compaction for state recovery.

Kafka achieves exactly-once delivery via two mechanisms: idempotent producers (each message gets a sequence number; brokers deduplicate retries) and transactions (atomic writes to multiple partitions). The transactional API allows consume-process-produce loops with exactly-once guarantees — critical for financial systems. Exactly-once comes with ~10% throughput overhead versus at-least-once.