Apache Kafka — Distributed Messaging System

Kafka is an open-source Distributed Message Storage platform.
Producers append records to topics; consumers read them in order.
Unlike a simple message queue that deletes messages When 1 reader reads the message, Kafka stores Topics(msg1,msg2..) on disk for a configurable time or size — so that consumer can replay a history.
Decouples services at very high throughput, ie Fast producer, slow consumer can be tied to the system.

Requirements

Non Functional

High throughput: Ingest millions of events/sec from many producers without a single bottleneck.
Scalability: Add brokers and partitions horizontally as traffic grows.
Fault tolerance: Survive broker/disk/network failures without losing committed data.
High availability: Keep serving reads/writes when nodes fail (leader election, replicas).
Durability Persist messages to disk; survive process restarts and crashes.
Ordering Preserve order within a partition (global order only if single partition).
Decoupling Producers and consumers evolve independently; no direct coupling.
Replay New or slow consumers can re-read history from a stored log.
Multi-subscriber Many consumer groups read the same topic independently.
Backpressure & retention Buffer spikes; drop or compact old data by policy, not by accident.

Where Kafka Use Cases

Good Fit

Event streaming / log aggregation — app logs, metrics, clickstreams into data lakes.
Microservice decoupling — order placed, payment captured, inventory updated as events.
Change data capture (CDC) — database changes streamed to search indexes or warehouses.
Real-time analytics — Flink/Spark/Kafka Streams processing pipelines.
Replay & reprocessing — rebuild a downstream system from the log after a bug fix.
High-volume pub/sub — many services subscribe to the same topic with separate consumer groups.

Poor fit:

Low-latency RPC — use gRPC/HTTP directly; Kafka adds ms–s latency and batching trade-offs.
Task queues with per-message ack delete — SQS/RabbitMQ patterns; Kafka is a log, not a mailbox.
Small messages, few TPS, simple setup — operational overhead of a cluster may be overkill.
Strict global ordering — only one partition gives global order; multi-partition = per-key or partition order.

Architecture

Cluster of brokers, topics split into partitions, and a special __consumer_offsets topic
Only 1 Topic can be stored in 1 Partition & A topic is spread across many partitions.
Reliable:
  Records/messages(immutable) are appended to a disk and are strictly ordered. Each record gets a monotonic offset (0, 1, 2, …) unique within that partition.
  Consumers track committed offsets so restart continues from last safe position.
  To locate a record you need topic + partition + offset.
Scalable: More nodes/paritions/disks can be added. Producers can batch and compress record
Fault tolerant: Each partition is replicated across brokers (replication factor rf). If a leader fails, a follower from the ISR is elected. Cluster continues serving as long as leaders exist for partitions
Each partition has one leader broker and zero or more follower replicas on other brokers.

struct Message/Payload {
  name
  age
  empId
}
┬────────────────────────────────────────  Kafka Cluser  ──────────────────────────────────────────────────────────┐
|                                                                                                                  |
|   ┬───────────────────────── BROKER-2(master) ────────────────────────────────────┐   ┬─ Broker1(master)──┐      |
|   | ┬────────────────── Partition-1   ────────────────────┐                       |   |                   |      |
|   | | ┬───────────── topic = Employee ──────────────┐     |                       |   ┴───────────────────┴      |
|   | | | msg1->msg2->msg5    msg6->msg9->msg7        |     |                       |                              |
|   | | ┴─────────────────────────────────────────────┴     |                       |                              |
|   | ┴─────────────────────────────────────────────────────┴                       |                              |
|   |                                                                               |                              |
|   |                       ┬────────────────── Partition-2 ────────────────────┐   |                              |
|   |                       | ┬───────────── topic = Orders ─────────────────┐  |   |                              |
|   |                       | | m1 m2 m3 ....                                |  |   |                              |
|   |                       | ┴──────────────────────────────────────────────┴  |   |                              |
|   |                       ┴───────────────────────────────────────────────────┴   |                              |
|   |                                                                               |                              |
|   |              ┬───────────────────────── Partition-n ────────────────────────┐ |                              |
|   |              | ┬───────────────topic = __consumer_offset ────────────────┐  | |                              |
|   |              | |  Consumer_Name  ID    Topic   Committed_Offset  State   |  | |                              |
|   |              | |  Consumer-1     uid1  Orders  Read till ms7     Alive   |  | |                              |
|   |              | |  Consumer-2     uid1  Orders  Read till ms7     Alive   |  | |                              |
|   |              | ┴─────────────────────────────────────────────────────────┴  | |                              |
|   |              ┴──────────────────────────────────────────────────────────────┴ |                              |
|   ┴───────────────────────────────────────────────────────────────────────────────┴                              |
|                                                                                                                  |
┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴
          

Read & Write paths

Write path: Producers Always write to the partition leader. Followers pull from leader via replication. Read Path: Consumers Always fetch from the partition leader. Metadata: clients bootstrap from any broker (bootstrap.servers), then use the cluster metadata API to find partition leaders (not by calling ZooKeeper directly in modern clients). Leader, follower, and ISR   For each partition replica set, one broker is the leader — all producer writes and consumer reads go to the leader (by default). Followers replicate the log tail from the leader.   ISR (In-Sync Replicas) is the set of followers caught up within a configurable lag. Only ISR members are eligible to become leader on failover. A leader is always in its own ISR.

Core concepts

Records, messages, and offsets

A record (message) is typically a (key, value, timestamp, headers) byte sequence. Records in a partition are immutable and strictly ordered. Each record gets a monotonic offset (0, 1, 2, …) unique within that partition. To locate a record you need topic + partition + offset. The key is optional; same key → same partition (hash) preserves order per key.

Topics and partitions

A topic is a named stream of records (e.g. orders, payments). Topics are split into partitions for scale. Rule: {1 topic → N partitions}. 1 partiton stores 1 topic

Brokers and cluster

A broker is a Kafka server (default port 9092) that stores partitions on local disk. A cluster is multiple brokers. Kafka is multi-master at the cluster level — any broker can host leaders for different partitions.
Bootstrap broker(s): first address clients connect to; broker returns full metadata (all brokers, topics, partition leaders).

Consumer groups

A consumer group is set of consumers that work together to read topics. 1 consumer is given resonsiblity to read 1 or more topics to partitions.
This helps to avoid all consumers reading same message and hence to scale (up to partition count).

Topic(Orders)
Partition 0(3 messages): A B C
Partition 1(3 messages): D E F
Partition 2(3 messages): G H I

Consumer group(OrderProcessors)
	Consumer 1 reads +--> Partition 0, Partition 2
	Consumer 2 reads +--> Partition 1
Note Consumer 1 never reads Partition-1
        

Offset progress is stored in the internal topic __consumer_offsets (committed offset per group, topic, partition). If a consumer dies, its partitions are rebalanced to surviving members.

High-water mark (HWM)

When consumer/subscriber reads from Leader. Leader never returns messages which have not been replicated to a minimum set of ISRs.
Leader keeps track of highest water mark. Highest watermark is the offset for that partition replication.
Example(in figure below): Leader does not return messages greater than offset=4, as it is the highest offset message that has been replicated to all followers.

Workflow — one message publish and consume

sequenceDiagram
    autonumber
    participant P as Producer
    participant B as Any Broker
(bootstrap)
    participant L as Partition Leader
Broker 2
    participant F as Follower
Broker 1
    participant C as Consumer
Group A
    participant O as __consumer_offsets

    P->>B: Metadata request — where is topic orders partition 1?
    B-->>P: Leader = Broker 2

    P->>L: Produce record (key=orderId, value=JSON)
    L->>L: Append to local log segment on disk
    L->>F: Replicate to follower ISR
    F-->>L: ACK replication
    L-->>P: ACK (acks=all)

    C->>L: Fetch records from offset 42
    L-->>C: Batch of records (up to HWM)
    C->>C: Process business logic
    C->>O: Commit offset 43
            

Role of ZooKeeper (and KRaft)

Classic Kafka (with ZooKeeper): ZooKeeper stores cluster metadata — broker registration, topic configuration, partition leader election, controller identity, and ACLs (in older setups). Brokers and the controller watch ZK for changes. Clients do not query ZooKeeper for leaders; they use the Kafka metadata protocol via brokers.

Modern Kafka (KRaft mode, ZooKeeper-less): metadata is stored in an internal Raft quorum (__cluster_metadata). New deployments should plan for KRaft; ZK is deprecated in Kafka 3.x+ and removed in Kafka 4.x. Conceptually the controller still tracks leaders, ISR, and broker membership — the storage layer changed.

Storing messages on disk, log retention, and client quotas

Log segments on disk

Each partition is a directory of segment files (e.g. 00000000000000000000.log). Appends are sequential — fast on HDD/SSD. An index maps offset → file position for fast lookup. Segments are rolled by size (segment.bytes) or time. This design is why Kafka handles very large volumes: it behaves like a distributed commit log, not an in-memory queue.

Log retention

Old data is removed by policy — not when a consumer reads it:
  retention.ms — delete segments older than N milliseconds (default 7 days).
  retention.bytes — cap total size per partition.
Compacted topics (cleanup.policy=compact) — keep latest record per key (changelog topics, Kafka Streams state). Tune retention for compliance (keep 90 days) vs cost (short TTL for high-volume telemetry).

Client quotas

Quotas throttle misbehaving clients so one tenant cannot saturate the cluster. Kafka supports producer/consumer byte-rate limits per client ID or user/principal (quota.producer.default, quota.consumer.default). Exceeding quota returns throttle time in responses. Use quotas in multi-tenant platforms and shared clusters.

Message delivery semantics

Common data issues and how to resolve them

1. Message loss.
Cause: acks=0/1, unclean leader election, producer not retrying. Fix: acks=all, min.insync.replicas=2, disable unclean.leader.election.enable, enable producer retries and idempotence.

2. Duplicate messages.
Cause: at-least-once retries, consumer crash after process but before commit (or vice versa). Fix: idempotent consumers (store processed IDs), idempotent producer, transactional writes, or dedupe in downstream store.

3. Out-of-order delivery.
Cause: multiple partitions, retries, different keys routed to different partitions. Fix: use same key for events that must stay ordered; single partition if global order required; partition count fixed at create time (or use rekeying carefully).

4. Consumer lag growing.
Cause: slow processing, too few consumers, hot partition skew. Fix: scale consumer group (≤ partition count), optimize handler, increase partitions and rebalance keys, monitor records-lag-max.

5. Hot partition / skew.
Cause: poor key choice (null key or constant key). Fix: choose high-cardinality keys; custom partitioner; salting for extreme hotspots.

6. Disk full.
Cause: retention too long, no size cap, replication multiplies storage. Fix: lower retention.ms/bytes, add disks, tiered storage (Kafka 3.x+), compression (compression.type=lz4/zstd).

7. Rebalance storms.
Cause: consumers join/leave frequently, long max.poll.interval.ms exceeded during heavy processing. Fix: static group membership, cooperative rebalancer, increase interval, decouple processing from poll loop.

8. Under-replicated partitions.
Cause: broker down, network issues, disk slow on follower. Fix: restore broker, replace failed disk, rack awareness, alert on ISR shrink; avoid rolling restart of too many brokers at once.