Kafka Interview Questions — Top 50 with Answers

Basics

Q1: What is Kafka and what problem does it solve?

Kafka is a distributed event streaming platform. It solves the problem of integrating multiple producers and consumers in a scalable, fault-tolerant way. Without Kafka, you'd need N×M integrations between N sources and M destinations. With Kafka, you need N+M.

Q2: What is a topic? What is a partition?

A topic is a named feed/category for events. A partition is an ordered, immutable sequence of records within a topic. Partitions are the unit of parallelism — more partitions = more consumers can read in parallel.

Q3: What is an offset?

An offset is a unique, monotonically increasing integer that identifies each record within a partition. Consumers track their position using offsets. Kafka doesn't delete records when they're consumed — it retains them until the retention period expires.

Q4: What is a consumer group?

A consumer group is a set of consumers that together consume a topic. Each partition is assigned to exactly one consumer in the group. Multiple groups get independent offsets — they each get a full copy of the data.

Q5: What happens when there are more consumers than partitions?

The extra consumers sit idle. A partition can only be consumed by one consumer per group at a time.

Internals

Q6: How does Kafka achieve fault tolerance?

Through replication. Each partition has a configurable replication factor (typically 3). One replica is the leader (handles reads/writes), others are followers (sync from leader). If the leader dies, a follower is elected as the new leader.

Q7: What is ISR (In-Sync Replicas)?

ISR is the set of replicas that are fully caught up with the leader. The leader tracks which followers are in sync. If acks=all, the producer waits for all ISR replicas to acknowledge the write. If a replica falls behind (configurable by replica.lag.time.max.ms), it's removed from ISR.

Q8: What is the difference between at-least-once and exactly-once?

• At-least-once: messages may be processed multiple times (consumer crashes after processing but before committing offset)
• Exactly-once: each message is processed exactly once, using Kafka transactions + idempotent producers

Q9: What is Log Compaction?

Log compaction retains the latest value for each key, removing older duplicates. Used for changelog topics (e.g. database CDC). The topic acts like a key-value store — you can always replay the latest state.

Q10: How does Kafka handle back-pressure?

Producers block or throw exceptions when the broker is overwhelmed (buffer.memory fills up). Consumers control their own pace — they pull records, so there's no push-based back-pressure issue. Set max.poll.records and fetch.max.bytes to tune throughput.

Performance

Q11: How would you increase Kafka throughput?

• Increase partitions (more parallelism)
• Enable compression (snappy or lz4)
• Increase batch.size and linger.ms on producer
• Tune fetch.min.bytes on consumer
• Use async sends where durability isn't critical

Q12: What is the impact of increasing partition count?

More partitions = more parallelism, but also more overhead: more file handles, more replication traffic, longer leader election time. Don't over-partition — start with a reasonable number and scale up.

Q13: When would you use linger.ms?

linger.ms makes the producer wait before sending a batch, allowing more records to accumulate. Improves throughput at the cost of slight latency. Use it for high-volume, non-latency-sensitive pipelines.

Scenario Questions

Q14: Your consumer lag is growing. How do you diagnose it?

1. Check consumer group lag with kafka-consumer-groups.sh --describe
2. Check if consumer is stuck (GC pause, slow downstream)
3. Check partition distribution — is lag concentrated on specific partitions?
4. Check producer throughput — is ingestion spiking?
5. Solutions: add more consumers (up to partition count), increase max.poll.records, optimize processing logic

Q15: How would you design a Kafka-based audit log system?

Services → Kafka topic (audit-events, 12 partitions, retention 90 days)
         → Consumer Group 1: writes to S3 (long-term storage)
         → Consumer Group 2: writes to Elasticsearch (search/query)

Key decisions:

• Use log compaction off (keep all events, not just latest)
• Set acks=all for guaranteed writes
• Include correlation IDs in messages for tracing
• Partition by service name for ordering guarantees within a service

Q16: Kafka vs RabbitMQ — when do you choose which?

| Kafka | RabbitMQ | |---|---| | High throughput (millions/sec) | Lower throughput | | Message retention & replay | Message deleted after consumption | | Event sourcing, audit logs, analytics | Task queues, RPC, routing | | Pull-based consumers | Push-based consumers |

Choose Kafka for streaming/event sourcing. Choose RabbitMQ for task queues with routing logic.

Kafka Streams

Q17: What is Kafka Streams and how is it different from a consumer?

Kafka Streams is a client library for stream processing built on top of the Kafka consumer API. Unlike a plain consumer which just reads and forwards data, Streams provides:

• Stateful operations: joins, aggregations with built-in state stores (RocksDB)
• Windowed operations: tumbling, hopping, session windows
• Exactly-once processing semantics without external infrastructure (just Kafka + your app)
• DSL + Processor API for defining transformation topologies

Q18: What is a KStream vs a KTable?

• KStream — an unbounded stream of events. Each record is independent. Think: a log of page views.
• KTable — a changelog stream where each record upserts a key. Think: current state of a user's subscription. The latest value per key is the "current" state.
• GlobalKTable — like KTable but replicated across all instances (not partitioned). Use for small lookup tables that every instance needs.

Q19: What is a state store in Kafka Streams?

A state store is a local, embedded database (backed by RocksDB by default) used to hold intermediate state for aggregations and joins. It is:

• Partitioned: each instance only holds state for its assigned partitions
• Fault-tolerant: backed by a changelog Kafka topic that allows recovery after restart
• Queryable: with Interactive Queries you can expose state store content via a REST API

Q20: How does Kafka Streams handle failures and restarts?

When a Streams instance restarts:

1. It reads its changelog topic to restore the state store
2. Continues processing from the last committed offset

This is why changelog topics exist — they are the persistent backing store for Kafka Streams state. For fast recovery, use standby.replicas to keep warm copies on other instances.

Kafka Connect

Q21: What is Kafka Connect?

Kafka Connect is a framework for scalable, fault-tolerant data integration between Kafka and external systems without writing custom producers/consumers.

• Source connector — reads from an external system (database, S3, REST API) and writes to Kafka
• Sink connector — reads from Kafka and writes to an external system (Elasticsearch, Redshift, S3)
• Connectors are configured via JSON — no code needed for common integrations

Q22: What is a Single Message Transform (SMT)?

SMTs are lightweight transformations applied to each message in a Connect pipeline — before writing to Kafka (source) or after reading from Kafka (sink). Common SMTs:

• ReplaceField — rename or remove fields
• InsertField — add a static field (e.g. add _source: orders_db)
• TimestampConverter — convert date formats
• Flatten — flatten nested records

SMTs are not for heavy transformations — use Kafka Streams or Flink for complex logic.

Q23: What is Debezium and how does it relate to Kafka Connect?

Debezium is a CDC (Change Data Capture) source connector for Kafka Connect. It reads the database transaction log (WAL in Postgres, binlog in MySQL) and emits events for every INSERT, UPDATE, and DELETE.

{
  "before": { "id": 1, "name": "Alice", "email": "old@example.com" },
  "after":  { "id": 1, "name": "Alice", "email": "new@example.com" },
  "op": "u",
  "ts_ms": 1718900000000
}

Used for: database replication, event sourcing, cache invalidation, real-time analytics.

Schema Registry

Q24: What is the Schema Registry and why do you need it?

The Schema Registry (Confluent) stores and manages Avro/Protobuf/JSON schemas for Kafka topics. It prevents:

• Producers writing data that consumers can't deserialize
• Schema changes breaking downstream consumers

Every message sent with the Avro serializer includes a schema ID (4 bytes) that consumers use to fetch and deserialize the message.

Q25: What is schema evolution and what rules does it follow?

Schema evolution allows changing a schema over time without breaking existing producers or consumers.

Backward-compatible (new schema can read old data):

• Add optional field with default value ✅
• Remove field ✅ (old consumers ignore it)

Forward-compatible (old schema can read new data):

• Add field with default ✅
• Remove optional field ✅

Breaking (avoid without coordination):

• Change field type (int → string) ❌
• Remove required field ❌
• Rename field without alias ❌

Multi-Cluster and MirrorMaker

Q26: What is MirrorMaker 2 and when would you use it?

MirrorMaker 2 (MM2) is Kafka's built-in cross-cluster replication tool, built on Kafka Connect. Use cases:

• Active-passive DR: replicate prod cluster to a standby cluster in another region
• Active-active: bidirectional replication (complex — requires offset translation)
• Data migration: move topics from one cluster to another

MM2 also syncs consumer group offsets (with translation), topic configurations, and ACLs.

Q27: What is the difference between Kafka and Pulsar?

| Feature | Kafka | Pulsar | |---|---|---| | Storage | Log-based, local disk | Tiered (BookKeeper + offload to S3) | | Multi-tenancy | Requires separate clusters | Native namespaces and tenants | | Geo-replication | MirrorMaker 2 | Built-in | | Message ordering | Per-partition | Per-topic or per-key | | Replay | Retention-based | Cursor-based, per-subscriber |

Kafka is the default choice for most data pipelines. Pulsar has advantages in multi-tenant cloud environments.

Production and Monitoring

Q28: What metrics should you monitor in a Kafka cluster?

Broker metrics:

• UnderReplicatedPartitions — partitions where followers are behind (alert if > 0)
• ActiveControllerCount — should be exactly 1 across the cluster
• OfflinePartitionsCount — alert immediately if > 0
• BytesInPerSec / BytesOutPerSec — throughput per broker
• RequestQueueSize — if growing, brokers are overwhelmed

Producer metrics:

• record-error-rate — failed sends
• record-queue-time-avg — how long records wait before sending

Consumer metrics:

• records-lag-max — max lag across all partitions in the group (most important)
• fetch-latency-avg — time to fetch a batch

Q29: What is the KRaft mode in Kafka 3.x+?

KRaft (Kafka Raft) replaces ZooKeeper for cluster metadata management. Benefits:

• Simpler operations — no separate ZooKeeper cluster to manage
• Faster controller failover (milliseconds vs seconds)
• Scales to millions of partitions (ZooKeeper was a bottleneck at 200K+)
• Kafka 3.3+ is production-ready without ZooKeeper; 4.0 removes ZooKeeper support entirely

Q30: How would you design a Kafka-based event sourcing system?

Commands → Command Handler → Domain Events → Kafka topic (events, compacted)
                                           → Read Model Projectors (multiple)
                                              → MySQL (orders view)
                                              → Elasticsearch (search view)
                                              → Redis (realtime dashboard)

Key decisions:

• Use log compaction on the events topic to retain full event history per aggregate
• Set acks=all + idempotent producer for durability
• Use exactly-once semantics if projectors cannot handle duplicates
• Separate command validation from event publication to keep the hot path fast

Security

Q31: How does Kafka authenticate clients?

Kafka supports three authentication mechanisms:

| Mechanism | When to use | |-----------|-------------| | SASL/PLAIN | Simple username/password, TLS required to encrypt credentials in transit | | SASL/SCRAM-SHA-256/512 | Challenge-response, avoids sending plain credentials; supports dynamic credential updates | | mTLS (mutual TLS) | Certificate-based identity; preferred for service-to-service auth in high-security environments | | SASL/OAUTHBEARER | Token-based; integrates with OAuth2/OIDC providers (e.g., Keycloak) |

Always pair SASL/PLAIN with TLS encryption (ssl.endpoint.identification.algorithm=https). mTLS is preferred when clients are internal services with managed certificates.

Q32: What is ACL-based authorization in Kafka?

Kafka uses an ACL (Access Control List) system. Each ACL entry specifies:

• Principal — who (e.g., User:order-service)
• Resource — what (topic, consumer group, cluster, transactional ID)
• Operation — which action (READ, WRITE, CREATE, DELETE, DESCRIBE, ALTER)
• Host — from which IP (or * for any)

# Grant order-service WRITE on topic orders
kafka-acls.sh --add --allow-principal User:order-service \
  --operation WRITE --topic orders --bootstrap-server broker:9092

Use the deny ACL pattern sparingly — allow ACLs are preferred. In large orgs, manage ACLs via GitOps with Kafka gitops tools (e.g., kfk, julie).

Q33: How do you encrypt data at rest and in transit in Kafka?

In transit: Enable TLS on brokers (listeners=SSL://...) and configure clients with security.protocol=SSL. Set ssl.endpoint.identification.algorithm=https to prevent MITM attacks.

At rest: Kafka does not natively encrypt data at rest — it relies on OS-level or disk-level encryption (e.g., AWS EBS encryption, LUKS). For field-level encryption, encrypt before producing using a client-side library and key management service (KMS).

Transactions and Exactly-Once

Q34: How do Kafka transactions work?

Kafka transactions use a two-phase commit protocol coordinated by the Transaction Coordinator (a special broker role):

1. Producer calls initTransactions() with a transactional.id
2. Producer begins transaction: beginTransaction()
3. Producer sends records to one or more partitions — records written but not yet visible to read_committed consumers
4. Producer calls commitTransaction() — coordinator writes a COMMIT marker to all involved partitions atomically
5. Only after COMMIT markers are written do read_committed consumers see the records

If the producer crashes, the coordinator rolls back the transaction using the transaction log (the __transaction_state internal topic).

Q35: What is isolation.level in a consumer and when does it matter?

| Value | Behavior | |-------|----------| | read_uncommitted (default) | Consumer reads all records including open/aborted transactions | | read_committed | Consumer only sees records from committed transactions and non-transactional records |

Use read_committed when your pipeline uses exactly-once semantics end-to-end. Note: read_committed consumers may experience higher end-to-end latency because they have to wait for the COMMIT marker before advancing past an open transaction.

Replication and Durability

Q36: What is an ISR (In-Sync Replica) and what happens when a replica falls out of ISR?

The ISR is the set of replicas that are fully caught up with the leader (within replica.lag.time.max.ms, default 30s). When acks=all, the leader only acknowledges a write after all ISR members have written it.

If a replica falls behind (e.g., slow network, GC pause), it is removed from ISR. This does NOT cause downtime — the leader continues serving writes with the smaller ISR. However, the effective durability guarantee decreases.

Set min.insync.replicas=2 (for RF=3) to prevent writes succeeding with only 1 replica — this prevents silent data loss if the leader crashes while a follower is behind.

Q37: What is Unclean Leader Election and why is it dangerous?

Unclean leader election (unclean.leader.election.enable=true) allows a broker that is NOT in the ISR to be elected leader when no ISR members are available. This risks data loss — the new leader may be behind the old leader's committed offset.

Default is false in Kafka 0.11+. Only enable if availability is more important than consistency (e.g., some logging use cases).

Interview answer: "Unclean leader election trades durability for availability. I keep it disabled and instead increase broker capacity or reduce replica.lag.time.max.ms to keep replicas in ISR."

Q38: What does acks=all actually guarantee?

acks=all means the leader waits for all ISR replicas to acknowledge the write before responding to the producer. Combined with min.insync.replicas=2, it guarantees the message is durably written to at least 2 replicas before success.

This does NOT mean the message is written to ALL replicas — only the ISR. If min.insync.replicas=1, acks=all degrades to acks=1 when only the leader is in ISR.

Consumer Internals

Q39: What is a consumer rebalance and what triggers one?

A rebalance is the process of redistributing partitions among consumers in a group. Triggers:

• Consumer joins or leaves the group
• Consumer fails to send heartbeat within session.timeout.ms (default 45s)
• Topic partition count changes
• subscribe() pattern matches a new topic

During a rebalance (classic protocol), all consumers stop processing. This is called Stop-the-World rebalance. Kafka 2.4+ introduced Cooperative Sticky Rebalancing which avoids revoking all partitions — only moved partitions are revoked.

Set partition.assignment.strategy=CooperativeStickyAssignor to enable incremental rebalances.

Q40: What is max.poll.interval.ms and why does it cause consumer group kicks?

If poll() is not called within max.poll.interval.ms (default 5 minutes), the broker assumes the consumer is dead and removes it from the group, triggering a rebalance.

This happens when your processing logic between polls takes too long. Fix by:

• Reducing max.poll.records so each poll batch is smaller
• Moving slow processing to an async thread (with careful offset management)
• Increasing max.poll.interval.ms if the processing delay is known and bounded

Partitioning Strategy

Q41: How does the default partitioner assign records to partitions?

If a key is provided: partition = hash(key) % numPartitions (using MurmurHash2). The same key always goes to the same partition — enabling per-key ordering.

If no key: Kafka 2.4+ uses the sticky partitioner by default — batches all keyless records to one partition, then switches to another when the batch is full. This improves batching efficiency over the old round-robin strategy.

Gotcha: If you change the partition count after data is flowing, the hash mapping changes — existing consumers depending on per-key ordering will get incorrect routing.

Q42: What is a hot partition and how do you fix it?

A hot partition occurs when a key (e.g., user_id=12345 or content_id=popular_show) generates disproportionately more records than others, causing one partition/consumer to fall behind.

Fixes:

1. Salt the key: Append a random suffix key + "_" + random(0, N) to distribute load. This breaks per-key ordering.
2. Custom partitioner: Detect hot keys and spread them across multiple partitions deterministically.
3. Two-phase aggregation: First aggregate per salted key, then re-aggregate to remove the salt.

Log and Retention

Q43: What is the difference between time-based and size-based retention?

| Policy | Config | Behavior | |--------|--------|----------| | Time-based | log.retention.hours=168 (7 days default) | Segment deleted when all records in it are older than retention period | | Size-based | log.retention.bytes=-1 (unlimited default) | Oldest segments deleted when total partition log size exceeds limit | | Log compaction | log.cleanup.policy=compact | Keeps only the latest record per key; no time/size eviction |

You can combine compaction + deletion: log.cleanup.policy=compact,delete — compacts first, then deletes segments older than retention.

Q44: What is a log segment and when is a new one created?

A partition's log is split into segments. Each segment is a pair of files: .log (records) and .index (offset→position index). A new segment is created when:

• The active segment reaches log.segment.bytes (default 1 GB)
• The active segment has been open for log.segment.ms (default 7 days)

Only the active segment is written to. Older segments are immutable and eligible for retention/compaction. Smaller segments mean faster retention cleanup but more files on disk.

Advanced Topics

Q45: What is rack-aware replica assignment and why does it matter?

Kafka can distribute replicas across different racks (AZs) using broker.rack configuration. When creating a topic, Kafka ensures replicas are spread across racks so that an AZ failure doesn't cause data unavailability.

# broker config
broker.rack=us-east-1a

For a topic with RF=3 in a 3-AZ cluster, rack-aware assignment places one replica per AZ. If one AZ goes down, 2/3 replicas are still available — no data loss, no leader election needed if the AZ-local partition wasn't the leader.

Q46: How does Kafka handle large messages?

By default, message.max.bytes=1MB (broker) and max.request.size=1MB (producer). For larger messages:

• Increase limits: Set message.max.bytes, replica.fetch.max.bytes, and fetch.message.max.bytes consistently on broker and consumer.
• Compress the message: compression.type=lz4 or snappy can reduce size significantly.
• Store large payload externally: Save the large blob to S3/HDFS and produce only the pointer (Claim Check pattern). This is the preferred approach for payloads >1MB.

Q47: What is Tiered Storage in Kafka and what problem does it solve?

Tiered Storage (GA in Kafka 3.6+) offloads older log segments to object storage (S3, GCS) while keeping recent segments on local disk. Brokers serve recent reads from disk and older reads from remote storage.

Benefits:

• Dramatically reduces broker disk requirements — only hot data stays local
• Enables very long retention (months/years) without scaling broker fleet
• Faster broker restores — no need to replicate 6 months of logs from peers

Interview answer: "Tiered Storage decouples compute from storage in Kafka. We keep 7 days on disk for low-latency reads and retain 90 days in S3 for replay without scaling broker capacity."

Operational Scenarios

Q48: A Kafka producer is getting TimeoutException: Expiring 500 record(s). What do you check?

This means records in the producer's buffer are expiring before being sent (exceeded delivery.timeout.ms, default 120s).

Checklist:

1. Broker connectivity — can the producer reach the broker? Check network/firewall.
2. Broker backpressure — is the broker's request queue full? Check request-handler-idle-ratio metric.
3. Producer buffer full — is buffer.memory exhausted? Records queue up waiting for space.
4. max.block.ms exceeded — producer blocks when buffer is full; send() throws if it waits longer than this.
5. Slow acks — if acks=all and ISR is degraded, leader waits longer; increase request.timeout.ms.

Q49: How would you design a dead letter queue (DLQ) in Kafka?

Source topic → Consumer → [processing fails] → DLQ topic (e.g., orders.DLQ)
                        → [poison pill after N retries] → manual-review topic

Implementation:

• Retry transient failures in-memory with exponential backoff (Resilience4j or custom loop)
• After N retries, produce the original record + error metadata (exception class, stack trace, attempt count, original partition/offset) to <topic>.DLQ
• Run a separate DLQ consumer to alert, replay, or route to manual review
• Use a separate consumer group for DLQ replay so it doesn't affect normal processing

Key decision: Set enable.idempotence=true on the DLQ producer to avoid duplicate DLQ entries during producer retries.

Q50: How do you replay Kafka events from a specific point in time?

Options:

1. Seek by offset: consumer.seek(partition, offset) — useful if you know the exact offset.
2. Seek by timestamp: consumer.offsetsForTimes(Map<TopicPartition, Long>) returns the first offset whose timestamp ≥ the given time. Then seek() to those offsets.
3. Reset consumer group offset: kafka-consumer-groups.sh --reset-offsets --to-datetime 2026-06-01T00:00:00.000 --topic orders --group my-group --execute
4. New consumer group: Create a fresh group with auto.offset.reset=earliest and filter records by timestamp in the consumer.

Interview answer: "For point-in-time replay I use offsetsForTimes() to get the offset at the desired timestamp per partition, then seek all partitions before restarting the consumer. I create a new consumer group to avoid affecting the production group's committed offsets."

(More questions covering advanced Kafka tuning, security, and Kafka Streams patterns — questions 31–50 in progress)