Ingestion / Kafka
Cover ingestion lanes, topic design, partitioning, schema validation, DLQ, replay windows, and first-stage consumers.
On this page
Ingestion lanes
Connect sources, topics, keys, retention, schemas, and fan-out clearly.
Kafka is the durable event backbone here because it decouples producers from many consumers while giving us replay, retention, ordered partitions, and clear schema contracts.
Quick jumps
Click to jumpMonthly users
200M-250M
Docs use both 200M+ and 250M assumptions
Daily active users
80M
Used for heartbeat and Kafka math
Peak concurrency
15M
Useful for QoE and live ops workloads
Separate client, CDC, and external batch clearly
The three-way ingest split should be obvious without reading long paragraphs.
Selected lane
Client Events
Mobile / Web / TV
Event Gateway
Schema validation
Kafka playback/browse/search topics
Clients batch events every few seconds instead of sending a network request for every click or heartbeat.
The gateway validates schema, authenticates source, adds ingestion_time, app/device/region metadata, and privacy tags.
High-volume events like video.heartbeat and browse.title_impression need separate topic design from low-volume auth or billing events.
Connect event families to Kafka topics and keys
Keep retention, schema, and partition choices visible and interview-ready.
Kafka topic explorer
playback.heartbeat.raw
Retention
7 days
Key
user_id / profile_id / session_id
Format
Avro or Protobuf + Schema Registry
Producer
Event Gateway
Consumers
- Flink sessionizer
- Watch-time aggregator
- QoE monitor
- Lake ingestion
Risk
Interactive partition calculator
required partitions = ceil(peak / per_partition x headroom)
Required partitions
260
Ceiling with headroom included
Show what happens after publish
A single topic is not the destination; it fans out to raw sinks, stream jobs, DLQ, and feature paths.
Explain why Kafka sits at the center
Use a concise justification that covers replay, decoupling, and many consumers.
Say This In Interview
Kafka is the durable event backbone here because it decouples producers from many consumers while giving us replay, retention, ordered partitions, and clear schema contracts.
30-second answer
FastKafka is the durable event backbone here because it decouples producers from many consumers while giving us replay, retention, ordered partitions, and clear schema contracts.
2-minute answer
BalancedCover ingestion lanes, topic design, partitioning, schema validation, DLQ, replay windows, and first-stage consumers. Focus on PIPELINES, 9 min, Connect sources, topics, keys, retention, schemas, and fan-out clearly..
Deep answer
DeepKafka is the durable event backbone here because it decouples producers from many consumers while giving us replay, retention, ordered partitions, and clear schema contracts. Then go layer by layer through PIPELINES, 9 min, Connect sources, topics, keys, retention, schemas, and fan-out clearly., including trade-offs, failure handling, and why this shape is right for the consumer SLAs.
Last reviewed June 2026 · By Prasoon Parashar
Numbers are interview assumptions, not real Netflix internal figures.