YouTube data platform — Ingestion / Kafka
01
Ingestion Route
->
->
->
->
->
->
Rejected before Kafka
Invalid / unauthorized->Reason code->Metric or quarantine
Auth and quota failures return explicit responses; investigable schema failures enter a bounded DLQ.
Accepted once, consumed independently
Kafka->Bronze sinkFlink sessionsFraudQoERec features
A slow recommendation consumer does not block the raw lake sink or live playback metrics.
02
Topic Design
TopicKeyState it protects
Kafka hot buffer: 7 days
Enough for consumer recovery, offset reset, and short operational replay.
Bronze: long-term event truth
Immutable accepted events support historical replay, audit, and corrected recomputation.
03
Partitioning
Start from the state that must stay ordered; choose a different key for each computation.
04
Capacity
Playback heartbeat example
6.94M peak HB/s/35K safe HB/s/Flink task= 199x 1.4 headroom = 279provision about 280The event-rate constraint needs 195 partitions and the byte-rate constraint needs 138; consumer parallelism is larger at 279, so it wins. Every safe-throughput number must come from a representative load test.
05
Failure Drill
Detect the hot video from per-partition skew, spread it across controlled shards, aggregate each shard, then merge the small partials by video.
06