Chapter 5 of 9: Ingestion / Kafka
Current section: Ingestion Route
YouTube data platform — Ingestion / Kafka 01
Ingestion Route 01 i Client SDK ->
02 i Edge Collector ->
03 i Consent Gate ->
04 i Validate ->
05 i Enrich ->
06 i Publish ->
K i Kafka
Rejected before Kafka
Invalid / unauthorized -> Reason code -> Metric or quarantine
Auth and quota failures return explicit responses; investigable schema failures enter a bounded DLQ.
Accepted once, consumed independently
Kafka -> Bronze sink Flink sessions Fraud QoE Rec features
A slow recommendation consumer does not block the raw lake sink or live playback metrics.
Producer Safety Owned by player and producer SDK i Stable event ID Batch + compress Jittered retry Offline queue Payload limit Clock-skew metadata Privacy tags Client version Collector Protection Owned by edge ingestion service i Authentication Tenant quota Rate limit Maximum payload Decompression guard Circuit breaker Reason codes
02
Topic Design Topic Key State it protects
playback.heartbeatviewer + attemptSession order i engagementvideo + shardVideo actions i impressionscoped userRecommendation sequence i ads.eventsad requestRevenue attribution i searchscoped userSearch funnel i
Kafka hot buffer: 7 days Enough for consumer recovery, offset reset, and short operational replay.
Bronze: long-term event truth Immutable accepted events support historical replay, audit, and corrected recomputation.
03
Partitioning S ⓘ Sessionization viewer + playback attempt V ⓘ Video Counters video + controlled shard U ⓘ User Sequences privacy-scoped viewer $ ⓘ Ad Attribution request or impression
Start from the state that must stay ordered; choose a different key for each computation.
04
Capacity Event Rate peak events/s / safe events/partitionByte Rate peak bytes/s / safe bytes/partitionParallelism required consumer tasksFailure Spread minimum rack / zone spread
Playback heartbeat example
6.94M peak HB/s/ 35K safe HB/s/Flink task= 199 x 1.4 headroom = 279 provision about 280
The event-rate constraint needs 195 partitions and the byte-rate constraint needs 138; consumer parallelism is larger at 279, so it wins. Every safe-throughput number must come from a representative load test.
05
Failure Drill 01 ⓘ Viral Spike Video v-7 surges 02 ⓘ Hot Partition One task saturates 03 ⓘ Detect Skew Compare partitions 04 ⓘ Shard Key v-7 + shard 00..63 05 ⓘ Merge 64 partials -> v-7
Detect the hot video from per-partition skew, spread it across controlled shards, aggregate each shard, then merge the small partials by video.
06
Delivery + Operations At-least-once i Retry instead of losing events Event Dedup i Count one logical event once Revenue Safety i Prevent duplicate ad billing Hot Retention i Keep seven days for replay Lag + Rebalance i Find consumers falling behind Poison Isolation i Skip permanently bad records safely Quota Isolation i Protect critical YouTube traffic Recovery Test i Prove failures are recoverable
