withsoon
Data Engineering Designs
Data Engineering
YouTube
↗
Progress
Chapter 4 / 9
←
01
Start Here
02
Requirements
03
Event Sources
04
Architecture
05
Ingestion / Kafka
06
Real-Time Streaming
07
Data Modeling
08
Batch + Lakehouse
09
Governance / Quality
→
YouTube data platform — Architecture
01
Architecture Map
Producers
Player SDKs
Stable IDs
YouTube players emit playback attempts, heartbeats, engagement, impressions, and client QoE from phones, browsers, TVs, consoles, and embeds.
YouTube Edge
Server-observed delivery evidence
CDN and media-delivery systems provide server-side evidence for segment requests, cache behavior, response errors, bytes, and latency.
Services + CDC
Authoritative business state
Home, Search, Ads, Upload, Subscription, and operational databases publish authoritative decisions and reference-state changes.
→
Ingestion
Event Gateway
≥99.99% ingest
Regional collectors authenticate producers, validate contracts and consent, add trusted ingest metadata, and expose rejection reasons.
Kafka
Durable replay
Kafka separates YouTube producers from Flink, raw-lake ingestion, fraud detection, feature generation, and future consumers.
→
Processing
Flink
Seconds-to-minutes freshness
Stateful event-time jobs reconstruct playback, deduplicate bounded retries, update live counters, detect QoE regressions, and build online features.
Spark + dbt
Complete-history correction
Batch jobs use complete history for late-data closure, full deduplication, session reconstruction, invalid-traffic decisions, large joins, and backfills.
→
Lakehouse
Bronze Iceberg
Long-term evidence
Every accepted event lands unchanged in partitioned Parquet on object storage for audit, replay, backfill, and metric correction.
Silver + Gold
Certified truth
Silver standardizes and deduplicates events; Gold publishes versioned views, watch time, retention, revenue, experiment metrics, and training features.
→
Serving
Realtime Stores
Low-latency reads
Redis-like counters, Pinot/Druid segments, and the online feature store serve different low-latency YouTube access patterns.
Serving Split
Fit-for-purpose latency and access control
BigQuery-like warehouses, Pinot/Druid, offline feature views, and product APIs expose certified YouTube data to different consumers.
Orchestration
Dependency control
Airflow or Dagster coordinates YouTube certification, reconciliation, compaction, feature generation, backfills, and controlled snapshot promotion.
Governance
Trusted definitions
Governance applies YouTube schema, metric, quality, lineage, privacy, security, retention, and ownership rules across every architectural layer.
02
Platform Planes
Control plane
Defines how pipelines are allowed to run
↓ configures
01
Schema Registry
Blocks an incompatible playback.heartbeat, impression, ad, search, or QoE payload before a new Player SDK or YouTube backend release starts publishing it.
02
Metric Registry
Pins qualified_view_vN, watch_time_ms, audience retention, ad attribution, and experiment metrics to the exact reviewed YouTube semantic version used by Flink and batch.
03
Catalog + Lineage
Traces playback.heartbeat from the Player SDK through Kafka and Flink into Silver playback events, Gold video metrics, Creator Studio, and recommendation features.
04
Policy Engine
Carries analytics consent, personalization consent, kids-content restrictions, residency, and deletion rules from the YouTube event envelope into tables and feature views.
05
Airflow / Dagster
Waits for YouTube Bronze partitions and trust decisions, runs daily view/watch-time certification, checks reconciliation, and promotes the approved Gold snapshot.
06
IAM + KMS
Separates access to viewer identifiers, creator analytics, ad-revenue evidence, and public aggregates while encrypting YouTube Kafka topics, Iceberg tables, and serving stores.
07
SLO Config
Stores the YouTube targets used here: live views in 10–60 seconds, trending in 1–5 minutes, Creator P95 under 2 seconds, and official metrics within 24–48 hours.
Data plane
Moves and transforms YouTube events
Gateway
Receives Player SDK heartbeats, Home/Search impressions, ad events, QoE, and CDC; validates schema and consent, assigns ingest_time, then publishes or records an explicit rejection.
→
Kafka
Keeps playback, impression, engagement, search, ads, QoE, and CDC in separate replayable YouTube topic families consumed by Flink, Bronze sinks, fraud jobs, and feature pipelines.
→
Flink
Keys playback by playback_attempt_id, deduplicates event_id, applies event-time watermarks, sums played_delta_ms, and updates live views, trending, QoE alerts, and online recommendation features.
→
Iceberg
Stores immutable YouTube Bronze events, conformed Silver playback and business events, and certified Gold views, watch time, retention, revenue, experiment metrics, and offline features.
→
Spark
Replays complete Bronze history to include late offline viewing, apply updated invalid-traffic decisions, rebuild sessions, reconcile provisional counts, and publish certified YouTube metrics.
→
Serving
Routes official YouTube data by workload: Creator Studio slices to Pinot/Druid, internal scans to the warehouse, recommendation data to feature views, and public counts through caches and APIs.
03
Ownership Flow
Playback
Engagement
Search
Ads
Creator
QoE
Trust
Experiments
Producer owner
Player + Playback teams
ⓘ
Player + Playback teams owns emission of playback.* + qualified-view inputs: stable event IDs, source event time, consent context, release safety, retry behavior, and source-side data loss.
→
Domain contract
playback.* + qualified-view inputs
ⓘ
The Playback data domain owns schema meaning, compatibility, quality thresholds, privacy classification, event-time rules, and metric inputs.
→
Shared platform
Gateway · Kafka · Flink · Iceberg · Spark
ⓘ
The Data Platform team carries playback.* + qualified-view inputs through the gateway, Kafka, Flink, Iceberg, and Spark with replay, scaling, checkpoints, access control, and recovery. The Playback domain still owns what the data means.
→
Product owner
views · watch time · retention
ⓘ
The Playback data domain certifies its Silver and Gold products, publishes versions, meets SLOs, reconciles changes, and supplies rollback and replay procedures.
→
Consumers
Creator · Recs · Ads · Public counters
ⓘ
Creator · Recs · Ads · Public counters read views · watch time · retention through its published YouTube data contract. They declare latency and quality needs but do not independently redefine Playback metrics from raw events.
Domain team
owns meaning, quality, products, and SLOs
Platform team
owns reusable infrastructure and reliability
Governance
approves shared metrics, privacy, and financial controls
