YouTube data platform — Batch + Lakehouse
01
Batch Architecture
End-to-end architecture
Contract failure → Quarantine with reason, fix producer or parser, then replay from Bronze.
Certification failure → Keep the last certified snapshot and alert the accountable owner.
Storage foundation
Layer contracts
Partitioning + file layout
Certified outputs
Transaction correctness
Iceberg committed one complete, isolated, durable table snapshot.
Metric correctness
Inputs were complete, semantics passed, differences were explained, and the accountable owner approved release.
02
Daily DAG
↓
↓
↓
↓
↓
↓
↓
↓
↓
↓
Where pipeline controls apply
03
Publishing
Data Quality Checks
Row countUniquenessRequired nullsRelationshipsDistributionFreshnessBusiness reconciliationFinance approval
04
Reconciliation
↘ ↙
Delta explanation ledger
Late dimension repair
Assign UNKNOWN key→
Retain natural key→
Dimension arrives→
Reopen affected dates→
Publish corrected version
Keep three dates: event date tells when viewing happened; processing date tells when the platform handled it; publication date tells when consumers received the certified correction.
05
Interview Answer
I use batch processing to publish YouTube’s certified historical metrics for views, watch time, audience retention, revenue, payouts, and point-in-time ML datasets. Ingestion first closes the processing window and records the exact Kafka ranges, CDC snapshots, and files in a source manifest, so the Airflow or Dagster DAG cannot start from partial input. Spark then reads immutable Bronze Iceberg tables, deduplicates events, builds playback sessions, enforces consent, and applies versioned trust-and-safety decisions to create reusable Silver evidence. Incremental planning selects only changed dates and keys, while deterministic transforms, run-scoped staging, and partition replacement make every retry idempotent. Gold candidates are reconciled with matching streaming windows, tested, reviewed where required, and promoted with one atomic pointer change. Parquet on object storage provides efficient analytical scans, while Iceberg provides snapshots, time travel, schema evolution, hidden partitioning, and safe backfills. Creator Studio, BI, finance, recommendation features, and training jobs therefore read the same certified release with a reproducible explanation for every material correction.