YouTube Data Engineering — Batch + Lakehouse

Two-minute interview path

Publish one reproducible certified history

Lead with this

Lead with the immutable evidence and atomic release path. Then explain incremental recomputation, data-quality gates, and why batch is allowed to correct the faster streaming estimate.

Close inputsRecord Kafka ranges, CDC snapshots, files, schemas, and cutoff time.
Build evidenceDeduplicate and sessionize Bronze into reusable Silver facts.
RecomputeProcess only changed dates and keys with idempotent deterministic jobs.
ReconcileExplain stream-versus-batch differences at the same grain and version.
PromotePublish one tested Gold snapshot atomically and retain rollback history.

01

Batch Architecture

Core argument:Immutable Bronze evidence flows through deterministic Silver reconstruction into candidate Gold products; Iceberg snapshots make every input and output reproducible.

Explore the batch architectureEight-stage flow, storage foundation, layer contracts, partitioning, and outputs.Explore

End-to-end architecture

Contract failure → Quarantine with reason, fix producer or parser, then replay from Bronze.

Certification failure → Keep the last certified snapshot and alert the accountable owner.

Storage foundation

Layer contracts

Partitioning + file layout

Certified outputs

Transaction correctness

Iceberg committed one complete, isolated, durable table snapshot.

Metric correctness

Inputs were complete, semantics passed, differences were explained, and the accountable owner approved release.

Explore architecture concept

Events + CDCPlayback, ads, engagement, metadata

Kafka carries high-volume behavioral events while CDC and governed files carry video, channel, rights, policy, campaign, and monetization changes. The ingestion layer closes each processing window with the exact Kafka offsets and source snapshots that landed.

A closed, reproducible input inventory containing event ranges, CDC snapshot IDs, governed files, schema versions, and owning producer teams. Example: A playback heartbeat and a later channel-monetization change arrive through different producers, but both receive immutable source coordinates before batch processing starts.

02

Daily DAG

Core argument:The orchestrator waits for a complete manifest, plans changed partitions, runs deterministic transforms, validates the candidate, and only then permits publication.

Explore the daily DAGFour phases, individual tasks, dependencies, idempotency, and pipeline controls.Explore

↓

→

↓

→

↓

→

↓

Where pipeline controls apply

Explore DAG stage

Ready Phase

This is an orchestration boundary, not a data transformation. Ingestion publishes a small manifest only after it has closed the window and recorded expected Kafka ranges, CDC snapshots, and files. The DAG waits for that signal, then proves every listed input is readable.

A verified closed window whose expected Kafka offsets, CDC snapshots, and governed files are complete and contract-compatible. Example: The daily job does not calculate yesterday’s views while one mobile-event partition or ad settlement file is still missing.

03

Publishing

Core argument:Write into run-scoped staging, test the candidate, record the release manifest, and switch one atomic pointer so consumers never observe a partial table.

Explore the publish protocolStaging, quality checks, approval, atomic promotion, rollback, and consumer refresh.Explore

Data Quality Checks

Row countUniquenessRequired nullsRelationshipsDistributionFreshnessBusiness reconciliationFinance approval

04

Reconciliation

Core argument:Compare stream and batch only at the same key, window, timezone, and metric version; attribute every material delta before replacing provisional history.

Explore reconciliationAligned inputs, delta ledger, decision gates, corrections, and late dimension repair.Explore

↘ ↙

Delta explanation ledger

Late dimension repair

Assign UNKNOWN key→

Retain natural key→

Dimension arrives→

Reopen affected dates→

Publish corrected version

Keep three dates: event date tells when viewing happened; processing date tells when the platform handled it; publication date tells when consumers received the certified correction.

05

Interview Answer

Core argument:Close inputs, rebuild trusted evidence incrementally, reconcile the candidate, and atomically promote one certified snapshot for every consumer.

Open the complete interview answerThe full batch and lakehouse answer to rehearse.Explore

I use batch processing to publish YouTube’s certified historical metrics for views, watch time, audience retention, revenue, payouts, and point-in-time ML datasets. Ingestion first closes the processing window and records the exact Kafka ranges, CDC snapshots, and files in a source manifest, so the Airflow or Dagster DAG cannot start from partial input. Spark then reads immutable Bronze Iceberg tables, deduplicates events, builds playback sessions, enforces consent, and applies versioned trust-and-safety decisions to create reusable Silver evidence. Incremental planning selects only changed dates and keys, while deterministic transforms, run-scoped staging, and partition replacement make every retry idempotent. Gold candidates are reconciled with matching streaming windows, tested, reviewed where required, and promoted with one atomic pointer change. Parquet on object storage provides efficient analytical scans, while Iceberg provides snapshots, time travel, schema evolution, hidden partitioning, and safe backfills. Creator Studio, BI, finance, recommendation features, and training jobs therefore read the same certified release with a reproducible explanation for every material correction.

YouTube data platform — Batch + Lakehouse

Publish one reproducible certified history

Batch Architecture

End-to-end architecture

Storage foundation

Layer contracts

Partitioning + file layout

Certified outputs

Daily DAG

Where pipeline controls apply

Publishing

Data Quality Checks

Reconciliation

Delta explanation ledger

Late dimension repair

Interview Answer

Loading your data design lesson

YouTube data platform — Batch + Lakehouse

Publish one reproducible certified history

Batch Architecture

End-to-end architecture

Storage foundation

Layer contracts

Partitioning + file layout

Certified outputs

Daily DAG

Where pipeline controls apply

Publishing

Data Quality Checks

Reconciliation

Delta explanation ledger

Late dimension repair

Interview Answer