Progress
1 visited · 0 revised
Chapter
7 / 10
On this page
Daily DAG
Visualize the batch publication path
Official daily truth should feel like an orchestrated system, not a text paragraph about Spark.
Interview flow
Raw -> Trusted -> Official -> Visible
If you remember just one sequence for this section, use this one. It gives you a clean story before you zoom into DAG mechanics.
Step 1
Raw source arrives
Source
Kafka playback topics + CDC profile/title snapshots + partner metadata files
Output
Complete Bronze partitions for the batch window
Batch should start only after all expected raw inputs for that day or hour have landed.
Step 2
Clean into Silver
Source
bronze.playback_events + content/profile dimensions
Output
silver.playback_events and trusted fact tables
This step parses raw JSON, removes duplicates, fixes timestamps, joins dimensions, and creates reusable clean tables.
Step 3
Publish official Gold
Source
Trusted Silver facts + dimensions
Output
gold reporting marts such as content watch-hours and daily business metrics
Only after DQ passes do we compute the final business tables that dashboards and leadership should trust.
Step 4
Make it visible
Source
Published Gold tables
Output
BI dashboards, warehouse tables, lineage, and downstream consumers
After publish, we refresh the systems that read Gold so everyone sees one consistent version of the truth.
Daily DAG
Follow the run in order. Hover or click a node to inspect what changes there.
Selected node
Wait for source partition
Inputs
Logic
Output
Why this node matters
This is where the batch run protects itself from partial truth. Starting too early creates expensive downstream correction work.
Say it out loud
At wait for source partition, the pipeline takes bronze partition readiness + upstream completion signals and applies do not start downstream transformations on partial source data. so the next stage receives eligible batch run window.
Show how multiple sources move through the lakehouse
Instead of isolated tools, show the order: raw inputs land, Bronze stores them, Spark builds Silver, Gold publishes official truth, and serving refreshes.
Storage flow
Sources to Bronze to Silver to Gold to Serving
This is the batch storage story in order: multiple raw inputs land first, trusted tables are built in the lakehouse, official Gold is published, and only then do serving systems refresh.
Step 1
Raw inputs
AWS
MSK + S3 drops
Open source
Kafka + CDC / partner files
Why this step exists
Step 2
Bronze landing
AWS
S3 + Glue Catalog
Open source
Iceberg Bronze tables
Why this step exists
Step 3
Silver build
AWS
EMR / Glue
Open source
Spark + dbt
Why this step exists
Step 4
Gold publish
AWS
S3 + Iceberg snapshots
Open source
Iceberg / Parquet marts
Why this step exists
Step 5
Serving
AWS
Redshift / Athena
Open source
Trino / Pinot
Why this step exists
Show what must pass before publish
Batch only becomes official when partitions are ready, checks pass, and downstream refreshes are safe.
Publish checklist
No partial truth crosses this line
The release rule is simple: if any gate fails, official tables stay untouched and the team debugs the staging run instead of patching dashboards later.
Do we have complete raw inputs for the run window?
Wait for upstream Bronze partitions and completion signals before Spark starts.
Reject partial windows so downstream metrics never mix yesterday's complete data with today's incomplete arrivals.
Is the trusted output internally consistent?
Run duplicates, nulls, impossible timestamp, and reconciliation checks on Silver and pre-publish Gold snapshots.
Write a staging Iceberg snapshot first so failed validations block publication without corrupting official tables.
Can consumers see one clean official version?
Swap trusted snapshots into official tables only after checks pass.
Refresh BI, update lineage, and notify consumers so everyone reads the same published truth.
Last reviewed June 2026 · By Prasoon Parashar
Numbers are interview assumptions, not real Netflix internal figures.