Progress
1 visited · 0 revised
Chapter
8 / 10
On this page
Trust flow
What blocks publish
Gold should publish only after explicit trust gates pass. Strong interview answers make those gates concrete.
How bad data recovers
Quarantine, replay, and backfill must look like normal operating paths rather than heroic cleanup work.
Who owns the failure
Every contract, DQ breach, and privacy workflow needs a named owner plus an escalation path.
Step 1
Contract
Schema, owner, PII tags, and compatibility rules are defined before an event family becomes real platform input.
Step 2
Validate
CI, registry checks, and pipeline validation stop malformed or semantically broken data before it fans out.
Step 3
Certify
Silver and Gold datasets become official only when reconciliations pass and an owner stands behind the table.
Step 4
Publish
Consumers only see one clean trusted version, with rollback, lineage, and audit signals preserved.
Schema evolution policy
If playback producers add optional fields like `player_build`, `network_type`, or `cdn_pop`, keep the event backward compatible so existing Flink jobs and Silver tables keep reading safely.
If a field changes meaning or type, such as watch-time semantics or QoE status enums, cut a new schema version and migrate Flink, Spark, dbt, and certified marts in a controlled window.
Before the producer deploys, CI should validate sample playback payloads against schema registry rules so broken contracts never silently enter Bronze.
If a bad version still escapes, freeze official publish and quarantine that versioned slice instead of letting corrupted playback facts roll into Gold KPIs.
Certified dataset rules
Certified Gold marts are the default source for dashboards, finance, and executive KPI reviews.
Silver is trusted engineering truth, but not automatically an official stakeholder-facing dataset.
Every certified table should expose owner, freshness SLA, lineage, last successful publish, and rollback version.
Interview answer
Additive schema changes are the safe default: new optional fields, new enum values behind compatibility rules, and new downstream consumers that do not change existing field meaning.
Blocks publish
Hard block
Missing source partitions, schema incompatibility, failed referential integrity, or reconciliation mismatch stops official publish.
Warns owner
Conditional alert
A non-critical dimension lag or metadata gap can alert owners without blocking Gold if business metrics remain trustworthy.
Consumer view
Release behavior
Failed runs stay in staging snapshots. Consumers keep reading the last clean official version until trust is restored.
Selected signal
Duplicate rate
Spike means dedupe or producer retry issue
Why it matters
The answer should connect this signal to business risk, owner, and release behavior, not just quote the number.
First owner
Usually starts with producer retries, event_id behavior, or idempotency drift.
First things to check
Check producer retries and idempotency keys first.
Check whether event_id dedup rules changed in stream or batch.
Check Bronze raw counts against Silver deduped counts.
Confirm duplicate spike is not isolated to one source partition or client version.
Reconciliation checks
Source -> Bronze: expected partitions and raw counts arrived for the publish window.
Bronze -> Silver: dedup, null-rate, enum, and timestamp sanity checks stay inside tolerance.
Silver -> Gold: business totals reconcile against trusted facts before official publish.
Fact -> dimension: unknown keys stay below threshold or route to repair before publish.
Metric versioning: changed formulas reconcile against prior definitions before replacing certified KPIs.
When to use this
Quarantine and replay after trust fails in staging
Use this path when Bronze has already landed and a Silver or pre-publish Gold build fails contract, DQ, or reconciliation checks before official publish.
Instead of letting the bad run leak into dashboards, keep consumers on the last certified version, isolate the broken partitions or records, fix the transform or schema issue, and replay only the affected slice.
Contract or DQ rule fails
Write record to quarantine / DLQ with reason code and owner
Notify producer or pipeline owner with sample payload and blast radius
Fix source schema, mapping, join logic, or threshold configuration
Replay quarantined records into Bronze or Silver repair flow
Re-run reconciliation, then publish only after validation turns green
Interview answer
A hard blocker means trust is broken, such as missing partitions, failed reconciliation, or referential-integrity failure. A warning-only breach means the issue is visible and owned, but the published business truth is still trustworthy.
Failure scenarios
Incident walkthrough
Gold content metrics wrong for last 30 days
What you see first
The first signal is usually a reconciliation miss between the daily Gold mart and an independently computed finance or stream-side control total. Product or finance may report that a title's watch-hours, completion rate, or revenue attribution has been off for weeks.
Why it matters
This is not just a dashboard glitch. Planning, content-performance reporting, and any executive review using that Gold mart are now reading the wrong official business number.
What I do immediately
Freeze new Gold publishes for that subject area, pin downstream consumers to the last certified snapshot, and narrow the blast radius by finding the exact bad date range, metric family, and upstream transform that introduced the drift.
How I restore trust
Fix the business logic or join error, recompute the affected Silver partitions, backfill only the impacted Gold windows, rerun reconciliation against control totals, and publish one audited replacement snapshot instead of patching tables manually.
How I stop repeat incidents
Keep metric definitions versioned, require pre-publish reconciliation against control datasets, store reproducible Iceberg snapshots, and make every Gold publish reversible so long-lived drift cannot stay hidden.
What I would actually say
If Gold is wrong for 30 days, I would not hot-fix dashboards. I would freeze publication, isolate the bad date range, rebuild from trusted Silver or Bronze, rerun reconciliation, and then republish one certified snapshot with an audit note.
Privacy ops
Protect sensitive fields in the analytics path
Sensitive fields
Classification
Direct PII
Interview framing
Explain where this field can exist in raw form, where it must be masked or tokenized, and why downstream Gold marts should not need the raw identifier at all.
Never expose raw email in analytics marts
Hash or tokenize before warehouse storage
Restrict raw access to a minimal approved set of services
Ownership and access model
Producer owner
Owns source schema, enums, payload correctness, and bad-event fixes at the source.
Pipeline owner
Owns Silver / Gold logic, DQ thresholds, replay correctness, and publish gates.
Data product owner
Owns certified metric semantics, stakeholder communication, and signoff for business-facing tables.
Privacy / governance owner
Owns access policy, delete propagation, audit evidence, and sensitive-field exceptions.
Privacy boundary
Raw identity fields may exist in tightly controlled Bronze or trusted engineering layers, but published Gold dashboards should read masked, tokenized, or aggregated derivatives.
If a dashboard or analyst asks for raw email, device ID, or billing details directly, that usually means the data product boundary is wrong and needs a narrower governed access path.
The key interview point is the boundary itself: raw identity for controlled operational workflows, identity-safe outputs for broad analytics consumption.
Always include
Encrypt raw playback, account, and billing-linked datasets across Kafka, S3, Iceberg, and warehouse copies so sensitive activity is protected in every hop.
Use RBAC plus column-level controls so product analysts can read content and engagement metrics without directly seeing raw member, device, or billing identifiers.
Apply stricter retention to raw member-linked events than to aggregated Gold metrics so privacy policy matches the sensitivity of each layer.
Keep audit trails for who accessed restricted tables and who approved replay or correction workflows after data-quality incidents.
Enforce regional residency when member-linked data or billing fields cannot legally move across processing regions or exports.
Mark which published marts are safe for broad BI use and which trusted tables remain engineering-only so consumers do not bypass governance boundaries.
Interview answer
You keep an auditable completion record showing request scope, resolved subject keys, affected datasets, completion timestamp, and the downstream systems that were rebuilt, purged, or invalidated.
Last reviewed June 2026 · By Prasoon Parashar
Numbers are interview assumptions, not real Netflix internal figures.