withsoon
withsoon/System Design/Netflix Data EngineeringPIPELINES

Architecture

Walk from event emitters to validation, Kafka, streaming, Bronze/Silver/Gold, warehouse, features, replay, and governance.

View Backend Track

Chapter

4 / 17

On this page

Layered map

PIPELINESChapter 410 min

Show the full Netflix data journey as one layered system map.

My architecture is layered: emit, validate, publish to Kafka, process in streaming and batch, store in Bronze/Silver/Gold, and serve BI plus features with governance around every layer.

PIPELINES10 minShow the full Netflix data journey as one layered system map.

Quick jumps

Click to jump

Monthly users

200M-250M

Docs use both 200M+ and 250M assumptions

Daily active users

80M

Used for heartbeat and Kafka math

Peak concurrency

15M

Useful for QoE and live ops workloads

Architecture canvas

Interactive layered architecture

Click any node to see its inputs, outputs, failure modes, and interview framing. Highlight paths with the overlay buttons.

Interactive architecture canvas

Click nodes to inspect. Use overlay buttons to trace replay, governance, and cost paths.

Node detail drawer

Kafka / MSK

What it does

Durable, replayable event buffer between producers and all stream and batch consumers.

Why it exists

Decouples ingest from Flink, lake ingestion, fraud detection, QoE, and feature generation.

Connected path

Client Apps β†’ Event Gateway β†’ Kafka / MSK β†’ Flink Streaming β†’ Bronze Lake β†’ BI + ML Consumers

Quick jumps

Click to select a node

Legend

StreamingBatchStorageServingGovernance
Event journey

Emit β†’ validate β†’ publish β†’ stream β†’ store β†’ aggregate β†’ serve

Walk through this sequence aloud before naming specific tools.

Emit→
Validate→
Publish→
Stream→
Store Bronze→
Clean Silver→
Aggregate Gold→
Serve
Decision cards

Defend the major architectural choices

Know why each design decision was made β€” interviewers probe these directly.

Decision

Kafka vs Kinesis

Kafka when you want rich ecosystem, strong replay patterns, and open-source portability.

Decision

Flink vs Spark Streaming

Flink for complex low-latency event-time stateful logic like sessionization and watch-time.

Decision

Spark batch vs dbt

Use Spark for heavy cleansing and big transforms; use dbt for SQL-first Silver-to-Gold modeling where warehouse/lakehouse engines are strong.

Decision

Iceberg vs Delta vs Hudi

Iceberg is a strong default here because of open multi-engine access, snapshots, deletes, and table evolution.

Decision

Redshift vs Athena / Trino

Warehouse for curated BI and finance; Athena/Trino for open ad hoc lake queries.

Decision

Redis vs DynamoDB for online features

Redis for ultra-hot counters and ephemeral features, DynamoDB for durable keyed feature serving.

Say this

Narrate the architecture in one flow

Cover layers in order: producers, ingestion, Kafka, processing, lakehouse, serving, governance.

Say This In Interview

My architecture is layered: emit, validate, publish to Kafka, process in streaming and batch, store in Bronze/Silver/Gold, and serve BI plus features with governance around every layer.

Last reviewed June 2026 Β· By Prasoon Parashar

Numbers are interview assumptions, not real Netflix internal figures.

Was this tab useful?