Netflix Data Engineering — Requirements

Estimation story

Show the math path before the answer

A light estimation flow works better than a dense table. Hover any step to see what it is doing.

→

Why this step matters

Choose a simple workload model

Start with a small set of assumptions that you can defend easily on a whiteboard.

Safe baseline

Use one baseline and derive everything from it

Assumptions, calculations, and outputs should read like one flow. Hover any assumption or result to see what it really means.

Assumptions

Derived flow

→

Assumption detail

Use one interview-safe counting unit such as active profiles or active devices. The point is not the exact Netflix internal number; the point is to stay internally consistent through the whole derivation.

Daily events

20B

100M × 200 = 20B events/day

This is the total from the simplified workload class we are using in the interview: active profiles or devices generating behavior plus telemetry throughout the day.

Derived numbers

Make the calculations readable

This is the spoken math chain: what you multiply, what you divide, and why the next number follows from the previous one.

Daily events

100M active profiles/devices × 200 events/day

20B/day

→

Average events/sec

20B / 86,400 sec/day

≈ 231K/sec

→

Peak events/sec

231K × 5 peak factor

≈ 1.15M/sec

→

Raw payload/day

20B × 1 KB average size

≈ 20 TB/day

Use the simple estimate first. If the interviewer wants more realism, then you can add machine telemetry, CDN/device logs, replication, fan-out, and replay retention as second-order workload classes.

Board formulas

Write only the formulas you can defend on the board

Hover each formula so the math stays teachable instead of looking like memorized jargon.

Formula meaning

Average events/sec

2.2

average_events_per_sec = daily_events / 86,400

Example

20B / 86,400 ≈ 231K/sec

Convert a daily workload into a per-second baseline.

Always say that per-partition and per-task limits come from benchmarks, not universal constants.

Punchline

Keep the whole estimate in one place

After the formulas, keep the whole answer on one internally consistent model.

Interview-safe baseline

The small, internally consistent model you can derive live.

2.1

Daily events

20B

100M × 200 = 20B events/day

Average events/sec

231K/s

20B / 86,400 ≈ 231K/s

Peak events/sec

1.15M/s

231K × 5 ≈ 1.15M/s

Raw payload/day

20 TB/day

20B × 1 KB ≈ 20 TB/day

Peak network ingress

1.15 GB/s

1.15M/s × 1 KB ≈ 1.15 GB/s

Storage sizing

Retention is tiered, not one giant forever-hot store.

Iceberg / S3

Hot

Kafka 4-6h

Short replay window for fresh incidents and consumer recovery.

Using the 20 TB/day baseline, 6 hours is 20 × (6/24) ≈ 5 TB of hot replayable raw data.

Warm

S3 raw 1-2d

Replay buffer beyond Kafka retention if a correction arrives later.

Using the 20 TB/day baseline, two warm raw days means about 40 TB before compression and lifecycle movement.

Cold / long-term

Iceberg years

Partitioned, compacted, snapshotted, then aged into cheaper tiers.

Using the 20 TB/day baseline, 20 × 365 ≈ 7.3 PB/year before compaction, snapshot expiry, or cheaper storage classes.

If raw output lands around 20 TB/day, then a year of retained raw copies is already around 7.3 PB/year before compaction, snapshot cleanup, or colder storage classes.

Compute sizing

Keep the heuristics compact and tie them back to partitions and target lag.

Flink + Spark

Flink sizing

parallelism ≈ topic_peak_throughput / per_task_throughput

Start from Kafka partitions and target lag, then round up to a checkpoint-friendly number.

If a hot topic peaks at 1M msg/sec and one task safely handles 25K msg/sec, then 1,000,000 / 25,000 ≈ 40 tasks before headroom.

Spark sizing

num_executors ≈ total_cores_needed / executor_cores

Use the same executor sizing logic you would use in any large Spark batch workload.

If a batch job needs ~2,000 cores and you run 5 cores/executor, then 2,000 / 5 ≈ 400 executors.

The real punchline is not the exact executor count. It is that both stream and batch must consume from the same Kafka-backed log so replay, correction, and fan-out stay coherent.

At this scale, the answer is not “buy a bigger box.” The answer is partitioned Kafka, benchmarked stream parallelism, replay-friendly retention, and batch plus stream consuming from the same underlying log.

“I derive throughput from assumptions, convert it into peak load, then size partitions, parallelism, and retention from that peak rather than guessing static infrastructure numbers.”