YouTube Data Engineering — Real-Time Streaming

Two-minute interview path

Turn continuous activity into fast, correct, replaceable signals

Lead with this

Explain the keyed event-time pipeline first. Then show how YouTube reconstructs playback, publishes low-latency products, and lets certified batch correct incomplete live history.

RouteKafka keeps each playback attempt ordered and replayable.
ReconstructFlink applies event time, state, timers, and deduplication.
ComputeJobs produce sessions, counters, trends, QoE, fraud, and features.
PublishStable keys and versioned upserts make late updates safe.
CertifyBatch reconciles complete history and replaces provisional windows.

01

Streaming Jobs

Core argument:Split the stream by decision latency: dedicated jobs consume shared Kafka evidence and publish sessions, counters, trends, alerts, and online features with stable keys.

Explore the streaming jobsInputs, six job responsibilities, live outputs, and YouTube examples.Explore

Kafka Topics

Playback · engagement
search · ads · QoE

->

Live Outputs

Pinot · Redis
alerts · versioned facts

02

Session State

Core argument:Keep one state record per playback attempt. Playing heartbeats add bounded time; pause, buffer, seek, duplicate, close, and late-reopen rules update that same versioned session.

Explore playback stateState machine, record fields, event rules, inactivity close, and late reopen.Explore

Normal path

video_start->Startedheartbeat (PLAYING)->Playingvideo_end / complete->Ended

Playing self-loop: each valid PLAYING heartbeat adds a bounded played delta and updates last_valid_event_time.

Branches from Playing

Late/offline event: reopen the same playback-attempt key, recalculate the session, and emit a versioned upsert—never a duplicate fact.

Identity, Timing, Playback, Experience, and Correctness are not separate services.They are five groups of fields inside one Flink state record for one playback attempt.

heartbeat / pause / seek->find attempt p-42->read + update its state->apply session rules->emit p-42 version N

Inside one state record

Rules applied after each event

03

Time + Windows

Core argument:Use event time for truth, watermarks for bounded waiting, and a late-data side path for evidence outside the live allowance. Never silently discard delayed mobile events.

Explore event time and windowsWatermarks, window types, late heartbeats, recovery, and live-versus-certified truth.Explore

Late mobile heartbeat

Put delayed playback back into the session where it happened

Example allowance: 5 min

10:02Heartbeat createdThe phone records event_time=10:02 for playback attempt p-42.->

OfflineStored on phoneThe tunnel has no signal, so the SDK keeps the event in its bounded local queue.->

10:07Arrives at FlinkThe network returns five minutes later. Processing time is 10:07, but event time remains 10:02.->

PlaceUse event timeFlink puts the heartbeat into p-42's original session instead of treating it as new activity at 10:07.->

CorrectUpsert p-42 v3The session total is recalculated and the sink replaces p-42 version 2 with version 3.

Arrives after the live allowance?Send it to a late-event side output -> retain it in Bronze -> nightly batch updates the official session and daily totals. The event is delayed, not discarded.

The five-minute allowance is an interview assumption. In production, choose it from measured mobile/TV delay distributions and the freshness target.

04

Output Contracts

Core argument:Every live product has a stable business key and update rule, so replay and late evidence replace the intended version instead of creating duplicate facts.

Explore output contractsKeys and write semantics for counters, sessions, trending, alerts, and features.Explore

OutputStable keyUpdate model

05

Runtime + Operations

Core argument:Use Flink where keyed state, event-time timers, and low latency matter; operate it through checkpoint health, lag, watermark, state, rescaling, and sink-commit signals.

Explore runtime and operationsFlink placement, Spark trade-off, recovery controls, state lifecycle, and production monitoring.Explore

Where the engine fits

Kafka

ordered event streams

->

Flink Runtime

runs session, trending, fraud, QoE, counter, and feature jobs
holds keyed state · fires timers · checkpoints progress

->

Serving + Storage

Pinot · Redis · alerts · lake

Operations surround this runtime.They detect lag, preserve state during failure, control memory, support upgrades, rescale jobs, and verify that outputs still commit.

07

Interview Answer

Core argument:Route by playback attempt, reconstruct with event time and state, publish versioned live products, and let certified batch replace incomplete history.

Open the complete interview answerFive-part speaking structure and the full answer to rehearse.Explore

Goal:turn continuous playback and engagement events into seconds-old watch sessions, counters, trends, alerts, and recommendation features without letting retries or delayed phones corrupt the results.

1. Bring together

Send every event for one playback attempt to the same Flink worker.

2. Reconstruct

Use event time and state to understand play, pause, buffering, seek, and completion.

3. Compute live

Produce watch time, video velocity, QoE, fraud, and recommendation signals.

4. Correct safely

Deduplicate retries and replace an older session version when delayed data arrives.

5. Certify later

Let nightly batch reconcile complete history and publish the official numbers.

Say thisThe goal of the real-time layer is to turn raw YouTube activity into useful signals within seconds while keeping playback sessions correct. I route all heartbeat, pause, seek, buffer, and end events for the same playback attempt to one Flink worker so it can rebuild that attempt in order and maintain its state. Event time and watermarks place delayed mobile events into the session where they happened; duplicates are ignored by event ID, and accepted late changes replace the older session version. Separate streaming jobs produce live video counters, trending features, fraud signals, QoE alerts, and recent recommendation features. These outputs are provisional for fast product decisions, while the nightly batch pipeline publishes the final deduplicated and fraud-filtered truth.

YouTube data platform — Real-Time Streaming

Turn continuous activity into fast, correct, replaceable signals

Streaming Jobs

Session State

Time + Windows

Put delayed playback back into the session where it happened

Output Contracts

Trending

Runtime + Operations

Interview Answer

Loading your data design lesson

YouTube data platform — Real-Time Streaming

Turn continuous activity into fast, correct, replaceable signals

Streaming Jobs

Session State

Time + Windows

Put delayed playback back into the session where it happened

Output Contracts

Trending

Runtime + Operations

Interview Answer