Progress
1 visited · 0 revised
Chapter
5 / 10
On this page
Streaming jobs
Visualize the near-real-time processing layer
This is where topics become trusted live facts, alerts, and features.
Flink jobs
Pick one live pipeline to see its input, state, timing, and output path.
Selected job
Playback Sessionizer
silver.fact_watch_session updates
Flow
Input topics
playback.heartbeat.raw, video.buffer.*, video.pause, video.seek
KeyBy
user_id + profile_id + content_id + device_id
Window
Session windows with inactivity timeout
Watermark
10 minutes
What this job is doing
This job reads heartbeat, buffer, pause, and seek events for the same viewing attempt, deduplicates retries, and keeps one mutable keyed state object for the active session. Every valid event updates last_event_time, watch_seconds, pause_count, seek_count, and watched_segments until the close rule fires.
Why this shape works
It keys by user/profile/content/device because one playback attempt needs ordered state on one lane. That lets Flink decide whether the next heartbeat extends the same session, closes it, or starts a new one after inactivity.
Where the output goes
The emitted session updates feed Silver fact tables, downstream QoE analysis, and later Gold rollups. This is the step that turns noisy micro-events into trusted session-level truth.
Stateful logic
active session, seen event_ids, watched_segments, pause_count, buffering_seconds
Failure behavior
Backpressure or state blowup causes lag; checkpoint failures require restart from durable state.
Watch-time truth
See exactly what counts, what stays separate, and how completion is computed.
Interactive timeline
Heartbeats count. Pause does not. Buffering is tracked separately.
Counted watch seconds
120
Paused seconds
Not counted
Buffering seconds
Tracked separately
Toggle definitions
Engagement Watch Time
Total Watch Time
0–10 min + 5–15 min = 20 minutes
Unique Content Coverage
0–15 min = 15 minutes of unique coverage
Counting rules
Count `heartbeat_interval_sec` only when `is_playing = true`, `is_paused = false`, and the event is valid plus non-duplicate.
Store `total_watch_seconds`, `session_seconds`, `buffering_seconds`, and `unique_content_seconds_watched` as separate measures so one UX issue does not pollute another business metric.
Derived metrics
Compute `completion_pct` from `unique_content_seconds_watched / content_duration_seconds`, not from total watch time, so rewinds do not fake completion.
If buffering matters for QoE or UX alerts, keep it in its own metric family instead of inflating engagement watch time.
Turn noisy raw events into trusted sessions
The user should see exactly where pause logic, duplicates, and device changes are handled.
Session scenarios
Switch the situation and see how Flink decides session boundaries and journey rollups.
Selected scenario
Normal playback
User watches normally and stops. The session closes on stop/complete.
One playback session with clean start/end, watch seconds, and QoE counters.
Raw events → session
Flink session state
What Flink does
Flink sees a straightforward ordered sequence: play, periodic heartbeats, then stop. It opens session state on the first play, increments watch metrics on each valid heartbeat, and closes the session explicitly when stop or complete arrives.
Why this matters
This is the baseline path every other scenario is compared against. If the normal path is not crisp, pause logic, late updates, and crash recovery all become harder to reason about.
Decision rules
Key by user/profile/content/device
Close on video.stop or video.complete
Streaming speed still needs correction paths
Watermarks and late updates belong right next to the real-time pipeline story.
Late event simulator
Drag arrival delay to see how the event should be handled.
On-time event
Process normally with streaming state and live outputs.
Late but allowed
Update previous windows or sessions if within allowed lateness.
Very late event
Route to late_events and correction flow.
Process normally in the stream path.
Replay flow
Controlled correction, not manual copy-paste.
DLQ / quarantine catches malformed, schema-invalid, too-late, or business-rule-failed records
Engineer fixes mapping, schema, or transformation logic
Replay service marks selected records replayable and republishes to replay.playback.events
Streaming and batch pipelines consume replay idempotently
Silver and Gold tables are corrected using MERGE / partition overwrite
Audit records capture what was replayed, why, by whom, and what changed
DLQ is not data loss storage. It needs owner, SLA, dashboard, replay tooling, and audit.
One clean streaming answer
Keep the distinction between live facts and official batch truth clear.
Interview answer
The streaming layer turns raw events into trusted near-real-time facts using keyed state, watermarking, sessionization, and clearly defined metric logic like heartbeat-based watch time.
Last reviewed June 2026 · By Prasoon Parashar
Numbers are interview assumptions, not real Netflix internal figures.