WhatsApp · Big Data Engineering · The platform in one screen

Start Here

Separate the encrypted messaging path from the privacy-safe big-data platform, then follow operational facts into streaming, the lakehouse, analytics, and ML.

01 · Platform mission

Turn system behavior into trusted data products

The big-data platform observes the encrypted messaging system; it does not become the message store or require plaintext content.

Privacy-safe producers

Shared backbone

Big-data plane

Governed consumers

Stream processing

Flink validates, deduplicates, enriches, windows by event time, and detects anomalies with checkpointed state.

WhatsApp example

message_accepted and message_delivered are joined by tokenized message ID to update one-minute delivery success and p99 latency by region and app version.

Big-data responsibility

Use event-time watermarks, stable event-ID dedupe, bounded keyed state, broadcast reference data, incremental checkpoints, DLQs, and idempotent versioned sinks.

Interview point

Explain late data and state limits. Exactly-once processing still needs logical dedupe and an idempotent sink across system boundaries.

02 · System boundary

Know which plane owns each decision

Use one delivered message to explain the boundary. The online path fulfills the user promise; the big-data path observes that promise and improves the system later.

STEP 1

Online messaging plane

Deliver ciphertext

The client encrypts the payload. Edge and routing services authenticate, order, persist pending delivery state, fan out to devices, and retry offline recipients.

User outcome: message delivered

STEP 2

Asynchronous handoff

Emit a lifecycle fact

After the state transition, the delivery service emits message_delivered with event time, tokenized IDs, region, device/network class, latency, and retry count.

Durable fact: no plaintext

STEP 3

Big-data plane

Compute a data product

Kafka buffers the event; Flink deduplicates and updates reliability windows; Iceberg retains history for Trino, batch correction, dashboards, and approved ML.

Consumer: SLO + analytics

03 · Interview scope

Cover the product, then narrow to the data platform

A complete WhatsApp-like product is broad. Name the surrounding features once, then spend interview time on the big-data responsibilities that determine the architecture.

Product context · acknowledge briefly

Show product awareness by mapping each surface to the data it creates.

Messages + receipts

delivery lifecycle, latency, retries

Groups + multi-device

fan-out, membership, hot-key skew

Media + calls

transfer and quality telemetry

Presence + notifications

session and delivery outcomes

Status + engagement

privacy-safe view/adoption facts

Safety + backups

restricted signals and retention

Implementing chat routing, encryption protocols, or call media transport is outside this big-data round.

Design deeply · interview focus

01event production

02stream ingestion

03real-time processing

04lakehouse

05batch + backfill

06data modeling

07quality + contracts

08observability

09privacy + deletion

10cost + retention

11multi-region DR

12serving + ML

04 · Data products

Build governed products from WhatsApp operational events

Each product needs an explicit grain, source events, deterministic transformation, serving path, concrete WhatsApp use, and privacy contract.

SOURCE LIFECYCLE

Delivery reliability data product

message_accepted → message_routed → message_delivered → message_read

Product grain

Silver: one row per tokenized message-recipient-device delivery outcome. Gold: one row per region × platform × network × minute.

Source events

Messaging and delivery services emit accepted, routed, attempted, delivered, read, failed, and expired events.

Big-data build

Validate schema, deduplicate event_id, join lifecycle stages, calculate accepted-to-delivered latency, handle late receipts, then aggregate success rate and percentiles.

Serving path

Flink publishes minute windows to real-time OLAP; Iceberg retains Silver facts and certified Gold history for Trino and experiments.

WhatsApp example

A message is accepted in ap-south and delivered 132 ms later on Android over 4G. The event records the 132 ms outcome—not the message text.

Privacy rule

Never include plaintext, phone numbers, raw contact books, or unnecessary device identifiers.

05 · Say this

Open with the data boundary, then follow the facts

60-second answer

“I split the design into an online messaging plane and a big-data plane. The first owns encrypted message delivery and low-latency state. The second consumes privacy-safe lifecycle events and telemetry through Kafka or Pulsar. Flink validates, deduplicates, enriches, windows, and detects anomalies; raw and curated history lands in an Iceberg lakehouse for Spark, Trino, dashboards, and approved ML features. I design for event-time, replay, schema evolution, hot keys, regional failure, deletion workflows, and tens of millions of peak events per second—without requiring plaintext messages.”

Preparing chapter

Loading your data design lesson

The curriculum shell is ready while the requested chapter is being prepared. You can wait here or return to the design library.

Open design library

Continue learning

NetflixStreaming analytics platform

UberMarketplace data platform

YouTubeVideo analytics platform

WhatsApp · Big Data Engineering · The platform in one screen

Start Here

Separate the encrypted messaging path from the privacy-safe big-data platform, then follow operational facts into streaming, the lakehouse, analytics, and ML.

01 · Platform mission

Turn system behavior into trusted data products

The big-data platform observes the encrypted messaging system; it does not become the message store or require plaintext content.

Privacy-safe producers

Shared backbone

Big-data plane

Governed consumers

Stream processing

Flink validates, deduplicates, enriches, windows by event time, and detects anomalies with checkpointed state.

WhatsApp example

message_accepted and message_delivered are joined by tokenized message ID to update one-minute delivery success and p99 latency by region and app version.

Big-data responsibility

Use event-time watermarks, stable event-ID dedupe, bounded keyed state, broadcast reference data, incremental checkpoints, DLQs, and idempotent versioned sinks.

Interview point

Explain late data and state limits. Exactly-once processing still needs logical dedupe and an idempotent sink across system boundaries.

02 · System boundary

Know which plane owns each decision

Use one delivered message to explain the boundary. The online path fulfills the user promise; the big-data path observes that promise and improves the system later.

STEP 1

Online messaging plane

Deliver ciphertext

The client encrypts the payload. Edge and routing services authenticate, order, persist pending delivery state, fan out to devices, and retry offline recipients.

User outcome: message delivered

STEP 2

Asynchronous handoff

Emit a lifecycle fact

After the state transition, the delivery service emits message_delivered with event time, tokenized IDs, region, device/network class, latency, and retry count.

Durable fact: no plaintext

STEP 3

Big-data plane

Compute a data product

Kafka buffers the event; Flink deduplicates and updates reliability windows; Iceberg retains history for Trino, batch correction, dashboards, and approved ML.

Consumer: SLO + analytics

03 · Interview scope

Cover the product, then narrow to the data platform

A complete WhatsApp-like product is broad. Name the surrounding features once, then spend interview time on the big-data responsibilities that determine the architecture.

Product context · acknowledge briefly

Show product awareness by mapping each surface to the data it creates.

Messages + receipts

delivery lifecycle, latency, retries

Groups + multi-device

fan-out, membership, hot-key skew

Media + calls

transfer and quality telemetry

Presence + notifications

session and delivery outcomes

Status + engagement

privacy-safe view/adoption facts

Safety + backups

restricted signals and retention

Implementing chat routing, encryption protocols, or call media transport is outside this big-data round.

Design deeply · interview focus

01event production

02stream ingestion

03real-time processing

04lakehouse

05batch + backfill

06data modeling

07quality + contracts

08observability

09privacy + deletion

10cost + retention

11multi-region DR

12serving + ML

04 · Data products

Build governed products from WhatsApp operational events

Each product needs an explicit grain, source events, deterministic transformation, serving path, concrete WhatsApp use, and privacy contract.

SOURCE LIFECYCLE

Delivery reliability data product

message_accepted → message_routed → message_delivered → message_read

Product grain

Silver: one row per tokenized message-recipient-device delivery outcome. Gold: one row per region × platform × network × minute.

Source events

Messaging and delivery services emit accepted, routed, attempted, delivered, read, failed, and expired events.

Big-data build

Validate schema, deduplicate event_id, join lifecycle stages, calculate accepted-to-delivered latency, handle late receipts, then aggregate success rate and percentiles.

Serving path

Flink publishes minute windows to real-time OLAP; Iceberg retains Silver facts and certified Gold history for Trino and experiments.

WhatsApp example

A message is accepted in ap-south and delivered 132 ms later on Android over 4G. The event records the 132 ms outcome—not the message text.

Privacy rule

Never include plaintext, phone numbers, raw contact books, or unnecessary device identifiers.

05 · Say this

Open with the data boundary, then follow the facts

60-second answer