Preparing chapter
Loading your data design lesson
The curriculum shell is ready while the requested chapter is being prepared. You can wait here or return to the design library.
Preparing chapter
The curriculum shell is ready while the requested chapter is being prepared. You can wait here or return to the design library.
WhatsApp · Big Data Engineering · Regional, replayable, failure-isolated
Connect privacy-safe WhatsApp event producers to regional ingestion, streaming, the lakehouse, and analytical serving without placing analytics on the messaging path.
01 · Architecture map
Follow privacy-safe WhatsApp lifecycle events from operational producers through durable regional ingestion, fast streaming, immutable history, certified products, and workload-specific serving.
Producers
Ingestion
Processing
Lakehouse
Serving
Cross-cutting control plane
02 · System boundary
The architecture has a deliberate asynchronous seam: operational services own live messaging state, while the big-data platform owns replayable analytical truth. Choose a boundary to inspect its responsibility.
Boundary explanation
02 / 03
After a WhatsApp service commits a real state change, this bridge creates the matching analytical event. An outbox or CDC process prevents the common failure where delivery succeeds but its event disappears before reaching Kafka.
WhatsApp example
The message service commits message_accepted and an outbox record together. If Kafka is temporarily unavailable, the message remains accepted and the same outbox event is retried later with the same event_id.
What this stage decides
It validates the schema, adds a trusted ingestion time, preserves the source event time and region, verifies the privacy label, and sends invalid records to quarantine with a visible reason.
What it cannot decide
It never copies message text, encryption keys, raw phone numbers, contact books, or media bytes. A publishing delay must never roll back or block a successful WhatsApp delivery action.
Why WhatsApp splits the data path after Kafka
LIVE WHATSAPP HEALTH · SECONDS
As message_accepted, message_delivered, reconnect, push, and call-quality events arrive, Flink removes duplicate event IDs and calculates one- and five-minute metrics by region, app version, platform, and network class. Operations can see a delivery-latency spike quickly, but the view remains provisional because offline phones may report late.
CERTIFIED WHATSAPP TRUTH · MINUTES TO HOURS
Bronze keeps every accepted event unchanged. Spark later rereads the complete window, includes delayed mobile receipts, removes retry duplicates, and joins the device or group-membership version that existed when the message was sent. It rebuilds device-level delivery facts and publishes corrected daily reliability metrics only after reconciliation passes.
The same WhatsApp events provide fast incident signals first and corrected official metrics later.
