Preparing chapter
Loading your data design lesson
The curriculum shell is ready while the requested chapter is being prepared. You can wait here or return to the design library.
Preparing chapter
The curriculum shell is ready while the requested chapter is being prepared. You can wait here or return to the design library.
WhatsApp · Big Data Engineering · Owned, observable, privacy-safe
Govern message, recipient-device delivery, group, media, call, notification, status, and experiment evidence while preserving E2EE, explicit ownership, replayability, and independence from live messaging.
01 · Ownership + contracts
Ownership follows Message Service evidence through Kafka, Silver delivery facts, certified Gold products, and the SRE view that acts on them. Each handoff keeps a contract, an on-call owner, and release proof.
Responsibility
The Message Service team owns the meaning, completeness, privacy class, and compatible rollout of message_accepted, message_routed, message_delivered, and message_read.
WhatsApp example
Before publishing to messaging.lifecycle.v1, Message Service proves that event_id is stable across retries, conversation identity is protected, and no message body or phone number is present.
Release evidence
Schema subject, retry/offline/multi-device fixtures, compatibility result, deployment version, owner, and paging route.
Ownership boundary
Message Service fixes producer omissions and bad semantics; it does not own Kafka durability, Silver transformations, or dashboard refresh.
Example: safely add delivery_transport to message_delivered
Add optional field
Add delivery_transport to message_delivered without changing the meaning of existing fields.
Tolerant readers first
Deploy Flink, Spark, Pinot, Trino, and DLQ readers that accept both schema versions and unknown enum values.
Observe both shapes
Compare old and new records by app version, region, and schema ID before changing the producer population.
Roll out producers
Enable Message Service regions gradually while long-lived mobile and server versions continue to coexist.
Publish adoption
Expose field population, unknown-enum rate, consumer failures, and the oldest active schema version.
Strengthen later
Only make the field required after every supported producer and registered consumer proves compatibility.
02 · Quality gates
Follow message, group, media, call, and notification evidence from producer CI to the certified SRE view. Every gate states what it checks, what is quarantined, and what proof survives.
Gate promise
WhatsApp check
Adding a required delivery_transport field to message_delivered is rejected until tolerant readers are deployed and supported Message Service versions populate it.
Failure action
Stop that Message Service release; keep the previous schema active and page the Message Service on-call.
Proof retained
topic + schema ID + fixture result + policy result + rollout cohort
Message plaintext, phone/contact identity, or security evidence is exposed or unsafe.
Delivery, call, notification, or abuse truth could drive a wrong critical decision.
One bounded region, app version, network, or other dimension slice is incomplete.
A catalog label or noncritical description is wrong while the WhatsApp facts remain valid.
03 · Lifecycle reconciliation
Reconcile Message Service acceptance, Kafka admission, Bronze evidence, Silver lifecycle facts, and Gold exclusions at the recipient-device grain. Every unexplained delta stays visible and owned.
01
accepted transactions
02
messaging.lifecycle.v1 admitted
03
bronze.messaging_events archived
04
trusted + quarantined delivery facts
05
agg_delivery_health_5m + exclusions
Independent evidence
message_accepted + route decision + device eligibility + delivery/read receipts
Reconciliation key
message_id + protected recipient_device_key + ownership epoch
Must remain true
Every delivered or read row traces to one accepted logical message and an eligible recipient device; read implies delivered.
Late-data policy
Offline-device receipts may correct the daily fact later. Realtime remains labeled provisional; finalized Gold applies the allowed lateness rule.
04 · Freshness + observability
Separate Message Service delay, Kafka admission, Bronze archive, Silver validation, Gold publication, and dashboard refresh so a fast query can never hide stale delivery or call data.
What it means
T0 is the authoritative server event time or an explicitly classified client-observed time—not an untrusted phone clock treated as truth.
WhatsApp example
message_accepted uses the messaging service timestamp; call-quality samples retain both event and ingest time for skew detection.
Evidence retained
event_id + event_time + clock_source + producer
Measure
Message Service accepted counters versus emitted message_accepted records by region, app version, schema ID, and event type; serialization errors and client-upload delay.
What it detects
One Message Service region or old-app cohort stopped contributing valid lifecycle evidence even though message delivery still works.
WhatsApp impact
Messaging SRE can undercount accepted or delivered messages and report a false success rate for the affected cohort.
First response
Compare service transaction counters, outbox success, schema rejection, and messaging.lifecycle.v1 admission; page the producer without coupling remediation to live delivery.
05 · Privacy + access
Message bodies and encrypted media never enter analytics. Lifecycle metadata is minimized, tokenized, purpose-bound, retained for a declared horizon, and deleted through every eligible table, cache, and export.
Data at this boundary
Encryption keys, message bodies, media, phone numbers, raw contact books, and direct identifiers remain inside their product security boundary.
WhatsApp example
Messaging emits a delivery envelope; it does not export message plaintext to Kafka for analytics.
Protection
Service isolation, encryption, audited operational access, and no general analytical read.
At source
Encrypted body and media remain in the messaging plane.
Distributed form
Not present; analytics receives lifecycle metadata only.
Who may read
No analytical consumer.
Retention + deletion
Not applicable because content is not ingested.
Example: two-hour access for a WhatsApp delivery incident
A reliability engineer needs a narrowly governed slice of silver.message_delivery_event for one region—never message content or a general lake grant.
Request
Messaging Reliability requests recipient-device delivery evidence for one incident and region.
Owner approval
The silver.message_delivery_event owner confirms the grain, need, and incident scope.
Declare purpose
The request records reliability investigation—not broad exploration or user profiling.
Narrow the view
Policy exposes only approved lifecycle fields, coarse region, and pseudonymous device keys.
Audit every read
Queries, exports, dataset version, requester, and incident ID enter the access log.
Expire automatically
The two-hour grant closes and any approved export follows its retention/deletion rule.
06 · Failure + recovery
Drill lifecycle-topic lag, incompatible schemas, regional evidence gaps, failover duplicates, Iceberg publication, and privacy incidents. Repairs are bounded to analytical evidence and verified before certification.
Selected WhatsApp drill
Affected path
Kafka critical topics → stateful stream jobs → realtime delivery health
Symptom
Dashboard freshness rises while brokers or jobs may still report as running.
Contain safely
Expose stale age, protect critical consumers, pause non-critical groups, and avoid blind partition increases.
Repair path
Check traffic, lost parallelism, hot keys, sink latency, checkpoint duration, deployment changes, and throttled dependencies in that order.
Verification before close
Lag returns within SLO, checkpoint and sink latency stabilize, and affected windows reconcile to Silver.
WhatsApp invariant
Lag may delay analytics; it must never block or alter WhatsApp message delivery.
Near-zero RPO for accepted messaging.lifecycle.v1 evidence needed to reconstruct delivery reliability; restore recent ingestion and stream processing in minutes without storing message bodies.
agg_delivery_health_5m and agg_call_quality_5m may recover with minute-level RPO and hour-level RTO only when dashboards show stale age and live WhatsApp messaging continues.
Verify Kafka topic/partition offsets, Bronze manifests and checksums, Iceberg catalog backups, exact Gold control totals, access policy, and a tested regional restore drill.
