Preparing chapter
The curriculum shell is ready while the requested chapter is being prepared. You can wait here or return to the design library.
WhatsApp · Big Data Engineering · Reproducible history, controlled correction
Preserve bounded WhatsApp event ranges in Iceberg, define Bronze, Silver, Gold, and data-product contracts, enforce quality and privacy gates, maintain snapshots and files, and publish auditable repairs without affecting live messaging.
01 · Lakehouse flow
Follow a declared WhatsApp event range through preservation, conformance, point-in-time modeling, release gates, atomic publication, and purpose-built consumption. Hover, focus, or tap any component for its exact responsibility.
STAGE 01
Durable inputs
STAGE 02
Replayable evidence
STAGE 03
Reusable facts
STAGE 04
Official products
STAGE 05
Purpose-built reads
Coordinates source readiness, idempotent retries, compaction, expiration, and bounded backfills.
Connects every table and metric to source ranges, code, policy, owner, quality result, and snapshot.
02 · Layer contracts
The layers are not copies with nicer names. Each one changes the contract, allowed transformation, consumers, and publication guarantee.
Table family
messaging_events_rawdelivery_events_rawmedia_events_rawcalls_quality_rawTransformation contract
Storage rule
Partition primarily by ingestion day/hour and region. Retain according to event class and privacy policy—commonly 30–180 days, not forever.
Table family
message_delivery_eventgroup_membership_historymedia_transfer_attemptcall_quality_windowTransformation contract
Storage rule
Partition by a low-cardinality time transform; bucket or sort on stable tokens only when measured queries and skew justify it.
Table family
fact_message_deliveryagg_delivery_health_5mfact_call_qualityagg_network_quality_dailyTransformation contract
Storage rule
Publish an atomic snapshot with owner, grain, freshness, completeness, retention, lineage, and rollback target.
03 · Data products
A Gold table is useful only when its question, row grain, freshness, source facts, quality gate, and named consumer are explicit. Inspect each product to see the complete contract.
gold.agg_delivery_health_5m · DATA PRODUCT CONTRACT
Are accepted messages reaching eligible recipient devices reliably and within the regional latency objective?
Grain
One row per 5-minute window × region × platform × network class × app version
Freshness
1–5 minutes provisional; daily finalized
Named consumer
Messaging reliability on-call and release engineering
Built from
Recipient-device delivery lifecycles closed from accepted, routed, delivered, failed, expired, and retry facts.
Certification check
Accepted lifecycle coverage, duplicate rate, late-receipt budget, eligible-device denominator, and regional source-offset completeness.
WhatsApp example
The on-call isolates a p99 delivery regression to one Android release on 4G in ap-south without reading message text or raw phone numbers.
04 · Table + file layout
Choose grain first, then hidden partition transforms, sort order, file targets, compaction signals, and snapshot policy. Inspect each table for the reasoning behind its layout.
Compressed lake input
60.1 TB/day370B events/day × 650 B ÷ 4 compressionTarget data files
256–512 MBCompact measured small-file partitions toward the tested rangeFleet-wide file floor
≈117K/day60.1 TB/day ÷ 512 MB/file; distributed by table, region and timeIceberg tracks snapshots, manifests, data files, delete files, schemas, and partition specs. Rewrite through Iceberg-aware maintenance; never manually delete object-store files behind the table metadata.
05 · Iceberg mechanics
Iceberg is more than Parquet. Its catalog, snapshots, manifests, data files, delete files, schemas, and partition specs make atomic release, reproducibility, rollback, and policy-aware cleanup possible.
ICEBERG 01
What happens
Write only affected event-time or ingestion-time partitions, cluster within files for measured query patterns, and compact when file size, planning latency, or delete-file ratio crosses a tested threshold.
WhatsApp example
A dense call-quality hour and a sparse group-membership month use different table layouts and maintenance schedules instead of sharing one physical pattern.
Safety rule
Target roughly 256–512 MB files when benchmarks support it; never partition directly by message, user, device, or conversation identifiers.
If skipped or done badly
Tiny files multiply manifests, object-store requests, planning time, and commit conflicts across WhatsApp-scale regional writers.
06 · Batch DAGs
WhatsApp pipelines wait for bounded source ranges and reference history, repair late mobile outcomes, reconcile counts, and publish only after quality and privacy gates pass.
Pipeline
Starts when
Kafka offsets available
Processing
Append records and coordinates; quarantine rejected contracts
Output
Raw snapshot
Pipeline
Starts when
New late receipt IDs
Processing
Rebuild only affected message-device lifecycles
Output
Corrected Silver
Pipeline
Starts when
Group/device CDC complete
Processing
Build effective-dated membership and linked-device versions
Output
Point-in-time dimensions
Pipeline
Starts when
Region/day Silver complete
Processing
Reconcile facts, calculate certified metrics, publish snapshot
Output
Certified Gold
Pipeline
Starts when
Small files or metadata exceed limits
Processing
Rewrite files/manifests and expire safe old snapshots
Output
Healthy tables
07 · Replay boundary
The lakehouse can reconstruct trusted history. It cannot authorize a second customer-visible action or take ownership of message delivery, group state, media transfer, or calling.
SAFE ANALYTICAL REPLAY
Reapply deterministic transformations to immutable evidence, write idempotent analytical sinks, reconcile, and publish a new snapshot version.
NEVER REISSUE
Historical evidence never authorizes WhatsApp to resend a message, emit a second delivery/read receipt, alter membership, restart media transfer, ring a call, or notify a user.