Preparing chapter
Loading your data design lesson
The curriculum shell is ready while the requested chapter is being prepared. You can wait here or return to the design library.
Preparing chapter
The curriculum shell is ready while the requested chapter is being prepared. You can wait here or return to the design library.
BookMyShow · Data Engineering · Replay and restatement
Move lossless events through Bronze, Silver, and Gold with auditable backfills.
Lakehouse architecture flow
Follow BookMyShow data through preservation, validation, modeling, release gates, and purpose-built consumption.
STAGE 01
Durable inputs
STAGE 02
Replayable evidence
STAGE 03
Reusable truth
STAGE 04
Official products
STAGE 05
Purpose-built reads
ARCHIVE · SELECTED COMPONENT
Copies exact source records and proves which ranges are safely stored.
Receives
Kafka records, CDC positions, and validated external files.
What happens
Buffer records into healthy object sizes, write immutable objects, verify checksums and readable schemas, then advance the last fully archived offset for each partition.
Produces
Immutable objects plus batch manifests, checksums, and complete offset ranges.
BookMyShow example
If inventory partition 42 archives offsets 8,000–9,999, the manifest is committed only after that object can be read and its checksum matches.
Interview boundary
Mark an offset complete only after the object commit and validation succeed. Retry overlap is acceptable because Silver deduplicates it.
Gold products
Select a Gold product to see the question it answers, its exact row grain, how it is built, and what must pass before release.
gold_booking_daily
How many bookings and seats were confirmed, and what ticket value did they represent?
One row means
One row per business date × city × show × booking channel.
Refresh + consumer
Hourly, then finalized daily
Product and sales operations
Built from
fact_booking + fact_booking_seat with date, city, show, and channel dimensions.
How BookMyShow uses it
Compare show sales, channel conversion, confirmed seats, and customer-paid value without mixing booking grain with seat grain.
Example: For a Mumbai 7:30 PM show, the product team can compare app and web bookings while still counting a three-seat booking once as a booking and three times as seats.
Release checks
Confirmed booking IDs must be unique; seat-line counts and amounts must reconcile to the confirmed booking facts.
If this check fails, keep the previous certified Gold snapshot visible and hold this version for repair.
Iceberg lifecycle
Follow one BookMyShow table from efficient files to atomic publication, reproducible reads, retention, and safe cleanup.
ICEBERG STEP 01
What happens
Write Parquet files for only the affected event or business-date partitions. Compact excessive small files without changing the logical rows.
BookMyShow example
A blockbuster launch may create thousands of small hourly booking files; BookMyShow compacts that date and show bucket before BI scans it repeatedly.
Safety rule
Use event or business date as the primary partition and optionally bucket show_id for large tables. Never partition directly by customer or seat.
If skipped or done badly
Too many tiny files increase metadata work, slow Trino scans, and make every later snapshot more expensive.
Controlled correction
A BookMyShow backfill replaces derived analytical data; it never repeats booking, payment, seat, or customer side effects.
TRIGGER
Bad code, late evidence, or a reconciliation mismatch affects a known product and period.
STABLE CONSUMER LANE
What users keep readingISOLATED REPAIR LANE
What engineers rebuildWhat happens
Pause the affected Gold publication and downstream refreshes. Keep dashboards and exports on the last certified snapshot while the repair runs.
BookMyShow example
If partner settlement double-counts refunds, BookMyShow blocks the new settlement version before any payout file is released.
Evidence retained
Incident ID, affected product, current certified snapshot, blocked candidate snapshot, owner, and detection time.
Safety boundary
Analytical history can rebuild data products; it must not repeat customer or payment actions.
SAFE REPLAY
Events → staging tables → compared products
Reapply idempotent analytical sinks and compare with authoritative booking and money evidence.
NEVER REISSUE
Payment commands or notifications
A historical event is evidence—not authorization to charge, refund, mint a ticket, email, SMS, or push again.
