Amazon · Big Data Engineering · Immutable evidence, conformed facts, atomic truth

Batch + Lakehouse

Design the Amazon-like S3 and Iceberg lakehouse, Bronze/Silver/Gold contracts, Spark batch processing, file layout, Airflow readiness gates, atomic publication, compaction, and isolated historical backfills.

01 · Lakehouse flow

Source evidence → trusted facts → certified products

Follow a bounded commerce data range through preservation, conformance, business modeling, release gates, atomic publication, and purpose-built consumption. Hover, focus, or tap a component for its exact responsibility.

Swipe the flow

Sources

Preserve

Trust

Certify

Consume

AIRFLOW

Readiness, retries, backfills, maintenance

CATALOG + LINEAGE

Owners, snapshots, quality, policy, cost

SILVER

Trust · component explanation

Silver conformance

Silver enforces schemas, deduplicates stable event IDs, applies source sequence, normalizes timestamps and money, tokenizes sensitive identity, merges CDC, and resolves effective-dated catalog, seller, customer, and promotion context.

What comes in

Pinned Bronze snapshots plus approved reference-history versions

What comes out

Reusable order, item, payment, inventory, shipment, click, customer and offer facts

Design promise

One declared business grain and deterministic rule set per table

Amazon example

silver.order_item keeps one row per accepted item while preserving the selected offer, seller, price, tax, discount, and lifecycle state.

02 · Layer contracts

Give Bronze, Silver, and Gold one promise each

These layers are not three copies with nicer names. Select a layer to see what changes, who may consume it, and which shortcuts are forbidden.

Silver contract

Publish reusable conformed facts

order · order_item · payment_transaction · inventory_movement · shipment · click_event

Allowed transformations

✓Enforce types + contracts

✓Deduplicate event_id

✓Apply source sequence + CDC MERGE

✓Normalize time, money, units, identity

Physical rule

Use hidden time transforms and selective bucketing or sort order based on measured queries.

Named users

Data products, analysts with purpose approval, feature engineering

Hard boundary

Do not mix grains or freeze consumer-specific metrics into shared facts.

03 · Data products

Make every Gold product defend its grain and certification

A useful Gold table names the question, exact row grain, sources, freshness tier, quality gate, and consumer. Select a product to inspect the whole contract.

gold.finance_order_ledger

Finance order ledger

What is the certified monetary outcome of every eligible order and payment transaction?

Row grain

One ledger entry per order item, payment transaction, fee, refund, chargeback, or settlement allocation

Source facts

Order items · captures · refunds · fees · chargebacks · processor settlements

Freshness contract

Provisional intraday · certified after cutoff/T+1

Certification gate

Money conservation, currency integrity, processor control totals, no duplicate transaction reference

Named consumers

Finance close, audit, seller payout, executive net-sales reporting

04 · Table + file layout

Design Iceberg around grain and query shape

Choose row grain first, then hidden partition transforms, sort order, file targets, write mode, and compaction. High-cardinality business IDs are usually sort or bucket candidates—not directory partitions.

Selected Iceberg design

silver.order_item

One accepted order item with current conformed lifecycle state

256–512 MB Parquet

01 · Organize reads

Hidden partition

months(order_time), region

Sort order

order_id, order_item_id

02 · Control writes

Target file size

256–512 MB Parquet

Write pattern

Idempotent MERGE by order_item_id and source version

03 · Defend the choice

Most queries prune by business time and region; sorting helps point lookups and joins while Iceberg hides physical transforms.

Anti-pattern to avoid

Do not overwrite immutable event history or treat order-header status as every item’s status.

File-maintenance loop

Compact data files, then maintain Iceberg metadata

logical rows unchanged

01 · Trigger

Too many files or average size falls below the table threshold.

02 · Rewrite

Compact toward the measured 128 MB–1 GB target without changing row meaning.

03 · Maintain

Rewrite manifests, expire approved snapshots, and remove orphan files safely.

05 · Spark + Airflow

Size the work, choose the join, wait for the data

Spark executes large transformations; Airflow or MWAA coordinates declared inputs, retries, quality gates, reconciliation, and publication. A clock alone does not prove that a business period is ready.

25 TB input · 256 MB planning split

97,656.25 → about 100K splits

25 TB × 1,000,000 MB/TB
÷ 256 MB/split
= 97,656.25 input splits

This estimates input parallelism—not executor count or runtime. Also size decompression, shuffle, spill, joins, skew, output files, network, memory, retries, and SLA.

Use when

The build side is genuinely small after filtering and comfortably fits each executor.

Primary risk

A stale size estimate or unfiltered dimension can OOM every executor.

Design response

Measure serialized size, pin the dimension snapshot, broadcast explicitly, and validate join coverage.

Data-aware daily DAG

Readiness → conformance → reconciliation → publication

Owner · timeout · retry · SLA · runbook

Wait for readiness

Require source offset/file manifests and pinned Iceberg snapshots—not merely 02:00 on a clock.

Build Silver orders

Apply schema, dedupe, source sequence, CDC MERGE, money normalization, and item-grain rules.

Build Silver payments

Create immutable payment transactions and incorporate complete processor evidence for the cutoff.

Run data quality

Check uniqueness, nulls, references, enums, lateness, privacy, and source-to-Silver counts.

Build candidate Gold

Write the order ledger and dependent products into a shadow snapshot using pinned inputs.

Reconcile

Balance orders, captures, refunds, fees, settlements, sellers, currencies, regions, and reporting periods.

Publish atomically

Commit one approved Iceberg snapshot and record run, code, metric, source, and quality versions.

Notify consumers

Refresh BI or serving only after publication; expose data-as-of and retain the prior rollback target.

06 · Publish + replay

Publish one version and isolate every repair

Iceberg’s value is the metadata transaction around Parquet: pinned reads, schema and partition evolution, atomic snapshots, time travel, rollback, MERGE, deletion, and policy-aware cleanup.

Iceberg publication step

Pin inputs

Record every Bronze and reference snapshot ID, source range, schema, code, and metric version before compute starts.

What readers see

CURRENT

Snapshot 1842

Still certified

CANDIDATE

Snapshot 1843

Invisible to readers

Atomic visibility

No partial file set

Reproducibility

Pin any snapshot ID

Rollback

Keep the prior version

Replay boundary

Rebuild analytical truth without repeating commerce

isolated · idempotent · reconciled

01 · SAFE ANALYTICAL REPLAY

Evidence becomes a new candidate snapshot

Evidence

Shadow

Reconcile

Publish

Pin immutable inputs, isolate compute and writers, apply deterministic transformations, compare the affected range, and publish one new version.

02 · OPERATIONAL HARD STOP

Historical data never authorizes a side effect

NEVER

Place orderCapture paymentReserve stockCreate shipmentIssue refundNotify customer

Replay may rebuild facts, metrics, features, and serving views. It cannot call transactional services or repeat a customer-visible action.

Backfill invariant: live and historical writers use separate branches or shadow tables. Reconcile the affected range, then swap one metadata pointer; never let a multi-petabyte repair interleave uncontrolled writes with the certified table.

Separate write lanePinned rangeAtomic swap

Preparing chapter

Loading your data design lesson

The curriculum shell is ready while the requested chapter is being prepared. You can wait here or return to the design library.

Open design library

Continue learning

NetflixStreaming analytics platform

UberMarketplace data platform

YouTubeVideo analytics platform