Preparing chapter
Loading your data design lesson
The curriculum shell is ready while the requested chapter is being prepared. You can wait here or return to the design library.
Preparing chapter
The curriculum shell is ready while the requested chapter is being prepared. You can wait here or return to the design library.
Amazon · Big Data Engineering · Owned, observable, reconciled, least privilege
Govern Amazon-like orders, payments, inventory, fulfillment, sellers, customer behavior, and finance with explicit ownership, automated quality gates, reconciliation, lineage, privacy, tenant isolation, and recoverable publication.
01 · Ownership + contracts
Ownership follows operational commits through Kafka, trusted Silver facts, certified Gold products, and the consumer that acts on them. Each handoff keeps a contract, an on-call owner, and release evidence.
Responsibility
Orders, Payments, Inventory, Catalog, Fulfillment, and Seller services own event meaning, source completeness, stable IDs, sequencing, and compatible rollout.
Amazon example
Order Service proves order_item_id remains stable across retries and that price, discount, tax, shipping, quantity, currency, and offer version preserve their documented semantics.
Release evidence
Schema subject, compatibility result, retry fixture, deployment version, source counter, owner, and paging route.
Ownership boundary
A producer fixes missing or semantically wrong evidence; it does not own Kafka durability, Silver transforms, or BI refresh.
Example · safely evolve order_placed v3 → v4
Add safely
Add nullable fulfillment_program to order_placed v4 without changing existing monetary meaning.
Readers first
Deploy tolerant Flink, Spark, Iceberg, Trino, serving, and DLQ readers for v3 and v4.
Dual observe
Compare field population, enum drift, money totals, and failures by region and producer build.
Canary producer
Enable one marketplace cohort while the prior contract remains readable and replayable.
Publish adoption
Expose schema usage, unknown enums, null rate, consumer failures, and oldest active version.
Strengthen later
Require the field only after supported producers and registered consumers prove compatibility.
02 · Quality gates
Follow order, payment, inventory, fulfillment, seller, and behavior evidence from producer CI to the served product. Every gate states what it checks, what stops, and which proof survives.
Gate promise
Amazon check
Changing amount from minor units to decimal currency under the same field name is rejected even when the JSON remains syntactically valid.
Failure action
Stop that producer release, keep the previous schema active, and page the domain owner.
Proof retained
schema ID + fixture result + policy result + deployment cohort
PII, payment credentials, secrets, or another seller’s data is exposed.
Finance, orders, inventory, payouts, or fulfillment truth can drive a critical wrong decision.
One bounded marketplace, region, seller, category, currency, or hour is incomplete.
Noncritical metadata is wrong while the underlying commerce facts remain valid.
03 · Commerce reconciliation
Reconcile independent order, payment, inventory, fulfillment, seller, promotion, tax, and partner evidence at the correct grain. Every unexplained delta remains visible, assigned, and auditable.
01
Operational commit/outbox counters by domain and business date.
02
Accepted records plus attributable quarantines by topic and partition.
03
Contiguous offsets, manifests, row counts, and checksums.
04
Typed unique facts plus quarantined records with reason and owner.
05
Included facts plus documented exclusions and open exceptions.
Independent evidence
accepted order header + committed order items + promotion + tax + shipping
Reconciliation key
order_id + order_item_id + pricing_version + currency
Must remain true
Sum of item merchandise, discounts, tax, and shipping equals the documented order total under one rounding policy.
Late-data policy
Cancellations and amendments remain lifecycle evidence; they never overwrite the accepted commercial version.
04 · Freshness + observability
Separate producer delay, Kafka admission, Bronze archive, Silver validation, Gold publication, and serving refresh so a fast query can never hide stale orders, inventory, payouts, or finance data.
What it means
T0 is authoritative service commit time, paired with business event time when they differ.
Amazon example
Order acceptance uses the Order Service commit; a client click retains observed and ingest time separately.
Evidence retained
event_id + event_time + committed_at + source sequence
Measure
Service commits versus emitted events, outbox age, CDC position, schema rejection, and volume by domain, region, and version.
What it detects
A producer cohort stopped contributing valid order, payment, inventory, or fulfillment evidence.
Commerce impact
A healthy pipeline can still undercount commerce because the source never emitted the fact.
First response
Compare service counters, outbox/CDC state, and registry admission; page the producer owner.
05 · Privacy + access
Customer identity, addresses, payment evidence, search behavior, and seller data follow different policies. General analytics receives the minimum attributes, while access and deletion remain purpose-bound and provable.
Data at this boundary
Direct identifiers, addresses, device/IP signals, payment tokens, seller data, and behavioral text are labeled before admission.
Amazon example
Search text is Confidential because customers may type accidental PII; card credentials are prohibited from analytical events.
Protection
Contract policy, field allowlist, privacy class, owner, purpose, residency, and retention metadata.
At source
Email, phone, name, consent, and token mapping in a restricted identity system.
Distributed form
Surrogate customer_key and approved coarse attributes.
Who may read
Purpose-approved identity services and a small restricted operations group.
Retention + deletion
Consent/legal policy; delete or anonymize every eligible derived copy and record proof.
Example · two-hour access for a payment incident
An engineer needs a narrowly governed currency and processor slice—never broad customer identity, card credentials, or unrestricted S3 access.
Request
State dataset, columns, one region/seller scope, incident, purpose, and two-hour duration.
Classify
Resolve PII, payment, seller-tenant, residency, retention, and export restrictions.
Approve
Data owner plus Privacy/Security or Finance approves the minimum viable view.
Grant
Issue a JIT role to governed SQL or a controlled notebook—never blanket S3 access.
Observe
Log queries, rows, bytes, exports, unusual access, and policy decisions in real time.
Expire + prove
Revoke automatically, verify no uncontrolled copy remains, and retain an audit record.
06 · Failure + recovery
Drill incomplete sales, settlement mismatch, inventory drift, incompatible schemas, seller isolation, and privacy deletion. Repairs remain bounded to analytical evidence and are verified before certification.
Selected Amazon drill
Affected path
Order Service → Kafka → Bronze → Silver order items → gold.daily_sales
Symptom
Certified sales are materially below expected source and payment trends.
Contain safely
Stop Gold publication, show the prior snapshot and stale age, and notify Finance and BI consumers.
Repair path
Compare source commits, outbox/CDC, Kafka offsets, quarantines, dimension nulls, exclusions, and the latest transform by marketplace and hour.
Verification before close
Source, Silver, payment, and Gold totals reconcile; corrected partitions publish atomically with a linked postmortem.
Commerce invariant
Never make a bad total look complete by force-matching or silently changing exclusions.
Name the exact topic offsets, Iceberg snapshots, marketplaces, dates, currencies, sellers, products, and consumers affected.
Keep checkout independent and pin analytical consumers to the last certified snapshot or an explicit unavailable state.
Reconcile repaired outputs, record code/input/output versions and approvals, notify consumers, and add the failed invariant to automation.
