Amazon · Big Data Engineering · Decoupled, replayable, and failure-isolated

Architecture

Connect Amazon-like operational services to CDC, Kafka, streaming, the lakehouse, batch processing, serving, and ML without placing analytics on the checkout path.

01 · End-to-end map

Follow a fact from checkout to a decision

Hover, focus, or tap any component to see its exact responsibility. The architecture separates live commerce, durable movement, fast processing, historical correction, and fit-for-purpose serving.

Swipe the map

Producers

Ingestion

Processing

Lakehouse

Serving

LOG

Durable event backbone · component explanation

Kafka / MSK

Kafka is the durable distribution layer between Amazon-like producers and independent consumers. Flink, Bronze sinks, monitoring, fraud, and future products can consume the same retained facts at their own speed.

Why this design

Separate topic families by domain and workload. Key orders by order_id, payments by payment_id or order_id, inventory by SKU plus location, shipments by shipment_id, and behavior by session when local ordering matters.

What comes in

Validated domain events, CDC records, behavioral telemetry, feed notifications, schema versions, privacy labels, and producer identity.

What comes out

Retained partition logs, consumer offsets, replay windows, lag signals, dead-letter records, and independently scalable subscriptions.

Architecture promise · Durable replay with ordering inside the chosen business key

02 · System boundary

Keep the lakehouse off the checkout path

Select a boundary to understand what it owns, how facts cross it, and which decisions the analytical platform must never take over.

BOUNDARY 02 · EXPLANATION

Outbox + CDC bridge

A service commits the business change and an outbox row in the same database transaction. CDC retries that immutable record until Kafka accepts it, preserving a stable event ID and source sequence across failures.

Amazon example

The order is confirmed while Kafka is unavailable. Checkout still succeeds; the outbox record waits, then CDC publishes the same order_confirmed event when the backbone recovers.

What it must decide

The bridge preserves event time, trusted ingestion time, source position, schema version, business key, privacy class, and enough metadata to measure lag and replay safely.

What it must never do

Publishing delay must not roll back a valid order, and retrying must not create a second order, second payment, or duplicate revenue.

Live path · seconds to minutes

Useful now, explicitly provisional

Kafka and Flink power alerts, fraud signals, seller views, operational counters, and online features with bounded lateness and visible staleness.

Certified path · minutes to T+1

Complete, reconciled, reproducible

Bronze, CDC, settlements, late facts, Spark, and Iceberg rebuild the full history before finance, BI, and governed ML receive official results.

03 · Failure isolation

Design the recovery path before the happy path fails

Select an incident. Each response protects checkout, exposes degraded data honestly, replays from durable evidence, and proves correctness before certification resumes.

Detection signal

Kafka region unavailable

Publish latency and outbox backlog rise; regional consumers stop advancing.

Customer + product impact

Checkout and order commits continue because they do not wait for Kafka. Data-driven views may become stale and must say so.

Contain the blast radius

Retain outbox and CDC positions, apply producer backpressure and quotas, isolate behavior traffic, and stop any consumer that cannot tolerate stale inputs.

Recover from durable evidence

Restore or fail over the regional backbone, publish from the last durable source position with the same event IDs, and let idempotent consumers catch up.

Prove it is safe again

Compare source/outbox counts with Kafka and Bronze, verify no partition gaps, measure maximum event lag, and reconcile a complete business interval before clearing the incident.

Preparing chapter

Loading your data design lesson

The curriculum shell is ready while the requested chapter is being prepared. You can wait here or return to the design library.

Open design library

Continue learning

NetflixStreaming analytics platform

UberMarketplace data platform

YouTubeVideo analytics platform