Preparing chapter
The curriculum shell is ready while the requested chapter is being prepared. You can wait here or return to the design library.
Amazon · Big Data Engineering · Durable handoff, defensible ordering, bounded replay
Design the outbox and CDC handoff, domain topic strategy, partition keys, campaign-scale capacity, retry semantics, workload isolation, retention, and controlled replay for an Amazon-like commerce platform.
01 · Event handoff
Inspect the transactional handoff from an operational database to Kafka. The path makes event loss retryable without making MSK, Flink, or the lakehouse a checkout dependency.
Transactional facts
Order, payment, and inventory changes commit with a pending event, then publish asynchronously.
Client behavior
Schema, size, rate, consent, and prohibited-PII checks happen before a client batch enters Kafka.
Partner batches
Fingerprint files, verify manifests and control totals, then emit normalized records and completeness state.
Ingestion preserves the source contract’s event_id, business time, authority, schema, entity key, sequence and privacy class; it adds trusted receive time and Kafka coordinates.
02 · Topics + partition keys
Avoid one giant topic and one universal customer key. Separate domains by criticality, volume, retention, schema ownership, and the exact lifecycle order consumers can defend.
03 · Capacity + campaign surge
The equations reuse the Requirements assumptions. Two independent partition floors are calculated before adding regional allocation, broker-loss headroom, hot-key risk, consumer parallelism, and forecast growth.
Peak planning baseline
Decimal units · 1 KB average event · order and behavior baseline only
Peak order events
69,444.44/s
10M orders/day × 30 events/order ÷ 86,400 × 20Peak behavior events
347,222.22/s3B events/day ÷ 86,400 × 10Combined peak
416,666.67/s69,444.44 + 347,222.22 events/sLogical ingress
416.67 MB/s416,666.67 events/s × 1,000 bytes ÷ 10⁶Floor A · byte throughput
The 10 MB/s allowance is a benchmark input for the chosen record size, compression, replication, acknowledgement, and broker profile—not a Kafka constant.
Floor B · consumer work
Benchmark the slowest critical consumer with real schema decoding, dedupe state, checkpointing, enrichment, and sink latency.
Defendable baseline answer
max(55 byte lanes, 11 processing lanes) = 55 minimum baseline lanes
This is not one topic’s final partition count. Allocate by regional and domain peak, add payments, inventory, fulfillment, catalog and CDC, then include AZ loss, hot keys, growth and consumer parallelism.
Replicated broker writes
1.625 GB/s416.67 MB/s × RF3 × 1.30 headroomAlso model consumer egress, replica catch-up, disk retention, leader imbalance, rebalances, and broker/AZ failure.
Campaign surge playbook
Detect
Behavior produce rate, broker ingress, and consumer lag rise while order traffic remains normal.
Contain
Enforce behavior producer quotas, degrade optional sampling, preserve reserved broker and consumer capacity for order and payment topics.
Recover + prove
Scale its consumer groups, replay within the behavior retention window, then reconcile Kafka offsets to Bronze counts and sampling declarations.
04 · Reliability
Kafka is durable at-least-once transport. Business correctness still depends on the source commit, stable identities, source sequence, checkpointed state, idempotent effects, reconciliation, and workload isolation.
Producer
acks=all · idempotence · compression · bounded retry
Broker
RF3 · min ISR · quotas · rack/AZ awareness
Consumer
checkpoint · event dedupe · idempotent sink · lag SLO
05 · Retention + replay
Retention follows criticality, volume, privacy, and recovery time. Kafka provides bounded hot replay; immutable Bronze provides economical history after offsets expire.
Topic family
Retention
7–14 days
Policy
deleteLong enough for critical replay, reconciliation, incident recovery, and common consumer outages; Bronze holds durable history.
Topic family
Retention
3–7 days
Policy
deleteBounded hot recovery for operational products while movement and shipment history lands continuously in Bronze.
Topic family
Retention
12–48 hours
Policy
deleteExtreme volume; sink raw evidence quickly and keep Kafka focused on transit rather than economical long-term storage.
Topic family
Retention
Current + tombstones
Policy
compact,deleteDistribute the latest keyed value and controlled deletions while Iceberg preserves effective-dated history.
Topic family
Retention
3–7 days
Policy
deleteCover connector and sink recovery; protect WAL/binlog retention separately because Kafka retention cannot repair an expired source log.
Topic family
Retention
7–30 days
Policy
deleteRetain invalid records long enough to fix and replay with restricted access, owner, reason code, and original source pointer.
Controlled replay
Identify topics, partitions, offsets, schemas, regions, business keys, and the exact time range affected.
Run a separately throttled consumer group so correction cannot steal capacity from live order and payment products.
Apply corrected code to a shadow table or serving version; never overwrite the certified product in place.
Compare source, outbox, Kafka, Bronze and output counts, then atomically promote or roll back the new version.
When offsets have expired, read immutable Bronze records with their original topic, partition, offset, schema ID and ingestion time. Kafka is the hot transport log; Bronze is the durable replay evidence.