Failures + Data Quality
Protect Uber's live marketplace and trusted history with measurable quality gates and rehearsed recovery paths.
Quality SLOs
Monitor infrastructure, pipeline health, record validity, and business reconciliation as separate signals.
Signals to monitor
Uber SLO
99.9% of accepted location events reach the regional live map within 5 seconds.
Separate producer delay from Kafka lag and Flink processing delay before scaling or failing over.
Failure response
For every Uber incident, state detection, blast radius, immediate mitigation, recovery, and the prevention change.
Selected incident
Bad GPS data
Speed above 300 km/h, teleport distance, sequence regression, and distribution drift checks.
The driver, trip, and affected H3 cells until the event is removed from aggregates.
Flag or drop the ping before live state; preserve it in quarantine with the validation reason.
Recompute affected trip routes and supply windows, then retroactively flag impacted trips.
Client attestation, sequence checks, multi-signal anomaly rules, and monitored model thresholds.
Quality gates
Stop bad records at the earliest trustworthy boundary and preserve enough evidence to explain every rejection.
On failure
Reject the write or route the producer version to quarantine.
Evidence retained
Schema ID + producer version + owner
Replay and recovery
Use retained source offsets and versioned outputs to recover correctness without rewriting trusted history in place.
Scenario: A route-distance bug affected Bengaluru trips for 3 days.
Bound impact
Find dates, trip IDs, and Kafka offset ranges.
Recompute
Run corrected code into isolated versioned tables.
Validate
Compare route samples, trip totals, fares, and settlements.
Promote
Atomically switch governed readers to the approved snapshot.
Never overwrite production Gold in place; finance and legal keep the old and new evidence.