Bound impact
Identify the Bengaluru dates, affected trip IDs, and exact Kafka offset ranges.
Produces
Impact manifest
Uber lakehouse flow
Regional events move through one controlled path from durable ingestion to official history.
Kafka Connect / S3 sink→land raw events
Spark on EMR→clean · dedupe · conform
Spark SQL→join · reconcile · aggregate
Uber layer contracts
Compare the table families, transformations, and storage responsibilities carried by each lakehouse layer.
BronzeRaw source history
SilverConformed event evidence
GoldReconciled business facts
Bronze contract
Table family
location_pings_rawtrip_events_rawdispatch_events_rawpayment_events_rawTransformations
- Preserve producer payload
- Attach Kafka coordinates
- Partition by region + event date/hour
Storage rule
S3-backed Iceberg; restricted precise GPS; dense 4-second pings retained for 7 days.
Silver contract
Table family
location_pings_cleantrip_events_cleandispatch_match_eventspayment_events_cleanTransformations
- Deduplicate by event_id
- Conform schema versions
- Tokenize PII
- Flag GPS jumps above 300 km/h
Storage rule
Cluster by city/H3 or trip; quarantine invalid rows; compact small files.
Gold contract
Table family
trip_factdriver_shift_factcity_day_factpayment_settlement_factTransformations
- Stitch rider + driver + dispatch
- Derive route from Silver GPS
- Reconcile processor settlement
- Build city-level aggregates
Storage rule
Versioned Iceberg snapshots for finance, city operations, BI, and governed ML.
Uber batch DAGs
The main workflows form a cadence: repair late trips, load reference data, publish daily facts, then close finance.
| Schedule | Pipeline | Starts when | Processing | Output |
|---|---|---|---|---|
| Hourly | Trip reconciliation | Late-event trip IDs | Rebuild only affected trip lifecycles | Corrected Silver |
| Data ready | Dimension loading | City, driver, vehicle changes | Create effective-dated definitions | Conformed dimensions |
| By 06:00 local | Daily Gold publish | Region/day Silver complete | Build and reconcile official facts | Certified snapshot |
| Daily close | Finance reconciliation | Processor settlement file | Match captures, refunds, and settlements | Settlement facts |
Trip reconciliationHourly
Starts when: Late-event trip IDs
Rebuild only affected trip lifecycles
Output: Corrected Silver
Dimension loadingData ready
Starts when: City, driver, vehicle changes
Create effective-dated definitions
Output: Conformed dimensions
Daily Gold publishBy 06:00 local
Starts when: Region/day Silver complete
Build and reconcile official facts
Output: Certified snapshot
Finance reconciliationDaily close
Starts when: Processor settlement file
Match captures, refunds, and settlements
Output: Settlement facts
Also orchestrated:Iceberg compactionprivacy deletionversioned backfillpartition retry
Versioned backfill
A backfill creates a reviewable replacement snapshot instead of silently changing the history already used by finance and legal.
Example: a route-distance bug affected Bengaluru trips for 3 days.
Recompute
Run corrected Spark code against retained Bronze into isolated staging tables.
Produces
Candidate snapshot
Validate
Compare route samples, trip counts, fare totals, and processor settlement.
Produces
Validation evidence
Promote
Approve the new Iceberg snapshot and atomically switch governed readers.
Produces
New certified version
Audit trail: source offsets → old snapshot → code version → validation evidence → new snapshot → approver