Data Engineer Roadmap
6-week structured path from SQL to system design. Check off topics as you go — progress is saved in your browser.
0% complete
SQL & Data Fundamentals
Master the query language used in every DE interview
- Window functionsintermediate
ROW_NUMBER, RANK, DENSE_RANK, LAG, LEAD, NTILE
- CTEs and subqueriesintermediate
WITH clauses, correlated subqueries, recursive CTEs
- Sessionization & funnel analysisadvanced
30-min gap sessions, CASE WHEN funnel, cohort patterns
- SCD Type 2intermediate
valid_from/valid_to patterns, is_current flag
- Query optimizationintermediate
Indexes, partition pruning, EXPLAIN ANALYZE
Kafka & Streaming
The most-asked streaming topic in DE interviews
- Kafka architectureintermediate
Topics, partitions, offsets, consumer groups, ISR
- Producer configintermediate
acks, linger.ms, batch.size, compression, idempotence
- Exactly-once semanticsadvanced
Idempotent producer + transactions + READ_COMMITTED
- Consumer lag & monitoringintermediate
kafka-consumer-groups.sh, lag metrics, alert strategy
- Log compactionintermediate
Key-based retention, CDC topics, tombstone records
Apache Spark
Batch processing backbone — required at most data companies
- DAG, stages, tasksintermediate
Lazy evaluation, shuffle boundaries, narrow vs wide transforms
- Shuffle & partitioningadvanced
spark.sql.shuffle.partitions, repartition vs coalesce
- AQE & join strategiesadvanced
Broadcast join, sort-merge, shuffle hash, AQE skew handling
- Memory tuningadvanced
execution vs storage memory, off-heap, OOM patterns
- Delta Lakeintermediate
ACID on Parquet, MERGE/upsert, time travel, OPTIMIZE
Airflow, dbt & Orchestration
Pipeline orchestration — essential for any analytics engineering role
- Airflow DAG basicsbeginner
Operators, sensors, XComs, TaskFlow API (@task)
- Airflow scheduling & backfillintermediate
catchup, depends_on_past, schedule_interval
- dbt models & testsbeginner
ref(), source(), schema tests, custom tests
- dbt incremental modelsintermediate
is_incremental(), unique_key, merge strategy
- Airflow vs Prefect vs Dagsterintermediate
Trade-offs, when to choose what, interview comparisons
Cloud & Data Warehouse
Cloud-native data stacks dominate modern DE roles
- BigQuery internalsintermediate
Columnar storage, partition pruning, slots, flat-rate vs on-demand
- Redshift tuningintermediate
DISTKEY, SORTKEY, VACUUM, ANALYZE
- Snowflake architectureintermediate
Virtual warehouses, micro-partitioning, time travel, zero-copy clone
- Lakehouse vs Warehouseadvanced
Delta/Iceberg/Hudi, when to use each, open table format trade-offs
- AWS/GCP/Azure services mapbeginner
S3/GCS/ADLS, Glue/Dataflow/ADF, EMR/Dataproc/HDInsight
System Design & Behavioral
The final round — design at scale and tell your story
- Netflix system designadvanced
Full playback flow, concurrency, CDN, failure scenarios
- Capacity estimationintermediate
QPS, storage math, back-of-envelope calculations
- Trade-offs in system designadvanced
SQL vs NoSQL, strong vs eventual, Kafka vs SQS, Redis patterns
- STAR behavioral storiesintermediate
5 impact stories: performance win, conflict, failure, scope decision, leadership
- Amazon Leadership Principlesintermediate
Map your STAR stories to 6–8 most common LP questions
Progress is stored only in this browser. Clear local storage or use "Reset progress" to start over.