Apache Spark is a distributed computation engine for processing large datasets across a cluster. It supports batch processing, SQL and DataFrames, Structured Streaming, machine-learning workflows, and graph-oriented computation through a shared execution architecture. A driver coordinates an application, executors process data partitions in parallel, and an external cluster manager supplies resources. Spark is primarily an execution engine, not a permanent storage system.
A structured Spark application
PySpark
from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.appName("daily-revenue").getOrCreate()
revenue = (
spark.read.parquet("s3://company/orders/")
.filter(F.col("status") == "COMPLETED")
.groupBy("order_date", "country")
.agg(F.sum("amount").alias("revenue"))
)
revenue.explain(mode="formatted")
revenue.write.mode("overwrite").partitionBy("order_date").parquet(
"s3://company/curated/daily_revenue/"
)
spark.stop()
Detailed explanation
- Spark reads from and writes to systems such as HDFS, S3/ADLS/GCS, Kafka, JDBC databases, and lakehouse tables. Data durability normally belongs to those systems rather than executor memory.
- Its core abstraction began with RDDs—immutable partitioned collections with lineage. Modern production pipelines usually use Spark SQL, DataFrames, and Datasets so Catalyst can optimize relational expressions.
- Transformations are lazy. An action or streaming trigger causes Spark to optimize the required plan, form jobs and stages, and schedule one task per relevant partition.
- Performance comes from a combination of DAG execution, pipelining, reduced intermediate materialization, caching when reuse exists, compact binary/columnar processing, code generation, source pushdown, and adaptive execution—not simply 'keeping all data in RAM.'
- Fault tolerance comes from deterministic lineage, task/stage retry, reconstruction of lost partitions or shuffle outputs, durable input, and checkpointing where recovery state must survive application failure.
- Spark still has expensive operations: shuffles use network and local disk, skew creates stragglers, large state can OOM, and poorly chosen partitions or joins can make it slower than a simpler engine.
Spark control and data pathsDriver
Requests CPU and memory resources
↔
Cluster manager
Cluster manager
Launches and manages worker processes
→
Executors
Driver
Sends tasks; receives status and small results
↔
Executors
Executors
Read and write distributed data partitions
↔
Sources and sinks
Executor partitions
Exchange shuffle blocks across the data plane
↔
Executor partitions
The driver coordinates work; it is not the normal path for the dataset. Executors read, shuffle, process, and write distributed data.
Production example
A retailer can use one Spark application to read partitioned orders from object storage, join reference data, calculate daily revenue, and write an Iceberg/Delta table. A separate Structured Streaming application can continuously transform Kafka order events using the same DataFrame engine.
Strong closing line
Spark is a unified distributed execution engine: it coordinates parallel work over external data while exposing both low-level and optimizer-driven APIs.
