Apache Spark Documentation

What it is

Apache Spark is an open source engine that processes large datasets in parallel across a cluster. You describe transformations with DataFrames or SQL in Python (PySpark), Scala, Java or SQL, and Spark plans and runs them across many machines. It powers Databricks, Amazon EMR, AWS Glue, Google Cloud Dataproc, Microsoft Fabric and Azure Databricks, so the same skills transfer across clouds.

Version

As of 2026-09 the latest documentation is for Spark 4.2.0. Spark 4 requires Java 17 or later and Python 3.10 or later. Cloud services often run a slightly older version (for example AWS Glue 5.1 runs Spark 3.5), so check which version your platform uses; the DataFrame API you use daily is largely the same.

The pages worth reading

• Quick Start: run PySpark locally and try the interactive shell.
• Spark SQL, DataFrames and Datasets guide: reading and writing Parquet and other formats, joins, aggregations, window functions, and partitioning output.
• The PySpark API reference, the page you will search most.
• Structured Streaming programming guide: the modern streaming API, with event time, watermarks and output modes.
• Tuning and the performance section of the SQL guide: caching, shuffle partitions, broadcast joins and adaptive query execution.
• Spark Connect, the client-server mode many managed services now use.

How to learn it

Install PySpark locally with pip and practise on a few gigabytes of Parquet on your laptop before touching a cluster. Learn to read the Spark UI: jobs, stages, tasks, shuffle sizes and skew. Most real Spark problems are too much shuffling, skewed keys, many small files, or collecting too much data to the driver.

Prerequisites

Solid Python and SQL, and an understanding of columnar formats such as Parquet. Spark is not the first tool to learn: many datasets fit comfortably in DuckDB or a warehouse, and knowing when you do not need Spark is part of the skill.

Is it worth your time?

Yes, once you have the basics. Spark remains a core requirement for many data engineering roles, especially at companies with large data volumes or a lakehouse on Databricks or Fabric.

Common questions

Is it free? Spark is open source under Apache 2.0. Cloud services charge for the compute they run it on.

Do I need Scala? No. PySpark and Spark SQL are what most data engineering teams use today.