AWS Glue Developer Guide
The official guide to AWS Glue, AWS's serverless data integration service: the Data Catalog, crawlers, Spark-based ETL jobs, Glue Studio and data quality rules. The Data Catalog alone is worth understanding, because Athena, EMR and Redshift Spectrum all use it to find tables in S3.
6.0 (Spark 4.1)
AWS docs · Sept 2026
Guide
Other
Amazon Web Services
Reference
Intermediate
Free
What it is
AWS Glue is a serverless data integration service. Its main parts are:
- The Glue Data Catalog: a central metadata store of databases and tables that describe data in S3 and elsewhere. Amazon Athena, Amazon EMR and Redshift Spectrum read table definitions from it.
- Crawlers: jobs that scan data and infer schemas into the catalog.
- ETL jobs: Apache Spark jobs (in Python or Scala) that run without you managing a cluster, authored in code, notebooks or the visual Glue Studio editor.
- Triggers and workflows for simple scheduling, plus AWS Glue Data Quality rules and sensitive data detection.
Versions
The Glue version decides which Spark and Python you get. As of 2026-09 the version table lists AWS Glue 6.0 (Spark 4.1.1, Python 3.13) and Glue 5.1 (Spark 3.5.6, Python 3.11), with 5.1 as the default for jobs created without a version. Glue 0.9, 1.0 and 2.0 reached end of life in April 2026, and AWS Glue for Ray has an end-of-support notice. Check the version table before following a tutorial.
The pages worth reading first
- AWS Glue concepts and the getting started pages.
- Setting up IAM permissions: most first failures are permissions.
- Getting started with the Data Catalog, and crawlers.
- Authoring jobs, then job bookmarks, which let a job process only new data on each run.
- Working with partitioned data in S3, and the pages on Iceberg, Hudi and Delta Lake support.
- Glue pricing, which is charged per data processing unit hour.
How it fits the AWS data stack
A common AWS pattern is raw files landing in S3, Glue jobs or crawlers cataloguing and transforming them into partitioned Parquet or Iceberg tables, Athena for ad hoc SQL, Redshift as the warehouse, and Amazon MWAA or Step Functions for orchestration. AWS also groups these services in Amazon SageMaker Unified Studio, which brings EMR, Glue, Athena and Redshift together in one interface.
Is it worth your time?
Yes, if you chose AWS. Glue appears in a large share of AWS data engineering job descriptions and on the AWS Certified Data Engineer Associate exam. Learn plain PySpark first, so you understand what Glue is running for you.
Cost warning
Glue jobs and crawlers bill while they run. Use small test datasets, set a timeout and the minimum number of workers, and create an AWS budget alert before experimenting.
Common questions
Glue or EMR? Glue is serverless and simpler for typical ETL; EMR gives more control for large or unusual Spark workloads.
Ready to start?
Opens on Amazon Web Services in a new tab.
Stuck on something specific?
Writing only gets you so far. If you want an answer to your situation rather than the general case, book a session and we will work through it together. Sessions are free for approved Sefism members, and a few slots open each week.
Follow along
New writing, resources and project ideas land here first.