amazon/aws-glue-libs repository overview

⁠AWS Glue 5.0 Docker Container

AWS Glue provides Docker container images that enable developers to build and test their ETL (Extract, Transform, Load) jobs in local development environments. These containers provide a development environment that closely resembles the AWS Glue service, offering:

• A consistent development experience across team members
• Ability to develop and debug Glue jobs without incurring significant cloud costs
• Support for various development workflows including script development and testing
• Integration with common development tools and IDEs
• Access to the same AWS Glue ETL libraries and dependencies available in the cloud environment

The containers support both batch and streaming workloads, and can be used with direct script development and test-driven development approaches.

⁠Features

• Based on Amazon Linux 2023
• Contains AWS Glue ETL Library with Apache Spark 3.5.4-amzn-0
• Includes open table format libraries:
  • Apache Iceberg (1.7.1.amzn.1-1.amzn2023)
  • Apache Hudi (0.15.0.amzn.5-1.amzn2023)
  • Delta Lake (3.3.0.amzn.0-1.amzn2023)
• Supports both x86_64 and arm64 architectures

⁠Usage

⁠Spark Submit

# Set up workspace and script locations
WORKSPACE_LOCATION=/local_path_to_workspace
SCRIPT_FILE_NAME=sample.py

# Run container with spark-submit
docker run -it --rm \
    -v ~/.aws:/home/hadoop/.aws \
    -v $WORKSPACE_LOCATION:/home/hadoop/workspace/ \
    -e AWS_PROFILE=$PROFILE_NAME \
    --name glue5_spark_submit \
    public.ecr.aws/glue/aws-glue-libs:5 \
    spark-submit /home/hadoop/workspace/src/$SCRIPT_FILE_NAME

⁠PySpark REPL Shell

# Launch interactive PySpark shell
docker run -it --rm \
    -v ~/.aws:/home/hadoop/.aws \
    -e AWS_PROFILE=$PROFILE_NAME \
    --name glue5_pyspark \
    public.ecr.aws/glue/aws-glue-libs:5 \
    pyspark

⁠PyTest

docker run -i --rm \
    -v ~/.aws:/home/hadoop/.aws \
    -v $WORKSPACE_LOCATION:/home/hadoop/workspace/ \
    --workdir /home/hadoop/workspace \
    -e AWS_PROFILE=$PROFILE_NAME \
    --name glue5_pytest \
    public.ecr.aws/glue/aws-glue-libs:5 \
    -c "python3.11 -m pytest --disable-warnings"

⁠Key Differences from Glue 4.0

• Single container image for both batch and streaming jobs
• Default user is hadoop (previously glue_user)
• JupyterLab and Livy has been removed (can be manually installed)
• Open table format libraries (Iceberg, Hudi, Delta) pre-loaded by default

⁠Security Advisory

⚠️ These container images are designed exclusively for local development and testing purposes.

• DO NOT deploy these containers in production environments
• DO NOT make the containers publicly accessible
• DO NOT expose unnecessary ports to the network
• DO NOT use these containers as production ETL runners

The containers should only be run in secure, isolated development environments.

⁠AWS Glue ETL Container Images for versions v2.0 to v4.0

Refer the following for usage instructions

• AWS Glue Documentation Developing and testing AWS Glue job scripts locally⁠.
• AWS Big Data blog article - Develop and test AWS Glue version 3.0 jobs locally using a Docker container⁠.