ilum/airflow repository overview

⁠Ilum Airflow

⁠Overview

The ilum/airflow image series extends the official Apache Airflow image with:

• Ilum Integration — pre-installed providers (Livy, Kubernetes, Spark) and a plugin that auto-provisions default connections
• Multi-arch support — built for linux/amd64 and linux/arm64
• Specialized variants — from minimal (-plain) through full Spark (-spark) to dbt/Cosmos (-dbt)

⁠TL;DR / Quick Start

# Run standalone (default variant)
docker run -d \
  --name ilum-airflow \
  -p 8080:8080 \
  ilum/airflow:<version> standalone

# Access UI: http://localhost:8080
# Login: admin / password generated in logs

For Kubernetes environments, use the official Airflow Helm Chart⁠ with image.repository: ilum/airflow and the appropriate tag.

⁠Supported Tags

• <version>: Full provider set (Livy, K8s, S3) + example DAG
• <version>-plain: Slim — Livy provider only, minimal size
• <version>-spark: Spark 3.5.7 + PySpark + Java 17, example DAGs
• <version>-dbt: Like -spark + dbt-core 1.9, dbt-spark, astronomer-cosmos

TIP:

• plain — microservices, minimal resource footprint
• spark — native PySpark jobs, Spark Connect
• dbt — extends spark variant; adds dbt orchestration via Cosmos and Spark Thrift Server support

⁠Environment Variables

⁠Required

⁠Optional (Auto-Connection Plugin)

The ilum_default_connections plugin auto-provisions connections at startup if they don't exist. You can override them:

Variables can also be defined with hyphens, e.g. AIRFLOW_CONN_ILUM-LIVY-PROXY.

⁠Data Persistence (Volumes)

# docker-compose example
volumes:
  - ./dags:/opt/airflow/dags
  - ./logs:/opt/airflow/logs
  - ./plugins:/opt/airflow/plugins

⁠Network Configuration

⁠Integration with Ilum in Kubernetes

# Override example for helm_aio / helm_airflow
extraEnv:
  - name: AIRFLOW_CONN_ILUM_LIVY_PROXY
    value: "livy://ilum-core.ilum.svc.cluster.local:9888"

⁠Security

⁠Non-root Architecture

The image runs as user airflow (UID 50000). When mounting host volumes, ensure directories have appropriate permissions:

# On host
sudo chown -R 50000:0 ./dags ./logs ./plugins

⁠Secret Hygiene

CAUTION: Never pass passwords directly in environment variables!

Recommended methods:

1. Kubernetes Secrets — mount as files or variables
2. _FILE pattern — e.g., AIRFLOW__DATABASE__SQL_ALCHEMY_CONN_FILE=/run/secrets/db_conn
3. External Secrets Operator — sync with Vault / AWS Secrets Manager

⁠Vulnerability Scanning

Before production deployment, scan the image:

# Trivy
trivy image ilum/airflow:<version>-spark

# Grype
grype ilum/airflow:<version>-dbt

⁠Auto-Provisioned Connections

The IlumDefaultConnectionsPlugin creates at startup:

TIP: Connections are auto-configured for the ilum namespace. For other namespaces, edit the host in the UI or override via environment variable.

⁠Building Images

Run all commands from the docker/ilum-airflow directory:

# Default 
docker buildx build --platform linux/amd64,linux/arm64 \
  -t ilum/airflow:<version> . --push

# Plain (slim)
docker buildx build --platform linux/amd64,linux/arm64 \
  -t ilum/airflow:<version>-plain -f Dockerfile-plain . --push

# Spark
docker buildx build --platform linux/amd64,linux/arm64 \
  -t ilum/airflow:<version>-spark -f Dockerfile-spark . --push

# dbt (Cosmos)
docker buildx build --platform linux/amd64,linux/arm64 \
  -t ilum/airflow:<version>-dbt -f Dockerfile-dbt . --push

⁠Image Verification

⁠Layer Analysis (Dive)

dive ilum/airflow:<version>-spark

Check for:

• No temporary files in layers
• Efficient layer caching
• Individual layer sizes

⁠Software Bill of Materials (SBOM)

# Syft
syft ilum/airflow:<version>-dbt -o spdx-json > sbom.json

⁠Image Contents — Details

⁠Dockerfile (default)

• Apache Airflow 3.1.1
• Providers: amazon, apache-livy, cncf-kubernetes, fab
• flask-oidc, flask-openid (for SSO)
• Example DAG: ilum-example.py

⁠Dockerfile-plain

• Apache Airflow 3.1.1
• Providers: apache-livy, fab
• Minimal footprint (~300MB smaller)

⁠Dockerfile-spark

Everything from default + :

• Apache Spark 3.5.7 (binaries in /opt/spark)
• Java 17 JRE (headless)
• PySpark 3.5.7 with sql, connect modules
• Plugin ilum_default_connections.py
• DAGs: create_random_data.py, read_random_data.py

⁠Dockerfile-dbt

Everything from -spark + :

• dbt-core 1.9.0
• dbt-spark 1.9.0 (with PyHive, session)
• astronomer-cosmos 1.11.2
• dbt project: ilum_dbt_project
• Cosmos DAG: ilum_dbt_thrift.py

⁠Support and Feedback

• Issues: Bug Tracker⁠
• Documentation: Ilum Docs⁠
• Slack: Join our Community⁠