ilum/core repository overview

⁠Ilum Core

⁠Overview

ilum/core is the central control plane for Ilum — a production-grade Spark orchestration platform designed for Kubernetes-native workloads. It provides a comprehensive REST API to launch, monitor, and manage Apache Spark applications across multiple clusters with enterprise-grade scalability and high availability.

• Spark Orchestration — launch and manage interactive and single Spark jobs via REST API
• Apache Livy Compatibility — embedded Livy 0.8.0-compatible REST endpoints for seamless integration
• Multi-Cluster Support — orchestrate Spark on Kubernetes, YARN, or local clusters from a single control plane
• Cloud Storage Integration — built-in connectors for AWS S3, Google Cloud Storage, and Azure Blob Storage
• Observability — OpenLineage integration for data lineage tracking
• Scalability & HA — stateless architecture with Kafka-based communication for horizontal scaling

⁠TL;DR / Quick Start

# Run Ilum Core (requires MongoDB, Kafka, MinIO)
docker run -d \
  --name ilum-core \
  -p 9000:9000 \

  -e MONGO_URI=mongodb://mongo:27017/ilum \
  -e SPRING_KAFKA_BOOTSTRAP_SERVERS=kafka:9092 \
  ilum/core:<version>

Note: For production deployments, use the official Ilum Helm Chart⁠.

⁠Supported Tags

• <version>: Ilum 6.6.1 + Spark orchestration engine + Java 21

⁠Environment Variables

⁠Infrastructure (Required)

⁠Optional Configuration

NOTE: Using grpc as communication type disables High Availability support. For production deployments with HA requirements, use kafka (default).

⁠Data Persistence (Volumes)

Ilum Core is stateless. All persistent data is stored in external infrastructure components (MongoDB, Kafka, MinIO).

TIP: For disaster recovery, ensure regular backups of MongoDB (metadata) and MinIO (Spark artifacts, event logs).

⁠Network Configuration

⁠Security

⁠Non-root Architecture

The image runs as user ilum (UID 1001). Ensure that mounted volumes have appropriate permissions.

⁠Secret Hygiene

CAUTION: Never pass sensitive credentials directly in environment variables!

Recommended methods:

1. Kubernetes Secrets — mount MongoDB URI, Kafka credentials, and MinIO keys as secrets.
2. IAM Roles (IRSA) — use cloud provider identity for storage access when running on managed Kubernetes.

⁠Vulnerability Scanning

Before production deployment, scan the image:

# Trivy
trivy image ilum/core:<version>

⁠Building the Image

Run commands from the root of the ilum-backend repository:

# Build the service
./gradlew clean build

# Build multi-platform Docker image
./gradlew dockerMultiplatformPush

⁠Image Verification

⁠Layer Analysis (Dive)

dive ilum/core:<version>

⁠Software Bill of Materials (SBOM)

# Syft
syft ilum/core:<version> -o spdx-json > sbom.json

⁠Image Contents — Details

⁠Dockerfile Components

• Base: Eclipse Temurin 21 JRE (via ilum/java:21.0.9_10-jre)
• Application: Spring Boot REST API (v6.6.1)
• Spark: Apache Spark 3.x binaries with cloud storage connectors
• Observability: OpenLineage 1.37.0 for data lineage
• Kyuubi Support: Kyuubi Spark SQL Engine 1.10.2 for serverless SQL
• Example Jobs: Pre-packaged PySpark and Scala job examples

⁠Cloud Storage Connectors

• AWS S3: Hadoop AWS 3.3.4 + AWS SDK Bundle 1.12.262
• Google Cloud Storage: GCS Connector (Hadoop3) 2.2.13
• Azure Blob Storage: Hadoop Azure 3.3.4 + Azure Storage 7.0.1

⁠API Documentation

The Ilum Core API is designed with an API-first approach.

Interactive API Documentation:

• Swagger UI: http://<ilum-core>:9000/docs/api/
• OpenAPI Spec: Available in the ilum-api-definition module

API Version Compatibility:

⁠Architecture & Features

⁠Job Types

⁠Interactive Jobs

Long-running Spark applications that accept consecutive workload requests without re-initialization overhead. Ideal for:

• Real-time data exploration
• Multi-tenant query workloads
• Serverless SQL via Kyuubi

⁠Single Jobs

Traditional batch Spark jobs submitted via API. Supports:

• Scheduled ETL pipelines
• One-off data transformations
• Integration with external orchestrators (Airflow, Argo Workflows)

⁠Communication Modes

⁠Cluster Support

• Kubernetes: Primary target for production workloads
• YARN: Legacy Hadoop clusters
• Local: Development and testing only

⁠Infrastructure Dependencies

Ilum Core requires the following components to operate:

• MongoDB: Metadata storage (clusters, jobs, sessions)
• Apache Kafka: Event streaming and inter-service communication (when using Kafka mode)
• MinIO: S3-compatible storage for Spark JARs, Python archives, configuration files, and logs

All dependencies are automatically deployed when using the ilum-aio Helm chart.

⁠Support and Feedback

• Issues: Bug Tracker⁠
• Documentation: Ilum Docs⁠
• Slack: Join our Community⁠