docker4ops/spark-with-hadoop repository overview

⁠Overview

This image provides a production-like single-node analytics stack you can run anywhere Docker is available.

It bundles:

1. Apache Spark (standalone master + worker)
2. Hadoop HDFS
3. Hive tooling (CLI / Beeline)

Each image is aligned with open-source Spark versions commonly used in Databricks Runtime (DBR).

For background, architecture, and version mapping, see the Spark with Hadoop on Docker⁠.

⁠What You Get

• Complete stack – Spark, HDFS, and Hive wired together and ready to use.
• Version fidelity – Tags map to specific Spark / Scala / Java combos that mirror OSS Spark used in DBR.
• Realistic semantics – Proper HDFS + Hive Metastore integration for accurate local reproduction.
• Fast setup – Pull and start in minutes; ideal for debugging, enablement, and CI smoke tests.
• Extensible – Mount your own data, code, and JARs; build custom images on top.

⁠Who It’s For

• Data platform engineers / SREs reproducing cluster issues locally.
• Developers testing integrations (Delta, JDBC, Kafka, etc.).
• Analysts and learners who need a safe, disposable Spark + HDFS + Hive sandbox.

⁠Tags & Compatibility

Tags correspond to specific Spark versions (for example: 3.4.x, 3.5.x, 4.0.0) and matching Scala/Java.
Choose the tag that matches your target runtime; for binary compatibility, align the Scala version as well.

A full version table and rationale are documented in the project site⁠.

⁠Quick Start

Pull an image

docker pull docker4ops/spark-with-hadoop:<tag>

⁠Run the container

docker run -d --name spark \
  -p 4040:4040 -p 8080:8080 -p 7077:7077 \
  docker4ops/spark-with-hadoop:<tag>

⁠Exec into the container and start Spark

docker exec -it spark bash
spark-shell        # or: pyspark

Spark UI: http://localhost:4040⁠