loum/jupyter-pyspark repository overview

Need a notebook with access to a simple PySpark⁠ interpreter? Otherwise, check an alternate implementation, jupyter-spark-pseudo⁠, which is closer to a real Spark cluster (but a bigger image).

⁠What's new with latest

• Support for multi-platform images⁠.
• Bump to Jupyter Notebook 7.3.2⁠.
• Bump to Apache Spark 3.5.4⁠.
• %%sql cell magic support.
• SparkSession support for delta-spark⁠.

⁠Supported tags and respective Dockerfile links

• Apache Spark 3.5.4⁠
  • Jupyter Notebook 7.3.2⁠
    • latest, 7.3.2-3.5.4⁠
  • Jupyter Notebook 7.2.3⁠
    • 7.1.2-3.5.4⁠
  • Jupyter Notebook 7.1.2⁠
    • 7.1.2-3.5.4⁠
• Apache Spark 3.5.1⁠
  • Jupyter Notebook 7.1.3⁠
    • latest, 7.1.3-3.5.1⁠
  • Jupyter Notebook 7.1.2⁠
    • 7.1.2-3.5.1⁠

⁠Quick reference

• GitHub: loum⁠/jupyter-pyspark⁠
• Where to file issues: https://github.com/loum/jupyter-pyspark/issues⁠
• Apache Spark home: Apache Spark 3.5.0⁠
• Jupyter Notebook: https://jupyter.org/⁠

⁠Quick start

⁠Launch the Jupyter Notebook Server Container

Place your notebooks under the notebook directory and run:

mkdir -pv notebooks && docker run --rm -d\
 --name jupyter-pyspark\
 --hostname localhost\
 --env JUPYTER_PORT=8889\
 --env DRIVER_MEMORY=2g\
 --volume $PWD/notebooks:/home/user/notebooks\
 --publish 8889:8889\
 loum/jupyter-pyspark:latest

To get the URL of your local server:

docker exec -ti jupyter-pyspark bash -c "jupyter notebook list"

⁠Terminate Jupyter Notebook Server Container

docker stop jupyter-pyspark

⁠So far, so good!

With an active Jupyter Notebook server a PySpark instance should be running behind the scenes. To validate, dump the SparkContext help in a Jupyter Notebook cell:

help(sc)

⁠Things to note when coding in your Notebooks...

Spark context available as sc.

SparkSession available as spark.

⁠Example

Taken from Spark examples⁠:

import sys
from random import random
from operator import add
​
partitions = 2
n = 100000 * partitions
​
def f(_):
    x = random() * 2 - 1
    y = random() * 2 - 1
    return 1 if x ** 2 + y ** 2 <= 1 else 0
​
count = sc.parallelize(range(1, n + 1), partitions).map(f).reduce(add)
print("Pi is roughly %f" % (4.0 * count / n))

[Stage 0:>                                                          (0 + 2) / 2]
Pi is roughly 3.143960