botanyhelp/job-wq-1

By botanyhelp

Updated over 1 year ago

works with this task: https://kubernetes.io/docs/tasks/job/coarse-parallel-processing-work-queue/

Image
Message queues
0

390

botanyhelp/job-wq-1 repository overview

kubectl create -f https://kubernetes.io/examples/application/job/rabbitmq/rabbitmq-service.yaml

kubectl create -f https://kubernetes.io/examples/application/job/rabbitmq/rabbitmq-statefulset.yaml

kubectl run -i --tty temp --image ubuntu:22.04
apt-get update && apt-get install -y curl ca-certificates amqp-tools python3 dnsutils
nslookup rabbitmq-service
env | grep RABBITMQ_SERVICE | grep HOST
RABBITMQ_SERVICE_SERVICE_HOST=10.0.147.152 #..but use real IP in cluster
export BROKER_URL=amqp://guest:guest@rabbitmq-service:5672
BROKER_URL=amqp://guest:guest@$RABBITMQ_SERVICE_SERVICE_HOST:5672
/usr/bin/amqp-declare-queue --url=$BROKER_URL -q foo -d
/usr/bin/amqp-publish --url=$BROKER_URL -r foo -p -b Hello
/usr/bin/amqp-consume --url=$BROKER_URL -q foo -c 1 cat && echo 1>&2
  • We are told to run this next command on our computer, as the comment says:
  • ..but that would require lots of things not mentioned in docs on this page
  • ..and so we run it in our pod, the one we are already in right now:
    • Run this on your computer, not in the Pod
    • EDIT: like I say, we ran in pod:
/usr/bin/amqp-declare-queue --url=$BROKER_URL -q job1  -d

for f in apple banana cherry date fig grape lemon melon
do
  /usr/bin/amqp-publish --url=$BROKER_URL -r job1 -p -b $f
done
  • now we are back on our computer and will create this Docker Image with
    • Dockerfile and a
    • python script
  • shown here:
#!/usr/bin/env python

# Just prints standard out and sleeps for 10 seconds.
import sys
import time
print("Processing " + sys.stdin.readlines()[0])
time.sleep(10)
# Specify BROKER_URL and QUEUE when running
FROM ubuntu:18.04

RUN apt-get update && \
    apt-get install -y curl ca-certificates amqp-tools python \
       --no-install-recommends \
    && rm -rf /var/lib/apt/lists/*
COPY ./worker.py /worker.py

CMD  /usr/bin/amqp-consume --url=$BROKER_URL -q $QUEUE -c 1 /worker.py
  • now we make python script executable
  • build tag and push Docker image
chmod +x worker.py

docker build -t job-wq-1 .

docker tag job-wq-1 <username>/job-wq-1

docker push <username>/job-wq-1
  • ..except we use our Docker hub username
  • its a little big with ubuntu and could make smaller with a slim python
  • here is the k8s job we will run in job.yaml:
apiVersion: batch/v1
kind: Job
metadata:
  name: job-wq-1
spec:
  completions: 8
  parallelism: 2
  template:
    metadata:
      name: job-wq-1
    spec:
      containers:
      - name: c
        image: gcr.io/<project>/job-wq-1
        env:
        - name: BROKER_URL
          value: amqp://guest:guest@rabbitmq-service:5672
        - name: QUEUE
          value: job1
      restartPolicy: OnFailure

kubectl apply -f ./job.yaml

WORKS!

Tag summary

Content type

Image

Digest

sha256:dd44e66ce

Size

37.3 MB

Last updated

over 1 year ago

docker pull botanyhelp/job-wq-1:1.0.1