Unidata on the XSEDE Jetstream Cloud
100K+
It is preferable to interface with the NSF Jetstream cloud via the Atmosphere web interface. However, this web dashboard is limited in two important ways:
1024) with the exception of ports 22, 80 and 443 which are open by default.If you are in either of these scenarios, you have to interface with Jetstream via the OpenStack API. The problem here is that there is some initial setup which is somewhat painful but that you do only once. We provide a Docker container that hopefully eases some of the pain as it has the OpenStack command line tools and convenience scripts installed. After that setup, you can interface with the Jetstream OpenStack command line via the openstack.sh script which will launch a Docker container that will enable you to:
Note, you do not have to employ this Docker container. It is merely provided as a convenience. If you choose, you can go through the Jetstream OpenStack API instructions directly and ignore all that follows.
Install Docker in your computing environment because we will be interacting with the OpenStack Jetstream API via Docker. This step should make our lives easier.
We will be making heavy use of the Unidata/science-gateway git repository.
git clone https://github.com/Unidata/science-gateway
cd science-gateway/openstack
At this point, you can either pull or build the science-gateway container:
docker pull unidata/science-gateway
docker build -t unidata/science-gateway .
We will be using the Jetstream API directly and via convenience scripts.
The next part involves downloading the openrc.sh file to work with our OpenStack allocation. You will have first login to the OpenStack TACC dashboard which will necessitate a password reset. Unfortunately, this login is not the same as the Jetstream Atmosphere web interface login. Also, follow the usual password advice of not reusing passwords as this password will end up in your OpenStack environment and you may want to add it in the openrc.sh file for convenience.
Download your openrc.sh file from the IU (not TACC) dashboard at https://iu.jetstream-cloud.org and move it to the openstack/bin directory.
Project, API Access, then select Download OpenStack RC File at top-right.TG-ATM160027-openrc.sh. You should rename it to openrc.sh.bin/openrc.sh (e.g., /home/jane/science-gateway/openstack/bin/openrc.sh).Edit bin/openrc.sh Password (Optional)
For convenience, you may wish to add your password to the openrc.sh file. Again, follow the usual advice of not reusing passwords as this password will end up in your OpenStack environment.
Edit the openrc.sh file and the supply the TACC resource OS_PASSWORD you reset earlier:
export OS_PASSWORD="changeme!"
Comment out
# echo "Please enter your OpenStack Password: "
# read -sr OS_PASSWORD_INPUT
openstack.sh
Start the unidata/science-gateway container with openstack.sh convenience script. The script take a -o argument for your openrc.sh file and a -s argument for the directory containing or will contain your ssh keys (e.g., /home/jane/science-gateway/openstack/ssh or a new directory that will contain contain your Jetstream OpenStack keys that we will be creating shortly). Both arguments must be supplied with fully qualified path names.
chmod +x openstack.sh
./openstack.sh -o </path/to/your openrc.sh file> -s </path/to/your/ssh directory>
Subsequently, when interacting with Jetstream via OpenStack API now and in the future, you will be using this container to create VMs, mount volumes, etc.
A wrapper script run.sh is provided, which assumes that directories bin/ and ssh/ exist in the working directory, and that bin/ contains openrc.sh:
./run.sh
You can use this run.sh script as a template for you to parameterize, perhaps for alternative openrc.sh files.
Create ssh Keys (Do This Once)
This step of ssh key generation is important. In our experience, we have not had good luck with preexisting keys. You may have to generate a new one. Be careful with the -f argument below. We are operating under one allocation so make sure your key names do not collide with other users. Name your key something like <some short somewhat unique id>-${OS_PROJECT_NAME}-api-key. Then you add your public key the TACC dashboard with openstack keypair create.
cd ~/.ssh
ssh-keygen -b 2048 -t rsa -f <key-name> -P ""
openstack keypair create --public-key <key-name>.pub <key-name>
# go back to home directory
cd
The ssh directory was mounted from outside the Docker container you are currently running. Your public/private key should be saved there. Don't lose it or else you may not be able to delete the VMs you are about to create.
Testing Setup
At this point, you should be able to run openstack image list which should yield something like:
| ID | Name |
|---|---|
| fd4bf587-39e6-4640-b459-96471c9edb5c | AutoDock Vina Launch at Boot |
| 02217ab0-3ee0-444e-b16e-8fbdae4ed33f | AutoDock Vina with ChemBridge Data |
| b40b2ef5-23e9-4305-8372-35e891e55fc5 | BioLinux 8 |
If not, check your setup.
At this point, we are past the hard work. You will employ the unidata/science-gateway container accessed via the openstack.sh convenience script to
If you have not done so already:
./openstack.sh -o </path/to/your openrc.sh file> -s </path/to/your/ssh directory>
We are ready to fire up VMs. First create an IP number which we will be using shortly:
openstack floating ip create public
openstack floating ip list
or you can just openstack floating ip list if you have IP numbers left around from previous VMs.
Create VM
Now you can boot up a VM with something like the following command:
boot.sh -n unicloud -k <key-name> -s m1.medium -ip 149.165.157.137
The boot.sh command takes a VM name, ssh key name defined earlier, size, and IP number created earlier, and optionally an image UID which can be obtained with openstack image list | grep -i featured. Note that these feature VMs are recommended by Jetstream staff, and have a default user corresponding to the Linux distribution flavor. For example,
$ openstack image list | grep -i featured
may yield something like:
| 4ada5750-4ba4-4cc6-8d12-9001fe04ae1b | JS-API-Featured-Centos6-Feb-13-2018 |
| 87df53d5-04bd-4bb8-862e-b67247f07f87 | JS-API-Featured-Centos7-Feb-13-2018 |
| 20e6ec66-a5ec-41fc-820c-08a2af5bd1eb | JS-API-Featured-Ubuntu14-Feb-13-2018 |
| a2c80fbf-2875-457a-b488-28c4afeb296b | JS-API-Featured-Ubuntu16-Feb-13-2018 |
The CentOS VMs will have a default of user centos and the Ubuntu VMs will have a default user of ubuntu.
Also see boot.sh -h and openstack flavor list for more information.
SSH Into New VM
At this point, you can ssh into our newly minted VM. Explicitly providing the key name with the ssh -i argument and a user name (e.g., ubuntu or centos) may be important:
ssh -i ~/.ssh/<key-name> [email protected]
At this point, you might see
ssh: connect to host 149.165.157.137 port 22: No route to host
Usually waiting for a few minutes resolves the issue. If you are still have trouble, try openstack server stop <vm-uid-number> followed by openstack server start <vm-uid-number>.
Adding Additional SSH Keys (Optional)
Once you are in your VM, it is probably best to add additional ssh public keys into the authorized_keys file to make logging in easier from whatever host you are connecting from.
You can create data volumes via the OpenStack API. As an example, here, we will be creating a 750GB data volume. You will subsequently attach the data volume:
openstack volume create --size 750 data
openstack volume list && openstack server list
openstack server add volume <vm-uid-number> <volume-uid-number>
You will then be able to log in to your VM and mount your data volume with typical Unix mount, umount, and df commands. If running these command manually (not using the mount.sh script) you will need to run kfs.ext4 /dev/sdb to create an ext4 partition using the entire disk.
There is a mount.sh convenience script to mount uninitialized data volumes. Run this script as root or sudo on the newly created VM not from the OpenStack CL.
Ensure Volume Availability Upon Machine Restart
You want to ensure data volumes are available when the VM starts (for example after a reboot). To achieve this objective, you can run this command which will add an entry to the /etc/fstab file:
echo UUID=2c571c6b-c190-49bb-b13f-392e984a4f7e /data ext4 defaults 1 1 | tee \
--append /etc/fstab > /dev/null
where the UUID represents the ID of the data volume device name (e.g., /dev/sdb) which you can discover with the blkid (or ls -la /dev/disk/by-uuid) command. askubuntu has a good discussion on this topic.
Opening TCP ports on VMs must be done via OpenStack with the openstack security group command line interfaces. In addition, this can be For example, to create a security group that will enable the opening of TCP port 80:
secgroup.sh -n my-vm-ports -p 80
Once the security group is created, you can attach multiple TCP ports to that security group with openstack security group commands. For example, here we are attaching port 8080 to the global-my-vm-ports security group.
openstack security group rule create global-my-vm-ports --protocol tcp --dst-port 8080:8080 --remote-ip 0.0.0.0/0
Finally, you can attach the security group to the VM (e.g., my-vm) with:
openstack server add security group my-vm global-my-vm-ports
umount External Volumes
There is also a teardown.sh convenience script for deleting VMs. Be sure to umount any data volumes before deleting a VM. For example on the VM in question,
umount /data
You may have to verify, here, that nothing is writing to that data volume such as Docker or NFS (e.g., docker-compose stop, sudo service nfs-kernel-server stop), in case you get errors about the volume being busy.
In addition, just to be on the safe side, remove the volume from the VM via OpenStack:
openstack volume list && openstack server list
openstack server remove volume <vm-uid-number> <volume-uid-number>
Tear Down
Then finally from the OpenStack CL,
teardown.sh -n unicloud -ip 149.165.157.137
For now, you have to supply the IP number even though the script should theoretically be smart enough to figure that out.
Cloud-computing promotes the notion of the throwaway VM. We can swap in VMs that will have the same IP address and attached volume disk storage. However, before swapping out VMs, we should do a bit of homework and careful preparation so that the swap can go as smoothly as possible.
Prerequisites
Create the VM that will be swapped in. Make sure to:
git diff and scrutinize ~/configcrontab -l10.0 address changes that need to be made (e.g., NFS mounts)docker-compose files)/etc/fstab and umount
Examine /etc/fstab to find all relevant mounts on "old" VM. Copy over fstab to new host (the UUIDs should remain the same but double check). Then umount mounts.
OpenStack Swap
From the OpenStack command line, identify the VM IDs of the old and new VM as well as any attached external volume ID:
openstack volume list && openstack server list
Then swap out both the IP address as well as zero or more external data volumes with the new server.
openstack server remove floating ip ${VM_ID_OLD} ${IP}
openstack server add floating ip ${VM_ID_NEW} ${IP}
for i in ${VOLUME_IDS}
do
openstack server remove volume ${VM_ID_OLD} $i
openstack server add volume ${VM_ID_NEW} $i
done
/etc/fstab and mount
Issue blkid (or ls -la /dev/disk/by-uuid) command to find UUIDs that will be inserted into the /etc/fstab. Lastly, mount -a.
It is possible to create a Kubernetes cluster with the Docker container described here. We employ Andrea Zonca's modification of the kubespray project. Andrea's recipe to build a Kubernetes cluster on Jetstream with kubespray is described here. These instructions have been codified with the kube-setup.sh and kube-setup2.sh scripts.
Make sure to run both kubectl and helm from the client and ssh tunnel (ssh ubuntu@FLOATINGIPOFMASTER -L 6443:localhost:6443)into the master node as described in the instructions.
First, modify ~/jetstream_kubespray/inventory/zonca/cluster.tf to specify the number of nodes in the cluster and the size (flavor) of the VMs. For example,
# nodes
number_of_k8s_nodes = 0
number_of_k8s_nodes_no_floating_ip = 2
flavor_k8s_node = "4"
will create a 2 node cluster of m1.large VMs. See Andrea's instructions for more details.
This spreadsheet will help you determine the size of the cluster based on number of users, desired cpu/user, desired RAM/user. Duplicate it and adjust it for your purposes.
openstack flavor list will give the IDs of the desired VM size.
Also, note that cluster.tf assumes you are building a cluster at the TACC data center with the sections pertaining to IU commented out. If you would like to set up a cluster at IU, make the necessary modifications located at the end of cluster.tf.
IMPORTANT: once you define an image (e.g., image = JS-API-Featured-Ubuntu18-May-22-2019) or a flavor size (e.g., flavor_k8s_master = 2), make sure you do not subsequently change it after you have run Terraform and Ansible! This scenario can happen when adding cluster nodes and the featured image no longer exists because it has been updated. If you must change these values, you'll first have to preserve your application data and do a gentle - IP preserving - cluster tear down before rebuilding it and re-installing your application.
At this point, to create the VMs that will house the kubernetes cluster (named "k8s-unidata", for example) run
kube-setup.sh -n k8s-unidata
This script essentially wraps Terraform install scripts to launch the VMs according to cluster.tf.
Once, the script is complete, let the VMs settle for a while (let's say ten minutes). Behind the scenes dpkg is running on the newly created VMs which can take some time to complete.
Check Status of VMs
Check to see the status of the VMs with:
openstack server list | grep k8s-unidata
and
watch -n 15 \
ansible -i $HOME/jetstream_kubespray/inventory/k8s-unidata/hosts -m ping all
Steps if VMs are Unhappy
If the check status process did not go smoothly, here are some thing you can try to remedy the problem.
If you see any errors, you can try to wait a bit more or reboot the offending VM with:
openstack server reboot <vm>
or you can reboot all VMs with:
osl | grep k8s-unidata | awk '{print $2}' | xargs -n1 openstack server reboot
If VMs stuck in ERROR state. You may be able to fix this problem with:
cd ~/jetstream_kubespray/inventory/k8s-unidata/
CLUSTER=k8s-unidata bash -c 'sh terraform_apply.sh'
or you can destroy the VMs and try again
cd ~/jetstream_kubespray/inventory/k8s-unidata/
CLUSTER=k8s-unidata bash -c 'sh terraform_destroy.sh'
Next, run
kube-setup2.sh -n k8s-unidata
If seeing errors related to dpkg, wait and try again or try these steps.
Run kube-setup2.sh -n k8s-unidata again.
Ensure the Kubernetes cluster is running:
kubectl get pods --all-namespaces
and get a list of the nodes:
kubectl get nodes --all-namespaces
You can augment the computational capacity of your cluster by adding nodes. In theory, this is just a simple matter of adding worker nodes in jetstream_kubespray/inventory/k8s-unidata/cluster.tf followed by running:
cd ~/jetstream_kubespray/inventory/k8s-unidata/
CLUSTER=k8s-unidata bash -c 'sh terraform_apply.sh'
Wait a bit to allow dpkg to finish running on the new node(s). Check the VMS. Next:
cd ~/jetstream_kubespray
CLUSTER=k8s-unidata bash -c 'sh k8s_scale.sh'
It is also possible to remove nodes from a Kubernetes cluster. First see what nodes are running:
kubectl get nodes --all-namespaces
which will yield something like:
NAME STATUS ROLES AGE VERSION
k8s-unidata-k8s-master-1 Ready master 42h v1.12.5
k8s-unidata-k8s-node-nf-1 Ready node 42h v1.12.5
k8s-unidata-k8s-node-nf-2 Ready node 41h v1.12.5
From the Kubernetes master node:
cd ~/jetstream_kubespray
CLUSTER=k8s-unidata bash -c 'sh k8s_remove_node.sh k8s-unidata-k8s-node-nf-2'
followed by running:
teardown.sh -n k8s-unidata-k8s-node-nf-2
from the openstack command line.
Without Preserving IP of Master Node
Once you are finished with your Kubernetes cluster you can completely wipe it out (think before you type and make sure you have the cluster name correct):
cd ~/jetstream_kubespray/inventory/k8s-unidata/
CLUSTER=k8s-unidata bash -c 'sh terraform_destroy.sh'
With Preserving IP of Master Node
You can also tear down your cluster but still preserve the IP number of the master node. This is useful and important when the IP of the master node is associated with a DNS name that you wish to keep associated.
cd ~/jetstream_kubespray/inventory/k8s-unidata/
CLUSTER=k8s-unidata bash -c 'sh terraform_destroy_keep_floatingip.sh'
Subsequently, when you invoke terraform_apply.sh, the master node should have the same IP number as before.
Note: AFTER invoking terraform_apply.sh remove the ~/.ssh/known_hosts line that corresponds to the old master node! This can easily be achieved by sshing into the new master node which will indicate the offending line in ~/.ssh/known_hosts. This will avoid headaches when invoking kube-setup2.sh.
Grafana is a monitoring engine equipped with nice dashboards and fancy time-series visualizations. Prometheus allows for monitoring of Kubernetes clusters.
Installing these monitoring technologies is fairly straightforward and described here.
You'll want to keep the master node security patched as it will have a publicly accessible IP number attached to a well known DNS name. If you see packages out of date upon login, as root user:
apt-get update && apt-get -y upgrade && apt-get -y dist-upgrade \
&& apt autoremove -y
reboot -h now
| ID | Name | RAM(GB) | Disk(GB) | VCPUs |
|---|---|---|---|---|
| 1 | m1.tiny | 2 | 8 | 1 |
| 2 | m1.small | 4 | 20 | 2 |
| 3 | m1.medium | 16 | 60 | 6 |
| 4 | m1.large | 30 | 60 | 10 |
| 5 | m1.xlarge | 60 | 60 | 24 |
| 6 | m1.xxlarge | 120 | 60 | 44 |
| 14 | s1.large | 30 | 120 | 10 |
| 15 | s1.xlarge | 60 | 240 | 24 |
| 16 | s1.xxlarge | 120 | 480 | 44 |
Content type
Image
Digest
Size
450.5 MB
Last updated
over 6 years ago
docker pull unidata/xsede-jetstreamPulls:
2
Jul 6 to Jul 12