Introduction
A Docker container is a platform to build, ship and run applications. Further information here. Using the Docker system enables to run test applications on your local Linux machine and make them ready for deployment in one external server machine or in a cloud server. In general, an application requires several sub-applications and dependencies in order to run properly. In order to manage and control this work-flow and to automate processes, Ansible is used. The advantages of a joint use of Ansible and Docker are described in this blog.
In the framework of Emodnet Chemistry 2, one of the final stages is to deploy the developed applications on the cloud at Cineca servers. It is a good practice to run test applications on the same OS as the external server. We ran our test on a Debian wheezy OS (Debian 7.8), as the cloud server runs on a Debian wheezy machine as well.
Preparation
Before shipping the application to a server, it needs to be prepared ad-hoc. The installation of some packages is required.
Install git version control system and create a src folder in your home directory. Inside src folder, clone the openearth/stack repository and checkout "emodnet" branch within the stack folder.
sudo apt-get install git cd ~ mkdir src cd src git clone https://github.com/openearth/stack.git cd stack git status git checkout emodnet git status
Create a virtual environment using virtualenvwrapper . Check if you have it already with apt-cache search virtualenvwrapper. If not, install it. Create a virtual environment called ansible, set workon ansible to work on ansible virtual environment and pip install it.
sudo apt-get install virtualenvwrapper mkvirtualenv ansible pip install 'ansible<1.7'
You might need to install Ansible from source and get some additional packages from debian distribution package manager:
sudo apt-get build-dep ansible sudo apt-get install python-dev
Ansible installation might require the installation of additional packages as:
sudo pip install paramiko PyYAML Jinja2 httplib2
Still from ansible virtual environment, run the playbook.yml with the following command:
cd ~/src/stack workon ansible ansible-playbook -i hosts playbook.yml -c local -K
As defined in the playbook, Docker will be installed and a Docker container will be created.
Shipping
The cloud server needs to have access to our Docker applications. As we need to run web services in docker containers, it can be useful to run a reverse proxy in front of the containers to simplify deployment, because of the hassles related to the addressing of a Docker container from the host. Usually containers are started and stopped in the server. A useful utility that takes into account these events is docker-gen.
A way to do that is to let the Virtual host act as reverse proxy. A number of operations need to be done by the server managers, who would need to know:
- which services need to be deployed
- which ports need to be open
- what is the proxy configuration
- the list of Docker commands as
sudo docker run <flags> <container name>
Interaction
Currently, the ansible tasks for the deployment of the docker container for Emodnet Chemistry 2 are here. However, sometimes just a few little changes to the services are needed. In that case, here you find summarized 8 steps to tackle in order to have a successful interaction with a docker container. The examples below are based on Emodnet Chemistry 2 and the server in use is ogs02.cineca.it, though the instructions below are valid in general.
- Work on local or Project folder:
svn Commitfunctions in OpenEarthTools (...\python\applications\wps\processes\ and ...\python\OpenEarthTools\openearthtools\io\pyodv\ ) OR/AND in the project folder (...\1207195\B. Measurements and calculations\wps_processes\ and \1207195\B. Measurements and calculations\docker\ ). - Access the server via WinSCP:
Copy the changes from local (or Project folder) to /home/deltares02/docker_images/ in the cloud server (ogs02.cineca.it). Access the server via Putty:
Check the running containers and the docker images available.
docker ps # list running docker containers. For instance: CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 6d04327023d0 wps_emodnet:pandas_upgrade "/bin/sh -c '/home/r 4 months ago Up 4 days 127.0.0.1:8081->8081/tcp boring_hopper
deltares02@ogs02:~$ docker images # list running docker images. For instance: REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE wps_emodnet new_pyodv_pywps4 566e2e24daec 3 months ago 1.296 GB wps_emodnet new_pyodv_pywps3 6c2f08bb04be 3 months ago 1.295 GB wps_emodnet new_pyodv_pywps2 f55ddb6bd287 3 months ago 1.294 GB ...
Execute commands in the docker machine:
deltares02@ogs02:~$ docker exec -it 6d04327023d0 bash # execute "bash" command in the container root@6d04327023d0:/tmp/emodnet# # The "bash" of the container is open and we can run commands in it.
During the previous
docker runcommand the VOLUME (shared filesystems) option was triggered: "-v /home/deltares02/docker_images:/mnt" (see http://docs.docker.com/reference/run/ )
Therefore we are now able to access the main machine via the docker container, and copy files from the mounted folder to the directory where the processes run. Then exit:root@6d04327023d0:/tmp/emodnet# cp /mnt/pyodv/odv2orm_query.py /root/src/wps/processes/pyodv/odv2orm_query.py # copy files from mounted folder root@6d04327023d0:/tmp/emodnet# exit # exit docker container
Commit the docker container by saving the image:
Check again what docker containers are running by typingdocker psand the saved images by typingdocker images. Then commit the changes. A long file name is returned as output:deltares02@ogs02:~$ docker commit 6d04327023d0 wps_emodnet:new_pyodv_pywps5 f31658a70624ce1c75c7fcf43689ffeab64fa1ed6d1b34c3c513e59f80bbe698
After that we can check if our new image is there:deltares02@ogs02:~$ docker images REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE wps_emodnet new_pyodv_pywps5 f31658a70624 40 seconds ago 1.252 GB wps_emodnet new_pyodv_pywps4 566e2e24daec 3 months ago 1.296 GB ...
Stop the former docker container and run the newly created image:
deltares02@ogs02:~$ docker stop 6d04327023d0 6d04327023d0
Now, typingdocker psagain will give an empty list of running containers. Let'sdocker runthe newly created image:docker run -p 8081:8081 -e "PYWPS_CFG=/root/pywps.cfg" -e "PYWPS_PROCESSES=/root/src/wps/processes" -v /home/deltares02/docker_images:/mnt --restart=always f31658a70624 [2015-09-03 16:12:05 +0000] [9] [INFO] Starting gunicorn 19.3.0 [2015-09-03 16:12:05 +0000] [9] [INFO] Listening at: http://0.0.0.0:8081 (9) [2015-09-03 16:12:05 +0000] [9] [INFO] Using worker: sync [2015-09-03 16:12:05 +0000] [14] [INFO] Booting worker with pid: 14
Testing:
Test the processes on the Web server via the brower. For instance:If the result is good, you can exit Putty window. If you open Putty again and access ogs02.cineca.it, you will get the new running container:
deltares02@ogs02:~$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 8ea44d2ce787 wps_emodnet:new_pyodv_pywps5 "/bin/sh -c '/home/r 6 minutes ago Up 6 minutes 127.0.0.1:8081->8081/tcp sleepy_mestorf
Save the image (and load it in a new machine):
Saving the Docker image is not always necessary but it's good practice. Saving the Docker image allows the loading and deployment on another machine.
Save the image as a tarball, zip the image, and copy it on a new machine:deltares02@ogs02:~$ docker save wps_emodnet:new_pyodv_pywps5 > /home/deltares02/docker_images/wps_emodnet_new_pyodv_pywps5.tar # Save as tarball. This might take some time if the image is large deltares02@ogs02:~$ gzip /home/deltares02/docker_images/wps_emodnet_new_pyodv_pywps5.tar # Zip the image. This can also take some time
The Docker image can now be copied to a new machine, loaded by unzipping the file, then "docker load" the image, find it and tag it:
deltares02@ogs02:~$ scp wps_emodnet_new_pyodv_pywps5.tar.gz <foreign machine> # Copy the tarball to the new machine. $ gunzip wps_emodnet_new_pyodv_pywps5.tar.gz # Unzip The tarball $ docker load < wps_emodnet_new_pyodv_pywps5.tar # Load the docker image $ docker images # Find the Image ID $ docker tag repositoryname:tag # Tag the image
