Child pages
  • Setting up pyWPS using a Docker container
Skip to end of metadata
Go to start of metadata

Introduction

A Docker container is a platform to build, ship and run applications. Further information here. Using the Docker system enables to run test applications on your local Linux machine and make them ready for deployment in one external server machine or in a cloud server. In general, an application requires several sub-applications and dependencies in order to run properly. In order to manage and control this work-flow and to automate processes, Ansible is used. The advantages of a joint use of Ansible and Docker are described in this blog.

In the framework of Emodnet Chemistry 2, one of the final stages is to deploy the developed applications on the cloud at Cineca servers. It is a good practice to run test applications on the same OS as the external server. We ran our test on a Debian wheezy OS (Debian 7.8), as the cloud server runs on a Debian wheezy machine as well.

Preparation

Before shipping the application to a server, it needs to be prepared ad-hoc. The installation of some packages is required.

Install git version control system and create a src folder in your home directory. Inside src folder, clone the openearth/stack repository and checkout "emodnet" branch within the stack folder.

sudo apt-get install git
cd ~
mkdir src
cd src
git clone https://github.com/openearth/stack.git

cd stack
git status
git checkout emodnet
git status

Create a virtual environment using virtualenvwrapper . Check if you have it already with apt-cache search virtualenvwrapper. If not, install it. Create a virtual environment called ansible, set workon ansible to work on ansible virtual environment and pip install it.

sudo apt-get install virtualenvwrapper
mkvirtualenv ansible
pip install 'ansible<1.7'

You might need to install Ansible from source and get some additional packages from debian distribution package manager:

sudo apt-get build-dep ansible
sudo apt-get install python-dev

Ansible installation might require the installation of additional packages as:

sudo pip install paramiko PyYAML Jinja2 httplib2

Still from ansible virtual environment, run the playbook.yml with the following command:

cd ~/src/stack
workon ansible
ansible-playbook -i hosts playbook.yml -c local -K

As defined in the playbook, Docker will be installed and a Docker container will be created.

Shipping

The cloud server needs to have access to our Docker applications. As we need to run web services in docker containers, it can be useful to run a reverse proxy in front of the containers to simplify deployment, because of the hassles related to the addressing of a Docker container from the host. Usually containers are started and stopped in the server. A useful utility that takes into account these events is docker-gen.

A way to do that is to let the Virtual host act as reverse proxy. A number of operations need to be done by the server managers, who would need to know:

  • which services need to be deployed
  • which ports need to be open
  • what is the proxy configuration
  • the list of Docker commands as sudo docker run <flags> <container name>

Interaction

Currently, the ansible tasks for the deployment of the docker container for Emodnet Chemistry 2 are here. However, sometimes just a few little changes to the services are needed. In that case, here you find summarized 8 steps to tackle in order to have a successful interaction with a docker container. The examples below are based on Emodnet Chemistry 2 and the server in use is ogs02.cineca.it, though the instructions below are valid in general.

  1. Work on local or Project folder:
    svn Commit functions in OpenEarthTools (...\python\applications\wps\processes\ and ...\python\OpenEarthTools\openearthtools\io\pyodv\ ) OR/AND in the project folder (...\1207195\B. Measurements and calculations\wps_processes\ and \1207195\B. Measurements and calculations\docker\ ).

  2. Access the server via WinSCP:
    Copy the changes from local (or Project folder) to /home/deltares02/docker_images/ in the cloud server (ogs02.cineca.it).

  3. Access the server via Putty:

    Check the running containers and the docker images available.

    docker ps # list running docker containers. For instance:
    CONTAINER ID        IMAGE                        COMMAND                CREATED             STATUS              PORTS                      NAMES
    6d04327023d0        wps_emodnet:pandas_upgrade   "/bin/sh -c '/home/r   4 months ago        Up 4 days           127.0.0.1:8081->8081/tcp   boring_hopper
    deltares02@ogs02:~$ docker images # list running docker images. For instance:
    REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
    wps_emodnet         new_pyodv_pywps4    566e2e24daec        3 months ago        1.296 GB
    wps_emodnet         new_pyodv_pywps3    6c2f08bb04be        3 months ago        1.295 GB
    wps_emodnet         new_pyodv_pywps2    f55ddb6bd287        3 months ago        1.294 GB
    ...
  4. Execute commands in the docker machine:

    deltares02@ogs02:~$ docker exec -it 6d04327023d0 bash # execute "bash" command in the container
    root@6d04327023d0:/tmp/emodnet# # The "bash" of the container is open and we can run commands in it.

    During the previous docker run command the VOLUME (shared filesystems) option  was triggered: "-v /home/deltares02/docker_images:/mnt" (see http://docs.docker.com/reference/run/ )
    Therefore we are now able to access the main machine via the docker container, and copy files from the mounted folder to the directory where the processes run. Then exit:

    root@6d04327023d0:/tmp/emodnet# cp /mnt/pyodv/odv2orm_query.py  /root/src/wps/processes/pyodv/odv2orm_query.py # copy files from mounted folder
    root@6d04327023d0:/tmp/emodnet# exit # exit docker container
  5. Commit the docker container by saving the image:
    Check again what docker containers are running by typing docker ps and the saved images by typing docker images. Then commit the changes. A long file name is returned as output:

    deltares02@ogs02:~$ docker commit 6d04327023d0 wps_emodnet:new_pyodv_pywps5
    f31658a70624ce1c75c7fcf43689ffeab64fa1ed6d1b34c3c513e59f80bbe698


    After that we can check if our new image is there:

    deltares02@ogs02:~$ docker images
    REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
    wps_emodnet         new_pyodv_pywps5    f31658a70624        40 seconds ago      1.252 GB
    wps_emodnet         new_pyodv_pywps4    566e2e24daec        3 months ago        1.296 GB
    ...
  6. Stop the former docker container and run the newly created image:

    deltares02@ogs02:~$ docker stop 6d04327023d0
    6d04327023d0


    Now, typing docker ps again will give an empty list of running containers. Let's docker run the newly created image:

    docker run -p 8081:8081 -e "PYWPS_CFG=/root/pywps.cfg" -e "PYWPS_PROCESSES=/root/src/wps/processes" -v /home/deltares02/docker_images:/mnt  --restart=always f31658a70624
    
    [2015-09-03 16:12:05 +0000] [9] [INFO] Starting gunicorn 19.3.0
    [2015-09-03 16:12:05 +0000] [9] [INFO] Listening at: http://0.0.0.0:8081 (9)
    [2015-09-03 16:12:05 +0000] [9] [INFO] Using worker: sync
    [2015-09-03 16:12:05 +0000] [14] [INFO] Booting worker with pid: 14
    
    
  7. Testing:
    Test the processes on the Web server via the brower. For instance:

    http://emodnet02.cineca.it/wps?service=wps&request=Execute&Identifier=bbox_plot_timeprofile&DataInputs=[bbox=18.8,19.2,57.2,57.8;parameter=EPC00007;z=ADEPZZ01;zlim0=0;zlim1=100;starttime=2000-01-01T00:00:00Z;endtime=2002-01-01T00:00:00Z;markersize=13.0;marker=o;alpha=0.5]&version=1.0.0

    http://emodnet02.cineca.it/wps?DataInputs=[z=ADEPZZ01;bbox=20.0225341796875,20.0774658203125,57.305864179687504,57.360795820312504;parameter=EPC00004;starttime=2000-01-01T00:00:00Z;endtime=2000-12-31T00:00:00Z;zlim0=0;zlim1=10;log10=0;markersize=12.0;alpha=1]&service=wps&request=Execute&Identifier=bbox_plot_timeseries&version=1.0.0

    If the result is good, you can exit Putty window. If you open Putty again and access ogs02.cineca.it, you will get the new running container:

    deltares02@ogs02:~$ docker ps
    CONTAINER ID        
    IMAGE                          COMMAND                
    CREATED             STATUS              PORTS                      NAMES
    8ea44d2ce787        wps_emodnet:new_pyodv_pywps5   "/bin/sh -c '/home/r   6 minutes ago       Up 6 minutes        127.0.0.1:8081->8081/tcp   sleepy_mestorf
  8. Save the image (and load it in a new machine):
    Saving the Docker image is not always necessary but it's good practice. Saving the Docker image allows the loading and deployment on another machine.
    Save the image as a tarball, zip the image, and copy it on a new machine:

    deltares02@ogs02:~$ docker save wps_emodnet:new_pyodv_pywps5 > /home/deltares02/docker_images/wps_emodnet_new_pyodv_pywps5.tar # Save as tarball. This might take some time if the image is large
    deltares02@ogs02:~$ gzip /home/deltares02/docker_images/wps_emodnet_new_pyodv_pywps5.tar # Zip the image. This can also take some time

    The Docker image can now be copied to a new machine, loaded by unzipping the file, then "docker load" the image, find it and tag it:

    deltares02@ogs02:~$ scp wps_emodnet_new_pyodv_pywps5.tar.gz <foreign machine> # Copy the tarball to the new machine.
    $ gunzip wps_emodnet_new_pyodv_pywps5.tar.gz # Unzip The tarball
    $ docker load < wps_emodnet_new_pyodv_pywps5.tar # Load the docker image
    $ docker images # Find the Image ID
    $ docker tag repositoryname:tag # Tag the image
  • No labels