Date: Thu, 28 Mar 2024 14:53:19 +0100 (CET) Message-ID: <1578881913.2534.1711633999383@v-public003.directory.intra> Subject: Exported From Confluence MIME-Version: 1.0 Content-Type: multipart/related; boundary="----=_Part_2533_1520685252.1711633999383" ------=_Part_2533_1520685252.1711633999383 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Content-Location: file:///C:/exported.html
A Docker container is a platform to build, ship and run applications. Fu= rther information here. Using the Docker system enables to run test a= pplications on your local Linux machine and make them ready for deployment = in one external server machine or in a cloud server. In general, an applica= tion requires several sub-applications and dependencies in order to run pro= perly. In order to manage and control this work-flow and to automate proces= ses, Ansible is used. The advantages of a joint use of Ansible an= d Docker are described in this blog.
In the framework of Emodnet Chemistry 2, one of the final stages is to d= eploy the developed applications on the cloud at Cineca servers. It i= s a good practice to run test applications on the same OS as the external s= erver. We ran our test on a Debian wheezy OS (Debian 7.8), as the clo= ud server runs on a Debian wheezy machine as well.
Before shipping the application to a server, it needs to be prepared ad-= hoc. The installation of some packages is required.
Install git version control system and create a src
=
folder in your home directory. Inside src
folder, clone<=
/code> the openearth/stack repository and
checkout
"emod=
net
" branch within the stack
folder.
sudo ap= t-get install git cd ~ mkdir src cd src git clone https://github.com/openearth/stack.git cd stack git status git checkout emodnet git status
Create a virtual environment using virtualenvwrapper
. Chec=
k if you have it already with apt-cache search virtualenvwrapper. If not, install it. Create a virtual environment called
ansible=
code>, set
workon ansible
to work on ansible
virt=
ual environment and pip install it.
sudo ap= t-get install virtualenvwrapper mkvirtualenv ansible pip install 'ansible<1.7'
You might need to install Ansible from source and get some additional pa= ckages from debian distribution package manager:
sudo ap= t-get build-dep ansible sudo apt-get install python-dev
Ansible installation might require the installation of additional packag= es as:
sudo pi= p install paramiko PyYAML Jinja2 httplib2
Still from ansible virtual environment, run the playbook.yml
with the following command:
cd ~/sr= c/stack workon ansible ansible-playbook -i hosts playbook.yml -c local -K
As defined in the playbook, Docker will be installed and a Docker contai= ner will be created.
The cloud server needs to have access to our Docker applications. As we = need to run web services in docker containers, it can be useful to run a re= verse proxy in front of the containers to simplify deployment, because of t= he hassles related to the addressing of a Docker container from the host. U= sually containers are started and stopped in the server. A useful utility t= hat takes into account these events is docker-gen.
A way to do that is to let the Virtual host act as reverse proxy. A numb= er of operations need to be done by the server managers, who would need to = know:
sudo docker run <flags> <=
container name>
Currently, the ansible
tasks for the deployment of the dock=
er container for Emodnet Chemistry 2 are here. However, sometimes just a few little changes to =
the services are needed. In that case, here you find summarized 8 steps to =
tackle in order to have a successful interaction with a docker container. T=
he examples below are based on Emodnet Chemistry 2 and the server in use is=
ogs02.cineca.it, though the instructions below are valid in general.
svn Commit
functions i=
n OpenEarthTools (...\python\applications\wps\processes\ and ...\python\Ope=
nEarthTools\openearthtools\io\pyodv\ ) OR/AND in the project folder (...\12=
07195\B. Measurements and calculations\wps_processes\ and \1207195\B. Measu=
rements and calculations\docker\ ).Access the server via Putty:
Check the running containers and = the docker images available.
docker = ps # list running docker containers. For instance: CONTAINER ID IMAGE COMMAND CRE= ATED STATUS PORTS NAMES 6d04327023d0 wps_emodnet:pandas_upgrade "/bin/sh -c '/home/r 4 m= onths ago Up 4 days 127.0.0.1:8081->8081/tcp boring_h= opper
deltare= s02@ogs02:~$ docker images # list running docker images. For instance: REPOSITORY TAG IMAGE ID CREATED = VIRTUAL SIZE wps_emodnet new_pyodv_pywps4 566e2e24daec 3 months ago = 1.296 GB wps_emodnet new_pyodv_pywps3 6c2f08bb04be 3 months ago = 1.295 GB wps_emodnet new_pyodv_pywps2 f55ddb6bd287 3 months ago = 1.294 GB ...
Execute commands in the docker machine:
deltare= s02@ogs02:~$ docker exec -it 6d04327023d0 bash # execute "bash" command in = the container root@6d04327023d0:/tmp/emodnet# # The "bash" of the container is open and w= e can run commands in it.
During the previous docker run
command the VOLUME (sh=
ared filesystems) option was triggered: "-v /home/deltares02/do=
cker_images:/mnt
" (see http://docs.docker.com/refere=
nce/run/ )
Therefore we are now able to access the main machine via the docker contai=
ner, and copy files from the mounted folder to the directory where the proc=
esses run. Then exit:
root@6d= 04327023d0:/tmp/emodnet# cp /mnt/pyodv/odv2orm_query.py /root/src/wps/proc= esses/pyodv/odv2orm_query.py # copy files from mounted folder root@6d04327023d0:/tmp/emodnet# exit # exit docker container
Commit the docker container by saving the image:
Check again what docker containers are running by typing docker ps=
code> and the saved images by typing
docker images
. Then commi=
t the changes. A long file name is returned as output:
deltare= s02@ogs02:~$ docker commit 6d04327023d0 wps_emodnet:new_pyodv_pywps5 f31658a70624ce1c75c7fcf43689ffeab64fa1ed6d1b34c3c513e59f80bbe698
After that we can check if our new image is there:
deltare= s02@ogs02:~$ docker images REPOSITORY TAG IMAGE ID CREATED = VIRTUAL SIZE wps_emodnet new_pyodv_pywps5 f31658a70624 40 seconds ago = 1.252 GB wps_emodnet new_pyodv_pywps4 566e2e24daec 3 months ago = 1.296 GB ...
Stop the former docker container and run the newly created image:
deltare= s02@ogs02:~$ docker stop 6d04327023d0 6d04327023d0
Now, typing docker ps
again will give an empty list of runnin=
g containers. Let's docker run
the newly created image:
docker = run -p 8081:8081 -e "PYWPS_CFG=3D/root/pywps.cfg" -e "PYWPS_PROCESSES=3D/ro= ot/src/wps/processes" -v /home/deltares02/docker_images:/mnt --restart=3Da= lways f31658a70624 [2015-09-03 16:12:05 +0000] [9] [INFO] Starting gunicorn 19.3.0 [2015-09-03 16:12:05 +0000] [9] [INFO] Listening at: http://0.0.0.0:8081 (9= ) [2015-09-03 16:12:05 +0000] [9] [INFO] Using worker: sync [2015-09-03 16:12:05 +0000] [14] [INFO] Booting worker with pid: 14
Testing:
Test the processes on the Web server via the brower. For instance:
<= a rel=3D"nofollow">http://emodnet02.cineca.it/wps?service=3Dwps&request= =3DExecute&Identifier=3Dbbox_plot_timeprofile&DataInputs=3D[bbox=3D= 18.8,19.2,57.2,57.8;parameter=3DEPC00007;z=3DADEPZZ01;zlim0=3D0;zlim1=3D100= ;starttime=3D2000-01-01T00:00:00Z;endtime=3D2002-01-01T00:00:00Z;markersize= =3D13.0;marker=3Do;alpha=3D0.5]&version=3D1.0.0
If the result is good, you can exit Putty window. If = you open Putty again and access ogs02.cineca.it, you will get the new runni= ng container:
deltare= s02@ogs02:~$ docker ps CONTAINER ID =20 IMAGE COMMAND =20 CREATED STATUS PORTS NAMES 8ea44d2ce787 wps_emodnet:new_pyodv_pywps5 "/bin/sh -c '/home/r 6= minutes ago Up 6 minutes 127.0.0.1:8081->8081/tcp sleepy= _mestorf
Save the image (and load it in a new machine):
Saving the Docker image is not always necessary but it's good practice. Sa=
ving the Docker image allows the loading and deployment on another machine.=
Save the image as a tarball, zip the image, and copy it on a new machine:<=
/p>
deltare= s02@ogs02:~$ docker save wps_emodnet:new_pyodv_pywps5 > /home/deltares02= /docker_images/wps_emodnet_new_pyodv_pywps5.tar # Save as tarball. This mig= ht take some time if the image is large deltares02@ogs02:~$ gzip /home/deltares02/docker_images/wps_emodnet_new_pyo= dv_pywps5.tar # Zip the image. This can also take some time
The Docker image can now be copied to a new machine, loaded by unz= ipping the file, then "docker load" the image, find it and tag it:
deltare= s02@ogs02:~$ scp wps_emodnet_new_pyodv_pywps5.tar.gz <foreign machine>= ; # Copy the tarball to the new machine. $ gunzip wps_emodnet_new_pyodv_pywps5.tar.gz # Unzip The tarball $ docker load < wps_emodnet_new_pyodv_pywps5.tar # Load the docker image $ docker images # Find the Image ID $ docker tag repositoryname:tag # Tag the image