Contents


The OpenEarth infrastructure supports a bottom-up approach for long-term project-transcending collaboration adhering to four basic criteria:

To make the OpenEarth infrastructure work, the following ICT components are required in practice(see figure):

  1. a linux server (either virtual or real),
  2. a subversion server to enable version control, backup and controlled access,
  3. a tomcat6 server running an OPeNDAP protocol to provide netCDF files,
  4. an apache2 webserver to provide data visualisation files like *.kml for visualisation in Google Earth, and
  5. the OpenSSH protocol to facilitate remote access to the server (for maintenance purposes).

The remainder of this tutorial addresses the steps needed to set up this infrastructure.

Create a virtual machine

Register at VMWARE and download "VMWARE Server" at https://www.vmware.com/. Make sure you write down the Windows code as it is needed to install VMWare Server. Once VMWare Server is installed (login with administrator rights if needed) open it and create a virtual machine:

Install LINUX operating system

Download ubuntu Desktop version 9.04 from http://www.ubuntu.com/GetUbuntu/download and install it (the download is an ISO-file). Make a map ISO in "d:\Virtual Machines" and copy the UBUNTU ISO image there. In VM Ware select the virtual server you created and select in the hardware list the CDROM/DVD option. Choose edit. Under the section "ISO Image" choose browse and browse to the UBUNTU ISO-image ("standard" then "ISO" then select "Ubuntu") you just copied. Now go to the console and start up the virtual machine (click the screen as indicated, start the machine by pressing the big play button that appears). Now VMWare will open with the virtual machine with the Ubuntu desktop. Select the Install icon. TIP: Your computer may require installation of a remote console plugin first.

Now the Ubuntu operating system will install (select all settings to your liking: keyboardtype, timezone etc.). After it is finished you need to reboot the virtual machine. TIP: UBUNTU will request to do this automatically, otherwise use the powerbutton in the upper right corner of your virtual desktop. If your virtual machine does not start up properly a possible cause could be that the ISO image is still selected in the CDROM/DVD drive (see previous steps). Unselecting it is likely to fix this problem. The virtual machine will indicate that the virtual CDRom drive is locked (?!?). You can select to override. You will be requested to create a useraccount and a password, select something useful like 'admin-oet' or so. TIP: write down your username and password, you will need them later!

Once the virtual machine is created select it:

cd Desktop/vmware-tools-distrib/
sudo ./vmware-install.pl

TIP: Select for all options the indicated default value (NB: some steps take some time to complete ... be patient!). When prompted for display size option [12] is recommended (1024x768).

Ubuntu will suggest to install security updates. You can install them if you want to (if your internet connection is slow maybe opt to do this later).

Install Subversion and create repositories

To setup the subversion service on your virtual machine, first install subversion using the synaptic package manager (see the section Overview of useful commands for tips on locating the synaptic package manager). You may need to reload for subversion to appear in the package list. Select "subversion" and "subversion-tools" and press 'Apply' to actually install these packages. Once subversion package is installed the instructions on

https://help.ubuntu.com/community/Subversion

can be followed to set up subversion repositories.

A user 'subversion' needs to be added by typing the following command in a terminal window:

sudo adduser subversion

A directory can be created where new repositories can be put:

sudo mkdir /home/svn
cd /home/svn

Tip: you will probably want to select a more appropriate directory for actual applications (something like /var/data/repos/). The appropriate location depends on a companies local preferences.

A new repository is created by issuing the following commands:

sudo mkdir myproject
sudo chown -R www-data:subversion myproject
sudo chmod -R g+rws myproject
sudo svnadmin create /home/svn/myproject
sudo chmod -R g+rws myproject

In each repository directory uncomment the password keyword in the svn config file "svnserve.conf" and add usernames and passwords to the passwordfile "passwd". Set the authentication settings to your preference: e.g. anon-access = none. Tip: Use 'sudo gedit' to start an editor program. You need to run gedit as sudo to allow filesaving.

To start the svnservice and make it remotely accessible start the following svnserve with the following command:

sudo svnserve -d --foreground -r /home/svn

or

sudo -u www-data svnserve -d --foreground -r /home/svn

NB: running svnserve in the foreground the cursor will not reappear (until the service is stopped with crtl-z) so don't wait for it to reappear. If you want to execute another command start a new terminal window. If no error messages appear subversion is running and you can now remotely access the repositories, e.g. with a program like TortoiseSVN.

Besides starting svnserve manually and running it on the foreground, it is also possible to make svnserve start automatically on bootup running it in the background. Follow the instructions on:

http://odyniec.net/articles/ubuntu-subversion-server/

under the section "Svnserve Initialisation Script". It basically comes down to copying the info from:

http://odyniec.net/articles/ubuntu-subversion-server/svnserve

to a file (e.g. using gedit) and saving it as "/etc/init.d/svnserve". Make sure the info in svnserve is edited to include the directory you selected for the location of repositories. The script is installed by running:

sudo update-rc.d svnserve defaults

Indicate your repository locations (e.g. /var/data/repos/) in the svnserve script:

DAEMON_ARGS="-d -r /var/data/repos"

To make sure the svnserve script is executed on bootup it needs to be made executable by:

sudo chmod +x /etc/init.d/svnserve

Subversion should now work and you should be able to commit your first files. For tips see the five easy steps to becoming an OpenEarth developer. The location of your repository will be svn://[ip-address][repositoryname]. You can find the IP address of your virtual machine under the summary tab of VMWare. The repository name you selected yourself in one of the previous steps.

Install Tomcat and Thredds (OpenDap server for netCDF files)

With the Synaptic package manager install Tomcat6 (identify all Tomcat6 packages by inserting 'Tomcat6*' in the search box). Select them and press Apply to install. Once the packages are installed you can start the Tomcat6 server by issuing the following command:

sudo /etc/init.d/tomcat6 start

or

sudo /etc/init.d/tomcat6 restart

Under "/var/lib/tomcat6/" create a directory "content" and set the proper groups and rights:

cd /var/lib/tomcat6/
sudo mkdir content
sudo chown tomcat6:tomcat6 content

sudo /etc/init.d/tomcat6 restart

Now that your Tomcat6 service is running you can install Thredds via a webinterface. However, Thredds will want to write logfiles. The standard security settings on your virtual machine are too strict. An adjusted .policy file should be added in "/etc/tomcat6/policy.d/" with the following content:

grant codeBase "file:${catalina.base}/-" {
permission java.security.AllPermission;
};

Name it either 00openpolicy.policy or 51openpolicy.policy so that it is run either first or last (this is indicated by the two digits in the name). Apparently this works. NB: A number that is not first or last still results in problems.

Now download Thredds and mount the file 'thredds.war' file (NB: file should have this exact name) by accessing http://\[ip-number\]:8080/manager/html, where [ip-number] is the ip-number of your virtual server. Once you deployed the war file a new option will appear in the Tomcat6 management webpage. Select 'start' to start the Thredds service.

Edit in "/var/lib/tomcat6/content/thredds" the file "tomcat-users.xml". Create a role admin and manager in the config.xml and create a user (e.g. admin) and assign it both roles. For more detail see:

https://help.ubuntu.com/8.10/serverguide/C/tomcat.html

Now you can configure the data to be served with THREDDS. There are some
manuals on the web ([Signell (USGS)|http://rsignell.tiddlyspot.com/#[A%20method%20for%20setting%20up%20TDS%20catalogs%20for%20the%20first%20time]], Unidata). Edit in "/var/lib/tomcat6/content/thredds" the file "catalog.xml". Place data at:

/var/data/

Set datasetRoot, datasetScan to the proper data location and indicate the names you wish to appear on the Thredds server. NB: no spaces are allowed in directory names. Change the wildcard to "*.nc".

For restricted datasets (make internal data available for remote access for your employees only) configure tomcat accordingly: http://www.unidata.ucar.edu/projects/THREDDS/tech/reference/RestrictedAccess.html. We advice to restrict access based on the http rather than based on for instance using ownership properties of the data on the underlying (linux) file server. Note that for remote access the netCDF java library cannot handle http authentication/encryption itself ([http://user:password@opendap.institution.nl]...).

Every now and then THREDDS needs to be updated. Check the Unidata THREDDS page for posts. Future releases can be anticipated in the THREDDS issue tracking system https://www.unidata.ucar.edu/jira/browse/TDS. To deploy a new THREDDS with only a few minutes downtime, do the following. Download the new war thredds.war (release 4.2.10 in this example) and prepare it ready for instant copying to the Tomcat webapps directory /var/lib/tomcat6/content/thredds/. Please read this check-list for upgrading.

sudo rename thredds.war thredds.war.4.2.9
sudo cp thredds.war.4.2.10 /var/lib/tomcat6/webapps/
cd /var/lib/tomcat6/content/webapps/
cp thredds.war thredds.war.old

NB You are not allowed to call it thredds.old.war as any file with trailing *.war will be unpacked by Tomcat. We advocate the labeling thredds.war.old and thredds.war.4.2.10 to prevent mistakes with releases. After this preparation do the actual re-deployment.

cd /var/lib/tomcat6/content/webapps/
sudo /etc/init.d/tomcat6 stop
sudo rm -rf thredds
cp thredds.war.4.2.10 thredds.war
sudo /etc/init.d/tomcat6 restart

Install Apache webserver (webserver for kml files)

In synaptic package manager select:

Create a dir, e.g. kml, in var/data/. Open /etc/apache2/sites-enabled/000-default in a text editor and make sure you point to it as /var/data/kml. This is described by Apache's documentroot documentation. You can also point to files in another directory than /var/data. This is described by Apache's Files Outside the DocumentRoot documentation. On our test server dtvirt5 we made the kml folder a subfolder of the data directory for THREDDS, such that this data folder is the only folder where we have to give write access to the group of data-administrators.

...
    DocumentRoot /var/www

    RewriteEngine on
    # /kml
    Alias /kml /data/kml
    <Directory /data/kml>
        Order allow,deny
        Allow from all
    </Directory>
...

In both cases make sure the DocumentRoot contains a representative index.html file that contains links to the SubVersion, OPeNDAP, and kml services. You can optionally lay-out the index.html with some logos, e.g. http://http://dtvirt5.deltares.nl/. You can also delete the default "It works!" index.html page, such that the subfolder kml wil appear (unless you relocated the kml folder as described above).

Restart apache service to apply settings

sudo /etc/init.d/apache2 restart

Now open /etc/apache2/sites-enabled/000-default and add (just after documentroot):

AddType application/vnd.google-earth.kml+xml .kml
AddType application/vnd.google-earth.kmz .kmz

Install OpenSSH to enable use of WinSCP

To remotely manage the virtual server install WinSCP. For WinSCP to work you must install OpenSSH-server with the synaptic package manager. For the data directory to be accessible by WinSCP make sure to edit the rights:

sudo chmod -R og+w /var/data
sudo chmod +s /var/data

Overview of useful Linux commands

Useful links

Apache

http://httpd.apache.org/docs/2.2/howto/auth.html Apache authentication and authorization documentation
http://httpd.apache.org/docs/2.2/mod/mod_dav.html Webdav
http://tomcat.apache.org/tomcat-6.0-doc/index.html Tomcat documentation

Filesystems

http://www.drbd.org/ Drbd

NetCDF

http://www.unidata.ucar.edu/software/netcdf/docs/netcdf-install/ NetCDF downloads

Matlab

http://mexcdf.sourceforge.net/ Matlab netcdf library

Python

http://www.scipy.org/ Python numerical package
http://code.google.com/p/netcdf4-python/ Netcdf python software

Conventions

http://www.inspire-geoportal.eu/InspireEditor/ Inspire editor
http://www.gfdl.noaa.gov/~vb/grids/gridspec-tools.html Gridspec

OpenDAP

https://help.ubuntu.com/8.10/serverguide/C/tomcat.html Install tomcat under ubuntu
http://opendap.org/download/hyrax.html Hyrax server download
http://docs.opendap.org/index.php/Hyrax Hyrax documentation
http://www.unidata.ucar.edu/projects/THREDDS/ Thredds server

References

http://en.wikipedia.org/wiki/DMZ_%28computing%29 DMZ information
http://en.wikipedia.org/wiki/Secure_copy Secure copy

GUI

http://www.epic.noaa.gov/java/ncBrowse/ NcBrowse
http://www.mpimet.mpg.de/fileadmin/software/cdo/ CDO

Tools

http://nco.sourceforge.net/ NCO
http://www.unidata.ucar.edu/software/idv/ IDV
http://www.unidata.ucar.edu/downloads/netcdf/netcdf-java-4/index.jsp NetCDF-UI
http://publicwiki.deltares.nl/display/OET/OpenEarth OpenEarthTools

Subversion

http://subversion.tigris.org/ Subversion
https://help.ubuntu.com/community/Subversion Subversion setup on ubuntu


The main authors of this article are Fedor Baart and Mark van Koningsveld.
Please note that others may also have edited the contents of this article: