Data

The OpenEarth philosophy aims to collect and disseminate environmental and lab data sets in a project-superseding manner rather than on a project-by-project basis. We believe that science and engineering have become so data-intensive that data management is beyond the capabilities of individual researchers. Data management needs to migrate from artisanal methods to 21st century technology. This implies data management needs to team up with IT-professionals, and v.v.. This belief is wide-spread, and is called the 4th paradigm. We recommend to read the 4th paradigm book. It illustrates the spreading belief that all sustainable solutions to manage data should be web-based and involve communities. OpenEarth aims to be a 4th paradigm workflow solution to let scientist and engineers collaborate in communities over the web. The need for teaming up science and IT is clearly illustrated in the Nature article. At the bottom of this page you can see a movie of the community activity in our raw data repository, a tool we got from the IT-world. Such communities should not only deal with data, but deal with numerical models and analysis tools as well. Data cannot be treated separately from the rest of science. Therefore OpenEarth aims to be an integral workflow for data, models and tools. For hosting such a the workflow we advocate collaboration with professional data centres such as 3TU datacentre, DANS and Pangea. Some data centers are member of DataCite, and can give you DOI for published data under conditions, enabling anyone to cite your web-based data.

To be an effective and sustainable 4th paradigm solution, OpenEarth has identified the most promising international standards for exchange of data over the web. These standards come from different realms. These standards are shown in the scheme below. We aim to work with all of these standards, but currently only use the bold ones on a daily basis. These include subversion to store not only the raw data, but also the processing software (scripts, settings) under version control using the web 2.0 Wikipedia approach: everyone can sign up for write access. This allows us to naturally attribute versions to data, an aspect that lacks in most of today data management solutions (known as provenance). For standardized data we use the netCDF format (NASA and OGC standard). With the CF vocabularies and EPSG codes^1,2 this becomes a very powerful data stack as described in an OceanObs'09 paper. We place the netCDf files on a THREDDS OPeNDAP server for dissemination of TBs of netCDF data over the web. OPeNDAP is available in many user software applications. It is for instance built-in for MATLAB since 2012, and it is optionally available for the R, python, ArcGis and many other netCDF programs^3,4.

For ecological data, which have an overwhelming amount of meta-data, we use a plain-vanilla Relational DataBase Manegement System (RDBMS). We chose the powerful, open source PostgreSQL implementation with PostGIS spatio-temporal add-on. We are working on adopting dedicated spatio-temporal standard as well. These standard allow for live server-side processing on the data to meet the demands of the user. They deliver tailored data. The OGC consortium is the international body for specifications of these standards. The EU INSPIRE directive prescribes these standards. For typical GIS data (flat, 2D or 2.5D) we already work with postgis, geoserver and geonetwork. However, these so-called WxS protocols still lack implementation in operational software for many specific demands of time-dependent, 3D, curvi-linear data products in our field. We do not develop WxS software ourselves, but just wait for the open source implementations, most under OSGeo umbrella, to cover the demands of our our field. By far the most promising WxS client and server implementation we indentified is ADAGUC by the Dutch met office KNMI. ADAGUC not only implemented the WCS standard to request data over the web very fast, but also the WMS standard to request imagery. For exchange of graphics of data, we chose to start working with the KML standards, the standard behind Google Earth that was also adopted as standard by OGC, but we will adopt WMS as well.