Include Page | ||||
---|---|---|---|---|
|
Section | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
|
Figure: the open standards (ISO or OGC) that OpenEarth chose as overall solution, and the open source implementations (components) of the open ststack, using OTS open source components.
Figure: the open standards (ISO or OGC) that OpenEarth chose as overall solution, and the open source implementations (components) of the open standards that OpenEarth chose as overall solution for a server stack.
Extract Transform Load
The data collection procedure and the relation between those standards is explained in the OpenEarth Data Standards document, developed in the framework of the EU FP7 Project MICORE and Building with Nature. The basis the 3-step ETL procedure well-known in the database world. ETL describes the process to Extract data from somewhere, Transform it to the strict database datamodel requirements, and Load it into the database. We extended ETL with one crucial extra step: provide the data again to users via the web. We believe that any effective data management solution should include users at the start of the ETL process and and the end. Loading data into the database and using data from the database should be possible from the work environment of the user. In the sketch above we explicitly included client and server to highlight the paramount importance of easy and immediate web-based Provide mechanisms of the data, that are not covered by ETL.
Step by step
ETL contains the followings steps:
- data is not just numbers and meta-information, but consists of raw data produced by the measuring equipment (e.g. volts) + processing scripts.
- raw data + scripts should be stored in the OpenEarthRawData repository enabling version control
- raw data should then be enriched with metadata and processed into useful data products (netCDF, PostgreSQL table) using transformation scripts that should also be put under version control in a repository
- resulting data products should conform to the best open source semantic standards available, e.g. CF, WoRMS
- data products should be made available easily via webbased interfaces (OPeNDAP, ODBC ore dedicated DB-APIs, WxS) but also with
...
- automatable procedures for widely-used data processing languages
...
- such as matlab, IDL, python, fortran, C
...
- and java (OpenEarth Tools).
- data products are primarily meant for dissemination, raw data and scripts are primarily meant for archiving.
To achieve this international standards are embraced like:
- NetCDF (self-describing, open source, widely-used file-format standard)
- CF parameter meta-data (vocabulary of quantities)
- Unidata units (vocabulary of units)
- EPSG spatial meta-data codes (vocabulary of coordinates)
- INSPIRE owner meta-data
- Open source Creative Commons philosophy or GNU
- OPeNDAP (as running operationally for MATROOS)
- OpenGIS
The data collection procedure and the relation between those standards is explained in the OpenEarth Data Standards document, developed in the framework of the EU FP7 Project MICORE. Currently data sets are being uploaded to the OPeNDAP test server. Examples from open datasets from the internet on OpenEarth include:
- Rijkswaterstaat coastal bathymetry data.
- Rijkswaterstaat coastal grain size fields.
- Rijkswaterstaat sediment atlas Wadden Sea (web source).
- Rijkswaterstaat DONAR database subset from www.waterbase.nl (water levels, salinity, temperature, SPM, chlorofyll) (web source).
- KNMI potential wind time series (web source).
- KNMI daily averaged meteo time series (web source).
- meta-data should be gathered and inserted into a central catalogue.
Other initiatives
Numerous other datasets have been or are being uploaded continually in the MICORE and Building with Nature research programs. And OpenEarth is not the only initiative to share and disseminate government-paid Earth science data freely on the web using open standards. We made an inventory of related initiatives. Our aim is to spread the use of the open standards and make them stick in our everyday work.
HTML | ||
---|---|---|
<object width="425" height="344"><param name="movie" value="http://www.youtube.com/v/7w2DBazX6g4&hl=en&fs=1"></param><param name="allowFullScreen" value="true"></param><param name="allowscriptaccess" value="always"></param><embed src="http://www.youtube.com/v/7w2DBazX6g4&hl=en&fs=1" type="application/x-shockwave-flash" allowscriptaccess="always" allowfullscreen="true" width="425" height="344"></embed></object> programmes.Some related initiatives (which OpenEarth does not intend to compete with, but to share with and to learn from):
Column | Wiki Markup |
|