DONAR MWTL time series data via Waterbase in OpenEarth:
The Rijkswaterstaat historical time series data can be accessed via Waterbase. Waterbase is a mechanism that provides a web-service for a subset of the DONAR (in Dutch:Data Opslag NAtte Rijkswaterstaat) database that contains all historical data for all water-related Rijkswaterstaat affairs. MWTL is the name of the monitoring programme (in Dutch: Monitoring Waterstaatkundige Toestand des Lands Milieumeetnet rijkswateren) DONAR also contains grids, lines etc. For these non time series data see the links on Rijkswaterstaats waterdata_waterberichtgeving, this also includes Near Real Time (NRT) data in a geoviewer. NRT data feeds are available via www.rws.nl/rws/opendata. Much information on RIjkswaterstaat data is also present in the HelpDeskWater. Currently Rijkswaterstaat is working on updating the IT infrastructure to connect everything via an internal Data Distribution Layer.
- Raw data + scripts (ascii + Matlab + Python)
- All ascii time series data are available via a live web service from Rijkswaterstaat called Waterbase. The Waterbase web site has a menu to guide you to the correct data which are available via a direct access download mechanism: http://live.waterbase.nl. There is en english version live.waterbase.nl/waterbase_wns.cfm?taal=en.
The waterbase menu is in fact only a macro to construct a url that is subsequently called to download the data. Passing a command to a server via a url is a so-called web service. The web service urls to download the data can also be constructed without the macro, and be called directly from other programs, for instance by Matlab. The syntax for the waterbase urls is fairly simple. For instance, to download the water levels from Katwijk in in the 18th century you can use this url. This url consists of a base-url followed by ampersand(&)-separated keywords.
http://live.waterbase.nl/wboutput.cfm? loc=KATWK &wbwns=1|Waterhoogte+in+cm+t.o.v.+normaal+amsterdams+peil+in+oppervlaktewater &byear=1737&bmonth=01&bday=01 &eyear=1808&emonth=01&eday=01 &output=Tekst &whichform=2
The method to pass command via url keywords is the typical for web services, and has the same syntax-grammer same as the international OGC WMS and WCS standards used for grids. For time series there is no internationally accepted and implemented set of keywords yet (WFS and SOS will do). Ampersand-separated keywords are also used in the operational Matroos system of Rijkswaterstaat.
- The ascii files that are returned by Waterbase need to be parsed. OpenEarthTools has (i) a Matlab function to parse these files into a Matlab struct:
rws_waterbase_load.m
and (ii) a python tool box. We recommend to choose the text files from Waterbase, and not the Excel files. The Excel files are not real (binary) Excel files, but are tab-delimited ascii files. This means that they do load well into Excel directly, but also that you cannot read them into Matlab withxlsread
.- Matlab Toolbox for downloading and parsing ascii files:
In OpenEarthTools we made a toolbox that constructs the Waterbase urls for specific combinations of parameter and location. The toolbox allows you to download the data in batch mode from within Matlab. These tools have prefixrws_waterbase_get*.m
. Together with the aforementioned function to parse the ascii files (rws_waterbase_load.m
), this forms a complete toolbox to work with MWTL data in Matlab.
https://svn.oss.deltares.nl/repos/openearthtools/trunk/matlab/applications/Rijkswaterstaat/ (register at oss.deltares.nl) - Python Toolbox for downloading and parsing ascii files: (under development)
https://svn.oss.deltares.nl/repos/openearthtools/trunk/python/applications/Waterbase/waterbase/ (register at oss.deltares.nl) - R script for downloading ascii files
Download Rijkswaterstaatdata from Waterbase or https://github.com/wstolte/WaterbaseR
R Shiny app for interactive visualization of Waterbase data for the North Sea (based on previously downloaded data, see previous point) (view on github) - Web service for downloading ascii files
http://live.waterbase.nl
- Matlab Toolbox for downloading and parsing ascii files:
- Standardized data (netCDF-CF on OPeNDAP)
- Working with the bunch of text files as returned is not very useful, despite the availability of the
rws_waterbase*.m
toolbox for Matlab. In addition, for use of the MWTL data in other analysis programs such as R and Python therws_waterbase*.m
toolbox is of no use. For both analysis languages, dedicated parse functions have to be programmed as well. Therefore, we extended therws_waterbase*.m
toolbox, so that it not only downloads the data in batch mode, but transforms the data to an internationally standardized format that can readily be used by R and python as well in addition to Matlab: netCDF-CF. This resulted in a complete workflow in Matlab that downloads Waterbase data automatically using the above mentioned web service urls, parses them into Matlab memory withrws_waterbase_load.m
, and usesrws_waterbase2nc.m
to save them as a netCDF-CF file , and adding preliminary Aquo meta-data. These netCDF files are made available on our open OPeNDAP server opendap.deltares.nl. We update this OpeNDAP server manually periodically. For easy querying, we harvest all meta-data, and cache that as a separate netCDF file called catalog.nc. We are planning to cache the meta-data also as a catalog.xml file, that can readily be ingested by meta-data servers such as geonetwork. As a final step we make an overview of all stations for one parameter in a kml file, aka known as Google Earth file. The Matlab scriptrws_waterbase_all.m
performs this entire workflow. It is run periodically to update our OPeNDAP server and kml server (see Figure below).- Toolbox for downloading & parsing ascii files, and transformation to netCDF+kml:
https://svn.oss.deltares.nl/repos/openearthtools/trunk/matlab/applications/Rijkswaterstaat/ (register at oss.deltares.nl) - NetCDF data on our OPeNDAP server
http://opendap.deltares.nl/thredds/catalog/opendap/rijkswaterstaat/waterbase/catalog.html - Manual on use of netCDF data on OPeNDAP server
Matlab or python or R
- Toolbox for downloading & parsing ascii files, and transformation to netCDF+kml:
- Working with the bunch of text files as returned is not very useful, despite the availability of the
- Visualized data overviews in Google Earth with previews of statistics (kml):
Our kml server contains kml files with an overview of all locations per parameter. The Google Earth time slider is used to show when data is present at a specific location. For each location a pop-up shows some statistics, and a preview of the time series and histogram. The kml contains links to the netCDF data, and to the original live.waterbase.nl web service url.
DONAR MWTL time series (waterbase) description:
- Rijkswaterstaat Waterbase.
http://www.rijkswaterstaat.nl/water/waterdata_waterberichtgeving/index.aspx - Rijkswaterstaat MWTL plan for 2010.
http://www.rijksoverheid.nl/documenten-en-publicaties/rapporten/2011/12/30/mwtl-meetplan-waterdienst-2012-monitoring-waterstaatkundige-toestand-des-lands-milieumeetnet-rijkswateren.html - Landelijk Meetnet Water.
http://www.rijkswaterstaat.nl/water/waterdata_waterberichtgeving/waterberichten/index.aspx - Matlab toolbox for DONAR-interface, the offline Rijkswaterstaat database behind Waterbase (not accessible via web). You might receive DONAR interface files when requesting data via http://www.helpdeskwater.nl/
http://www.modelit.nl/index.php/donar-matlab-utilities-kosteloos-beschikbaar - The company Xi advies has developed some tools to work with the DONAR data behind Waterbase:wetwetwet, metis
See also:
- DONAR Matlab package released Feb 2014
- Dataset documentation JarKus
- Dataset documentation Kustlijn
- web service examples: WCS primer, WMS primer
- NOAA SOS
- Dutch Aquo semantic annotation standard
- EMODNet physics
OpenEarth Matlab rws_waterbase*.m
toolbox