Prior to the 2016 NCK days icebreaker, OpenEarth.nl organizes a hands-on sprint session on metocean data. In this traditional OpenEarth sprintsession, Gerben de Boer and Kees den Heijer will help you to optimally use and share your metocean data. The whole chain from raw data, conversion to netCDF, querying netCDF, and visualizing will be discussed. The session is supposed to be interactive and participants are encouraged to bring their own data to work on. We will first experiment with some existing (open access) metocean datasets of the Dutch coast. We will then explain Matlab and python tools to convert your own data to netCDF to facilitate your own analysis with analysis tools such as  Delft3D-quickplot, ncBrowse and Panoply.

i)                 ADCP data

ii)                Rijkswaterstaat FerryBox

iii)               Rijkswaterstaat ScanFish

vi)               Rijkswaterstaat CTD

v)                Rijkswaterstaat MWTL Water Quality data

During this sprint session you’ll learn how to work with these free data in Matlab, python and/or R. Bring your own laptop to work with the data, and take home your data products and visualizations.

Time:             March 16^th, 16-20h (prior to icebreaker starts that evening).

Location:       Inspiration Center Grevelingen on the Brouwersdam in Zeeland.

Registration: http://www.nck-web.org/upcoming-events/nck-annual-conference/20-nck-days-2016 (If you would like to only visit the sprint session, but not the NCK days itself, then register via ml-nck-list@deltares.nl)

More info: Gerben de Boer (Gerben.deBoer@vanoord.com), Kees den Heijer (Kees.denHeijer@deltares.nl) and Willem Stolte (willem.stolte@deltares.nl)

Bring your own laptop!

3TU.Datacentrum and Deltares signed a cooperation agreement to confirm their collaborative effort on further development and promotion of the OpenEarth DataLab.

The OpenEarth DataLab, presently successfully in use for the Sand motor data (zandmotordata.nl), is implemented and hosted by 3TU.Datacentrum and based on the OpenEarth stack as designed by Deltares.

The signature of both parties on the agreement initiates the process of bringing the OpenEarth DataLab code open source.

For more information, contact the 3TU.Datacentrum team or the OpenEarth team of Deltares.

Invitation:

Prior to the 2015 NCK days icebreaker, OpenEarth.nl organizes a hands-on sprint session to explain how to access and visualize some great open access core-description and grain size datasets of the Dutch coast:

i)                 Sediment Atlas WaddenZee (RWS): 7500(!) distribution curves in Marsdiep-EmsDollard

ii)                TNO 13.000(!) core descriptions for the Dutch continental shelf via NODC/SeaDataNet (and much, much more data available via DINOloket.nl)

iii)               TNO interpolated dz10, dz50, dz90 maps with 200m resolution for Dutch Continental Shelf

iv)               Along the Dutch beaches, grain sizes for dune safety assessment (RWS)

During this sprint session you’ll learn how to work with these free data in Matlab and/or python. Bring your own laptop to work with the data, and take home your data products and visualizations.

Time:             March 18^th, 13:30-17:00  (max 10 participants, sponsored by NCK)

Location:       http://www.ijgenweisschoorl.nl (where icebreaker starts at 18:30)

Registration: Gerben J. de Boer (Van Oord) Gerben.deBoer@vanoord.com ism TNO, Deltares, TUD

Flyer:            NCK days 2015 sprintsession north sea grain sizes.pdf

Course information:

Access information for the data can be found at Dataset documentation.

You can download netCDF files (to prevent wifi overload) with BITSadmin. An example is for the TNO gain size maps is:

set from=http://opendap.deltares.nl/thredds/fileServer/opendap/tno/ncp/
set into=d:\opendap\tno\
 
bitsadmin /transfer Job001 /download /priority normal %from%slib_juli2007.nc %into%slib_juli2007.nc
bitsadmin /transfer Job001 /download /priority normal %from%grind_fbr2007.nc %into%grind_fbr2007.nc
bitsadmin /transfer Job001 /download /priority normal %from%dz10_juli2007.nc %into%dz10_juli2007.nc
bitsadmin /transfer Job001 /download /priority normal %from%dz50_juli2007.nc %into%dz50_juli2007.nc
bitsadmin /transfer Job001 /download /priority normal %from%dz90_juli2007.nc %into%dz90_juli2007.nc

You can access subsets these files with the Matlab netCDF package called ncread and with the python package netcdf4-python, see our Matlab example and

python example.

TNO grain size maps

A Matlab simple example how to make a publication quality figure is given here

%% laod openearthtools for handy tools such as nc2struct, pcolorcorcen, colorbarwithvtext, tickmap, axislat, findfont
oetsettings
%% load data (local cache), D contains data, M contains all metadata
[D,M] = nc2struct('slib_juli2007.nc')

%% load data (via opendap)
L = nc2struct('http://opendap.deltares.nl/thredds/dodsC/opendap/deltares/landboundaries/northsea.nc')

%% plot
pcolorcorcen(D.longitude,D.latitude,D.mud)
colorbarwithvtext([M.mud.long_name,'[',M.mud.units,']']) % add text inside colorbar to maximize use of paper space. automatically use metdata from netCDF file
tickmap('ll') % add lat/lon as last tickmarks, no need for space, that dilute efficient use of paper space (google: Edward Tufte)
grid on
axislat % fix aspect ratio
plot(L.lon,L.lat,'k')
set(findfont,'fontsize',8)
title({M.nc_global.institution,'NCP map mud [version jul 2007]'}) % automatically use metdata from netCDF file as title
print2screensize('slib_juli2007')

We used this image as Fig. 8 in our 2011 paper Mechanisms controlling the intra-annual mesoscale variability of SST and SPM in the southern North Sea.

Sedimentatlas waddensea

A second Matlab example is a dataset used for Fig 9 in our 2008 paper Modeling large-scale cohesive sediment transport affected by small-scale biological activity. Here's an exmaple how to work with these data, find more script in

https://svn.oss.deltares.nl/repos/openearthrawdata/trunk/rijkswaterstaat/sedimentatlas_waddenzee/scripts/.

 %oetsettings
%% load data
 F          = 'http://opendap.deltares.nl/thredds/dodsC/opendap/rijkswaterstaat/sedimentatlas_waddenzee/korrel.nc';
 D.lon      = ncread(F,'lon');
 D.lat      = ncread(F,'lat');
 D.cumphi   = ncread(F,'cumphi');
 D.diameter = ncread(F,'diameter');
[D.x,D.y]   = convertCoordinates(D.lon,D.lat,'CS1.code',4326,'CS2.code',28992);% wgs84 to RD
fraction = 10; % fraction to work with

%% add coastal data
L = nc2struct('http://opendap.deltares.nl/thredds/dodsC/opendap/deltares/landboundaries/northsea.nc')
[L.x,L.y]   = convertCoordinates(L.lon,L.lat,'CS1.code',4326,'CS2.code',28992);% wgs84 to RD

%% plot data
caxis            ([0 100])
scatter          (D.x,D.y,20,D.cumphi(fraction,:),'.')
colorbarwithvtext(['D < ',num2str(D.diameter(fraction)),' \mum [%]'])
axis equal
grid on
tickmap          ('xy')
axis(axis)
hold on
plot(L.x,L.y,'k')
box on
print2screensize(['sedimentatlas_waddenzee_fraction_',num2str(D.diameter(fraction)),'mm'])

%% plot in Google Earth 
 KMLscatter(D.lat ,D.lon ,D.cumphi(10,:),'fileName',['sedimentatlas_waddenzee.kml'],...
    'CBcolorTitle',['D < ',num2str(D.diameter(fraction)),' \mum [%]'],...
 'CBcolorbarlocation',{'W'});

We have opened the OpenEarthTools repository for reading to anonymous users. This makes it easier to link to apps, scripts, notebooks and tools in openearth from publications. 

It also helps in automating the deployment of our OpenEarth stack, which we use to expose our data, models and tools. 

As a part of the Delft Software Days (http://www.dsd-int.nl/) Deltares and 3TU.Datacentrum jointly organise the Data Science Symposium. The symposium will also be used as an occasion to celebrate the Dutch Data Prize 2012 that we won with OpenEarth.
Attendance is free of charge, but please make sure to register.

For more information see:

http://www.dsd-int.nl/symposium/data-science-symposium-addressing-4th-paradigm/

We recently moved a few projects over to github. The main code repository of OpenEarth is still hosted at subversion.

The projects that were moved are some projects were we expect collaboration with other software developers (BMI, MMI). These tend to appreciate the benefits of faster commits, local history and the pull request workflow over the more complex model of having a local as well as a remote repository.

Some have asked us why we are not moving everything to git. We have quite some experience in teaching people for whom version management is not an everyday, but more a few times per year activity. Learning the difference between init, pull, clone, fork, update, fetch, sync is not everybody's cup of tea.

One of the nice features of porting your code to github is that it is easy to setup a doi. This is also available for datasets in OpenEarth. If you want to have code moved to the openearth github, or if you want to know something about making your data or code citeable, please consult Fedor Baart or Gerben de Boer.

This is an example of the DOI for the Model Message Interface MMI code, which is cited for the IEMSS conference: http://dx.doi.org/10.5281/zenodo.9876

On Wednesday afternoon (13-17h) March 26th, an OpenEarth workshop is organised, as part of the NCK days 2014 at UNESCO-IHE. The theme of this year’s session is ‘Zandmotor’.

The monitoring data of the Zandmotor is stored at the 3TU datacentrum in a OpenEarth system (http://zandmotordata.nl).

The session will focus on the (newest) technical features of the server system and the actual data access (OPeNDAP and PostGIS/Geoserver) and vizualisation. Since the Zandmotor data is not publicly available yet, we will briefly discuss the procedure to get access to the data. We will practice with the same technology, using open data.

The session is organised by Kees den Heijer and Gerben de Boer.

• when: Wednesday March 26th, 2014 13.00-17.00h
• where: UNESCO-IHE, Delft
• sign-up: please register via http://nckdays2014.eventzilla.net/
• !!  Bring your own data and laptop !!

Friday Nov. 1st: OpenGIS Workshop

This collection of standards is treated today:

netCDF exercise for Delft software days OpenGIS workshop - 1 nov 2013

Delft Software Days 2013 - OpenGIS Workshop - netcdf

Delft Sotware Days 2013 - OpenGIS Workshop - vectors

WMS/KML exercise for Delft software days OpenGIS workshop - 1 nov 2013

• WMS manual: https://publicwiki.deltares.nl/display/OET/WMS+primer
• WMS into Google Earth manual: http://publicwiki.deltares.nl/display/OET/WMs+in+Google+Earth
• and here are a few urls to exercise with. These are the so called getcapabilities which always
  append request=GetCapabilities&service=WMS to the base url until the ?-mark.
  • Aerial images Minnesota
    • http://geoint.lmic.state.mn.us/cgi-bin/wmsll?request=GetCapabilities&service=WMS
    • http://gdsc.nlr.nl/wms/lufo2005?request=GetCapabilities&service=WMS
    • http://wms.agiv.be/ogc/wms/omkl?request=GetCapabilities&service=WMS
  • World bathymetry maps:
    • http://www.gebco.net/data_and_products/gebco_web_services/web_map_service/mapserv?request=GetCapabilities&service=WMS
    • http://geoport.whoi.edu/thredds/wms/bathy/etopo2_v2c.nc?request=GetCapabilities&service=WMS (from THREDDS)
  • Numerical model in netCDF-CF rendered straight into WMS. These examples include time as dimension:
    • http://data.ncof.co.uk/thredds/wms/METOFFICE-NWS-AF-BIO-DAILY?request=GetCapabilities&service=WMS (from THREDDS)
    • http://geoservices.knmi.nl/cgi-bin/RADNL_OPER_R___25PCPRR_L3.cgi?request=GetCapabilities&service=WMS

Monday Nov. 4th: Open Source Pre- and Postprocessing Workshops

Please see

• this tech-notes
  • TortoiseSVN
  • Matlab plotting into Google Earth
• and these webinars
  • 7. 'How to set up a Delft3D model with Delft Dashboard'
  • 8. 'Tsunami simulations using Delft Dashboard and Delft3D'
  • 15. 'Efficient pre- and post-processing in Delft3D using the OpenEarthTools'

More info on Deltares open source website

May 30th an OpenEarth data sprintsession for Matlab, python, R is organized at request for WaLTER. We will show how to work in DatTube mode with data from the web:

• PostgreSQL with PostGIS for relational data (ecological data and in situ physical data)
• netCDF-CF-OPeNDAP for grid data. Deltares and NIOZ produce 200m resolution data (images below) for the (Western) Wadden Sea in the PACE project. This yields about 1TB of data, which you can only work with in web-based mode, it is too much to copy.

We start with a lunch lecture first, and then proceed with a hands-on afternoon
how to extract a relevant subset of the data deluge with Matlab, python and R scripts.

• when: Thursday May 30th, 12:00 lunch lecture with sprint session afterwards until 16:00.
• where: NIOZ, Texel. route: www.nioz.nl
• Sign-up: gerben.deboer<at>deltares.nl
• !! Bring your own laptop with your own Matlab/Python/R!!

For tutorials see PostgreSQL for relational data and netCDF-CF-OPeNDAP for grids.

A new NetCDF file with the JARKUS transects up to 2012 is now available via http://opendap.deltares.nl/thredds/catalog/opendap/rijkswaterstaat/jarkus/profiles/catalog.html

A copy as well as previous releases can be found at http://dtvirt5.deltares.nl:8080/thredds/dodsC/opendap/rijkswaterstaat/jarkus/profiles/catalog.html

For more info, see http://publicwiki.deltares.nl/display/OET/Dataset+documentation+JarKus

March 13th we organize the next free OpenEarth sprintsession for a mixture of scientists/engineers (matlab, python and R) and policymakers (google earth).

The special topic of this session will be the mind-blowing vaklodingen dataset. Since 1927 Rijkswaterstaat has compiled bathymetry data of the entire Dutch foreshore up to 20 m depth, as well as the entire Wadden Zee and Zeeuwse Delta. The data are available at an amazing 20m resolution, with updates about every 8 year.

We made this data available via OPeNDAP (web-enabled netCDF) for ready access in Matlab, python and R. We provide tutorials and functions to query this enormous dataset on the fly via the web.

Also all data (23 GB of arc ac files) can be viewed after one mouse-click in flight simulator mode via Google Earth. This dataset
is also part of an easy-to-use dedicated portal for all Dutch coastal data.

• when: Wednesday March 13th, 14:00-18:00, drinks afterwards (NCK icebreaker).
• where: Vakantiepark Kijkduin / Parker Beachclub, for details and directions see NCK 2013 webpage
• Sign-up: Please register two ways:
  • Via the official NCK registry http://www.nck-web.org/index.php/nck-days.html
  • by sending an email to <gerben dot deboer at deltares dot nl>.
• !!  Bring your own data and laptop !!

Dec 18th Postgres-Postgis database sprintsession for matlab, python, R and more. Building with Nature, Zandmotor and two PMR (Project Mainportontwikkeling Rotterdam) monitoring programs jointly organize a sprint sessions on live web access to a relational geo database.

• when: Tuesday December 18nd, 13:00-16:30, drinks afterwards.
• where: Deltares, location Rotterdamseweg, Delft. route: www.deltares.nl
• Sign-up: Joost.Boerboom<at>deltares.nl
• !!  Bring your own data and laptop !!

For tutorials see PostgreSQL.

pdf:

See following link for jury comments (in Dutch):
http://www.dans.knaw.nl/content/winnaars-nederlandse-dataprijs-2012-bekend

See following links for news items:

• http://www.delta.tudelft.nl/artikel/openearth-en-political-mashup-winnen-dataprijs-2012/25803
• http://www.deltares.nl/nl/actueel/nieuwsbericht/item/14706/openearth-wint-nederlandse-dataprijs
• http://www.dredgingtoday.com/2012/10/29/the-netherlands-openearth-receives-dutch-data-award/

See the following picture for an impression of the Dutch Data Prize 2012 award ceremony:

Here an overview of the events during the day:

A pdf of the presentation (below display macro doesn't always work properly, download to view if presentation seems incomplete):

Today 3TU.Datacentrum (twitter) and DANS awarded the Dutch Data Award prizes as a mark of appreciation for researchers who gather and document data, and make it accessible. We are extremely proud to announce that OpenEarth won the Dutch Data Award 2012 in the category exact and technical sciences!!

Together with Gerben de Boer, Thijs Damsma, Joost van Wiechen and Marcel Stive we represented the OpenEarth commnunity and presented our case. Thanks to everyone for your contributions of the years. This award is true acknowledgement of all your hard work and a great stimulus for the future!