Data Standards Training Course

The Data Standards Training Course was developed in the framework of the EU FP7 Project MICORE and involves one day in which the participants are taken through a wide range of excercises. Although the schedule can be adapted to better match with the participants' specific needs and background knowledge the schedule of this course generally is as follows:

• 9:00-9:05 Data management in general
  • Opening and outline of the course
    1. Extract Taking measurements and storing the measured data into files.
    2. Transform Enriching gathered data with metadata and storing in a standard file format.
    3. Load Storing the files in a database - not dealt with in this course.
    4. Provide Giving access to the database.
• 9:05-9:30 Open Earth initiative
  • General introduction of the OpenEarth initiative
• 9:30-10:30 Extract
  • Excercise: loading data into subversion
• 10:30-12:00 Transform
  • Excercise: storing metadata
  • Excercise: converting to netcdf

• 12:00-13:00 Lunch

• 13:00-14:00 Provide
  • Excercise: reading netcdf and opendap
  • Excercise: selection of data
• 14:00-16:00 Using database (e.g. for modelling)
  • useful tools available (General)
  • useful tools available (XBeach)
  • Excercise: grid -> xbeach

Table of contents

Data management in Micore (WP2)

Download presentation

Brief reiteration of WP2 objectives

The goal of this work package is to define a data standard and archiving protocol to provide end-users with a comprehensive standardised database. The protocol will become part of the Open Earth Initiative (formerly known as McTools) as developed by Deltares and be freely available. The data standard and archiving protocol provides a four step database guideline for MICORE users to:

• Extract - Taking measurements and storing the measured data into files,
• Transform - Enriching gathered data with metadata and storing in a standard file format,
• Load - Storing the files in a database - not dealt with in this course, and
• Provide - Giving access to the database.

physical data for each pilot site (bathymetries, waves, tides etc.) in a form convenient for the assessment of storm risk along the European coast. This will include (for each pilot site) data collected from regular monitoring schemes, as well as historic physical data (collected in WP1) and measured physical data (collected in WP3). Where possible, the UCIT (pronounced Use it!) methodology will be used and developed further. Where needed other approaches will be adopted.

More detailed info on WP2 can be found here.

Open Earth Initiative

Download presentation

• • Explanation of the Open Earth Initiative (former McTools/UCIT)
  • Overview of Open Earth building blocks
  • Attention for a few developing conventions

Extract

Download presentation

• • Working with subversion
  • Excercise: Loading raw data into subversion

Transform

Download presentation

• • Excercise: Filling in required fields using the inspire metadata editor
  • Excercise: Storing metadata file on OpenEarthRawData
  • Excercise: Loading raw data in memory
  • Excercise: Converting to netcdf
  • Using CF convention

Provide

Download presentation

• • Reading NetCDF and OPeNDap
  • Selection of data

Using database

Download presentation

• • Useful tools available (General)
  • Useful tools available (XBeach)
  • Several examples
  • Excercise: run examples and vary input

Using the database - course examples

Things that came up

• • Consider to store models in the OpenEarthModels repository
    • Include metadata about the models (describe who set it up, what it is used for, grid, used data, software version etc.)
  • Store XBeach tools in OpenEarthTools repository
  • Consider to upload a list to the wiki with standard organisation names
  • Consider to upload a list to the wiki with standard data set terms (e.g. all should use bathymetry rather than bathy, bottom, etc.)

Photos