You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Current »

Preparation for the development of a model adapter

Model executable

All models are executed in FEWS by the General Adapter module (GA). All communication between GA and the executable is file based. This leads to the following requirements to the model executable:

Model data

The General Adapter module (GA) exports and imports model data, both model input and output and information about the model run. GA can export and import NetCDF-CF files.

  • Run info file
  • The run file is written by GA as input for an executable.
  • A run info file contains the start and end time of the simulation and other properties, settings, attributes, and parameters a model needs.
  • The default format is NetCDF (see below).
  • A model specific adapter can be required to convert and adjust the information in this file.
  • Model input and output
  • NetCDF-CF files (see below) are the preferred way of exchanging data between GA and executables.
  • A pre- and or post-adapter can be required to convert model files from these formats to a model native file format.
NetCDF-CF

CF is a set of conventions how to store grids and time series in a NetCDF file. NetCDF libraries are availablefor many languages, such as Java / Fortran /C / C++/ Python/ R / Matlab, see http://www.unidata.ucar.edu/software/netcdf/docs/faq.html#languages
A list of CF standard names can be found here: http://cf-pcmdi.llnl.gov/documents/cf-standard-names/standard-name-table/18/cf-standard-name-table.html
Definition of the run_info.nc: http://publicwiki.deltares.nl/display/FEWSDOC/05+General+Adapter+Module#05GeneralAdapterModule-Exampleofanexportednetcdfrunfile%28convertedtotextformat%29%3A

There are many viewers; three of them are widely used. They all have their own strengths.

Running tasks on a cluster

The following issues are related to clusters:

  • A file that describes how the task should be run on the cluster. Obviously, the content and format of this file are cluster specific. However, the dynamic information in this file is probably limited, e.g. T_Start, T_Stop and a restart file. A similar (but different) file is also necessary when you are not running on a cluster. The model adapter will have to generate this file, during the training there will be time for this topic.
  • FEWS cannot use GUI's like putty to make a connection (via SSH) to a cluster. A run on a cluster will have to be started using a script and/or the command line.
  • How can FEWS know when a run is finished? 

NetCDF – check list

Checklist to create netcdf grid data in the format of the Netcdf CF 1.6 conventions.
The full netcdf CF 1.6 conventions can be found on http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html
This checklist focuses on the points that are important to be able to import the netcdf grid data into Delft-FEWS.

  1. all data and coordinate variables should have a unique variable name and the following attributes:
    1. units (required, see http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html#units)
    2. standard_name (required, see http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html#standard-name)
    3. long_name (recommended, see http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html#long-name)
    4. _FillValue and/or missing_value (recommended)
    5. scale_factor (optional)
    6. add_offset (optional)
  2. the standard_name attributes should comply to the list with standard names, see http://cf-pcmdi.llnl.gov/documents/cf-standard-names/standard-name-table/22/cf-standard-name-table.html
  3. a data variable for a 2D grid must depend on three dimensions, representing time and two spatial dimensions (in the order time, y, x). The dimension names can be anything.
    For the time dimension there must be a corresponding time coordinate variable with the same name.
    If the grid is regular, then for each spatial dimension there must be a corresponding coordinate variable with the same name. See http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html#idp5553648
    If the grid is rectangular or curvilinear, then the y and x coordinate variables are both two-dimensional and depend on both spatial dimensions. In this case the data variable must have an attribute "coordinates" that refers to the y and x coordinate variables (in that order), e.g. coordinates="y x"
    See http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html#idp5559280
  4. if the data has a forecast reference time/analysis time/base time, then this time must be stored in a separate coordinate variable that contains only one value, which is the forecast reference time. This variable must have standard_name="forecast_reference_time". The name of this variable must be present in the coordinates attribute of the data variable, e.g. coordinates="analysis_time" or coordinates="y x analysis_time" If needed, different data variables can point to different forecast reference time variables. See http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html#scalar-coordinate-variables
  5. the time coordinate variable must either have an attribute "units" with a valid unit of time and/or an attribute "axis" with value "T".
  6. the time and spatial coordinate variables must have values either in ascending or in descending order.

NetCDF – unstructured grids

The UGRID-conventions form the first standard for storing in NetCDF files unstructured grids and data on those grids.
It serves as an extension on top of the well-known CF-conventions. Unstructured here means: 1D networks/graphs, 2D grids (with triangles/quads/polygonal cells), 3D layered grids and fully unstructured 3D grids, and combinations of all of these.

The UGRID-conventions are maintained on the following website: https://github.com/ugrid-conventions/ugrid-conventions
You can find a link to the latest UGRID-0.9 conventions there.

The current status is:

  • People from BAW, Germany are already writing UGRID-compliant files from the UnTRIM-solver, and also have a not-yet standardized extension for water quality data.
  • Deltares will adhere to UGRID-standards with the D-Flow Flexible Mesh solver next month.
  • The US NOAA and USGS are also actively involved, enriching their (measurement) data sets with the meta-information required by UGRID conventions.
  • More institutions involved.

Postprocessing support is still in its early stages, but some people have already been working on a ParaView-plugin, en Delft3D-QUICKPLOT supports UGRID.
See also the UGRID Discussion group: https://groups.google.com/forum/#!forum/ugrid-interoperability

Model requirements – check list

  1. Ensure the model executable can run without user interaction from the command line.
  2. The start and end time of the simulation is controllable via the command line or settings file
  3. Try to compile the model together with a NETCDF library
  4. Error messages are written to a log file. No message boxes are popping up in case of an error

What to bring - check list

For the model adapter training, please bring a laptop with the following:

  1. The model software.
  2. Source code of the model software if available.
  3. A small (fast) model that can run on the laptop, which is complete in the sense of input and output.
  4. All required model input.
  5. A cold and/or warm state to (re-)start the model from.
  6. A development suite of choice (for example Java, Matlab, see section 2.1) with the means to develop and compile the model adapter, including a NetCDF library (see section 2.1). You should be experienced with this language.
  • No labels

Disclaimer