You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 36 Next »

Contents

NetCDF formats in general

The NetCDF file format is very flexible in the ways that it allows the data to be stored in the file. Unfortunately this means that NetCDF files come in many different formats and the way the variables and dimensions (data structure) are organised in the file can be completely different for different NetCDF files. Therefore it cannot be expected that the FEWS NetCDF import will be able to import every single netcdf file that's out there. Because it is just not possible to anticipate all the possible data structures that people can come up with.

For this reason there exist several conventions that describe standard ways of organising the data in NetCDF files. The FEWS NetCDF import is based on the CF-1.6 conventions. This means that if a NetCDF file is in a format that is fully compatible with the CF-1.6 conventions, then there is a good chance that FEWS will be able to import the data from the NetCDF file. However there are a few exceptions, which are listed on this page. For instance FEWS can only import data that depends on a time dimension (time series), while the CF-1.6 conventions also describe formats for data that does not depend on time. In other words: Delft-FEWS restricts the NetCDF CF-1.6 conventions by applying extra limitations on the format of the NetCDF files.

Furthermore there are some exceptions where FEWS can import NetCDF data, even though it does not comply with the CF-1.6 conventions. This can be due to historic reasons. Or this can be due to the fact that some data formats are not (yet) part of the CF-1.6 conventions (e.g. spectral data), in which case these exceptions are also described on this page.

NetCDF formats that can be imported in Delft-FEWS

TODO work in progress

A first check to determine whether a given NetCDF file can be imported in FEWS is to use an online CF compliance checker, see e.g. http://cfconventions.org/compliance-checker.html. If the NetCDF file is not valid according to the CF compliance checker, then the NetCDF file can probably not be imported in FEWS. In this case please contact the data supplier that created the NetCDF file and ask them to make the NetCDF file CF-1.6 compliant. If the file is not CF-1.6 compliant then there is no reason to contact the FEWS support team.

Scalar Time Series Example

Netcdf file: scalar_time_series.nc

Text version: scalar_time_series.txt

Regular Grid Time Series Example

Netcdf file:  regular_grid_time_series.nc

Text version: regular_grid_time_series.txt

Curvilinear Grid Time Series Example

Netcdf file: curvilinear_grid_time_series.nc

Text version: curvilinear_grid_time_series.txt

Unstructured Grid (UGRID) Time Series

For an example of data in UGRID format, see https://github.com/ugrid-conventions/ugrid-conventions/blob/v0.9.0/ugrid-conventions.md

To import unstructured grid data in Delft-FEWS you need to add an irregular grid definition to the grids.xml configuration file for your data. For more information see 06 Grids.

Track Time Series (single trajectory per file)

For featureType "point" all points in the file are imported together to form a single track. For featureType "trajectory" the parser supports only a single track per file, i.e. variables in the NetCDF file must be one dimensional and only have a time dimension. For an example of the track data format that can be imported, see http://cfconventions.org/cf-conventions/cf-conventions.html#_single_trajectory

Checklist to create NetCDF data in the format of the Netcdf CF-1.6 conventions

TODO work in progress

This checklist focusses on the points that are important to be able to import the netcdf data into Delft-FEWS. The full NetCDF CF-1.6 conventions can be found on http://cfconventions.org/Data/cf-conventions/cf-conventions-1.6/build/cf-conventions.html.

Before creating/adapting any code/script to produce NetCDF files, it can be very useful to read the following webpages:

Furthermore some links with reference documentation about NetCDF files:

All data

The global attribute "Conventions" must be specified in the netcdf file (e.g. Conventions="CF-1.6"). If this attribute is not present or not valid, then the NetCDF Java library assumes a default convention (different from CF), which can cause unexpected behaviour in the NetCDF import in Delft-FEWS. Also see http://cfconventions.org/Data/cf-conventions/cf-conventions-1.6/build/cf-conventions.html#identification-of-conventions.

Grid data

  1. all data and coordinate variables must have a unique variable name and the following attributes:
    units (required)
    standard_name (required)
    long_name (recommended)
    _FillValue and/or missing_value (required if there are missing values in the data)
    scale_factor (optional to compact data)
    add_offset (optional to compact data)
    Also see http://cfconventions.org/Data/cf-conventions/cf-conventions-1.6/build/cf-conventions.html#units, http://cfconventions.org/Data/cf-conventions/cf-conventions-1.6/build/cf-conventions.html#long-name and http://cfconventions.org/Data/cf-conventions/cf-conventions-1.6/build/cf-conventions.html#standard-name.
  2. the standard_name attributes should comply to the list with standard names, see http://cfconventions.org/26.html.
  3. a data variable for a 2D grid must depend on three dimensions, representing time and two spatial dimensions (in the order time, y, x). The names of the dimensions can be anything. For the time dimension there must be a corresponding 1D time coordinate variable with the same name. If the grid is regular or rectangular, then for each spatial dimension there must be a corresponding 1D coordinate variable with the same name. See http://cfconventions.org/Data/cf-conventions/cf-conventions-1.6/build/cf-conventions.html#idp5553648. If the grid is curvilinear, then the y and x coordinate variables are both 2D and depend on both spatial dimensions. In this case the data variable must have an attribute "coordinates" that refers to the y and x coordinate variables (in that order), e.g. coordinates="y x". See http://cfconventions.org/Data/cf-conventions/cf-conventions-1.6/build/cf-conventions.html#idp5559280. Each coordinate variable must have an attribute "axis" that indicates which axis is defined by that coordinate variable. For the time coordinate variable the axis must be "T", for the x coordinate variable the axis must be "X", for the y coordinate variable the axis must be "Y".
  4. if the data has a forecast reference time/analysis time/base time, then this time must be stored in a separate coordinate variable that contains only one value, which is the forecast reference time. This variable must have standard_name="forecast_reference_time". The name of this variable must be present in the coordinates attribute of the data variable, e.g. coordinates="analysis_time" or coordinates="y x analysis_time". If needed, different data variables can point to different forecast reference time variables. See http://cfconventions.org/Data/cf-conventions/cf-conventions-1.6/build/cf-conventions.html#scalar-coordinate-variables
  5. the time coordinate variable must have an attribute "units" with a valid unit of time and an attribute "axis" with value "T". A valid unit of time is a string formatted as per the recommendations in the Udunits package, e.g. units="minutes since 1970-01-01 00:00:00.0 +0000". See http://cfconventions.org/Data/cf-conventions/cf-conventions-1.6/build/cf-conventions.html#time-coordinate.
  6. the time and spatial coordinate variables must have values either in ascending or in descending order.

Compacted data

Netcf data variables with attributes scale_factor and add_offset will be treated as compacted data and the values of these attributes will be used to uncompact the values of this variable.

Error messages from the Delft-FEWS NetCDF import

Invalid URL

If you get the error "invalid URL" then either the URL is invalid, or the OpenDAP server does not return any data. The latter either means that the OpenDAP server itself does not work correctly, or that the user does not have access rights to download the file on the OpenDAP server. This can be easily verified by following these steps:

  1. copy the URL of the netcdf file or catalog from the FEWS configuration or from the error message.
  2. if the URL ends with .nc, then append .html to it, so that it ends with .nc.html
  3. go to the resulting URL in a browser, e.g. Mozilla Firefox.
  4. if the browser gives an error, then there is a problem with the URL, the OpenDAP server or user access rights to the server. In this case please contact the administrator of the OpenDAP server (and not the FEWS support team).

Errors from the Java NetCDF library

If you get an error like e.g. "Failed to open NetCdf file from url" or "Variable name (...) must be unique within Group", then the netcdf file is probably corrupt or not CF-1.6 compliant. If the error message is accompanied by a stacktrace (in the FEWS log.txt file) then look for the first part of the stacktrace. Error messages from the Java NetCDF library can be recognised by a line that starts with something like "ucar.nc2.", e.g.:

java.lang.IllegalArgumentException: Variable name (hgtpv34768) must be unique within Group
version: 2014.02, build: 54174 (patched from 50595) May21, type: stable, jre: 1.7.0_40, os:Windows 7, mx: 1037m, db: Firebird
            at ucar.nc2.Group.addVariable(Group.java:458)
            at ucar.nc2.dods.DODSNetcdfFile.addVariable(DODSNetcdfFile.java:646)
            at ucar.nc2.dods.DODSNetcdfFile.constructConstructors(DODSNetcdfFile.java:634)
            at ucar.nc2.dods.DODSNetcdfFile.<init>(DODSNetcdfFile.java:277)
            at ucar.nc2.dataset.NetcdfDataset.acquireDODS(NetcdfDataset.java:805)

In this case please contact the data supplier that created the NetCDF file and ask them to make the NetCDF file CF-1.6 compliant. If the file is not CF-1.6 compliant then FEWS cannot import the file and there is no reason to contact the FEWS support team.

Related modules and documentation

Internal

External

  • No labels

Disclaimer