Contents
NetCDF formats in general
The NetCDF file format is very flexible in the ways that it allows the data to be stored in the file. Unfortunately this means that NetCDF files come in many different formats and the way the variables and dimensions (data structure) are organised in the file can be completely different for different NetCDF files. Therefore it cannot be expected that the FEWS NetCDF import will be able to import every single netcdf file that's out there. Because it is just not possible to anticipate all the possible data structures that people can come up with.
For this reason there exist several conventions that describe standard ways of organising the data in NetCDF files. The FEWS NetCDF import is based on the CF-1.6 conventions. This means that if a NetCDF file is in a format that is fully compatible with the CF-1.6 conventions, then there is a good chance that FEWS will be able to import the data from the NetCDF file. However there are a few exceptions, which are listed on this page. For instance FEWS can only import data that depends on a time dimension (time series), while the CF-1.6 conventions also describe formats for data that does not depend on time. In other words: Delft-FEWS restricts the NetCDF CF-1.6 conventions by applying extra limitations on the format of the NetCDF files.
Furthermore there are some exceptions where FEWS can import NetCDF data, even though it does not comply with the CF-1.6 conventions. This can be due to historic reasons. Or this can be due to the fact that some data formats are not (yet) part of the CF-1.6 conventions (e.g. spectral data), in which case these exceptions are also described on this page.
NetCDF formats that can be imported in Delft-FEWS
TODO work in progress
A first check to determine whether a given NetCDF file can be imported in FEWS is to use an online CF compliance checker, see e.g. http://cfconventions.org/compliance-checker.html. If the NetCDF file is not valid according to the CF compliance checker, then the NetCDF file can probably not be imported in FEWS. In this case please contact the data supplier that created the NetCDF file and ask them to make the NetCDF file CF-1.6 compliant. If the file is not CF-1.6 compliant then there is no reason to contact the FEWS support team.
Scalar Time Series Example
Netcdf file: scalar_time_series.nc
Text version: scalar_time_series.txt
Regular Grid Time Series Example
Netcdf file: regular_grid_time_series.nc
Text version: regular_grid_time_series.txt
Curvilinear Grid Time Series Example
Netcdf file: curvilinear_grid_time_series.nc
Text version: curvilinear_grid_time_series.txt
Unstructured Grid (UGRID) Time Series
For an example of data in UGRID format, see http://ugrid-conventions.github.io/ugrid-conventions/
To import unstructured grid data in Delft-FEWS you need to add an irregular grid definition to the grids.xml configuration file for your data. For more information see 06 Grids.
Track Time Series (single trajectory per file)
For featureType "point" all points in the file are imported together to form a single track. For featureType "trajectory" the parser supports only a single track per file, i.e. variables in the NetCDF file must be one dimensional and only have a time dimension. For an example of the track data format that can be imported, see http://cfconventions.org/cf-conventions/cf-conventions.html#_single_trajectory
NetCDF kickstarter tool
The netCDF kickstarter is a very useful tool for creating template netCDF files and is available on http://zandmotor.citg.tudelft.nl/netcdfkickstarter/ This tool can be used to create template netCDF files or to create scripts that generate template netCDF files. In this tool the user can specify what data variables the netCDF file should contain and the netCDF kickstarter will create a template netCDF file that contains the corresponding metadata variables and empty data variables, fully compliant according to the CF conventions. Then the user only needs to add the actual data values to this template file in order to create a CF compliant netCDF file. The kickstarter can also generate a script (in one of several programming languages) to create a template netCDF file that contains the metadata variables and empty data variables specified by the user. Then the user only needs to extend this script with commands to write the actual data values to the netCDF file.
Checklist to create NetCDF data in the format of the Netcdf CF-1.6 conventions
TODO work in progress
This checklist focusses on the points that are important to be able to import the netcdf data into Delft-FEWS. The full NetCDF CF-1.6 conventions can be found on http://cfconventions.org/Data/cf-conventions/cf-conventions-1.6/build/cf-conventions.html.
Before creating/adapting any code/script to produce netCDF files, it can be very useful to try the netCDF kickstarter tool described above ( http://zandmotor.citg.tudelft.nl/netcdfkickstarter/ ), and/or read the following webpages:
- http://www.unidata.ucar.edu/publications/factsheets/current/netcdf_factsheet.pdf
- https://www.unidata.ucar.edu/software/netcdf/docs/BestPractices.html
- http://cfconventions.org/Data/cf-conventions/cf-conventions-1.6/build/cf-conventions.html
Furthermore some links with reference documentation about netCDF files:
- https://github.com/cf-convention/cf-documents/blob/master/other/cf_overview_viewgraphs.pdf?raw=true
- https://www.unidata.ucar.edu/software/netcdf/workshops/2012/cf/index.html
- https://www.unidata.ucar.edu/software/netcdf/workshops/2012/index.html
All data
The global attribute "Conventions" must be specified in the netcdf file (e.g. Conventions="CF-1.6"). If this attribute is not present or not valid, then the NetCDF Java library assumes a default convention (different from CF), which can cause unexpected behaviour in the NetCDF import in Delft-FEWS. Also see http://cfconventions.org/Data/cf-conventions/cf-conventions-1.6/build/cf-conventions.html#identification-of-conventions.
Grid data
- all data and coordinate variables must have a unique variable name and the following attributes:
units (required)
standard_name (required)
long_name (recommended)
_FillValue and/or missing_value (required if there are missing values in the data)
scale_factor (optional to compact data)
add_offset (optional to compact data)
Also see http://cfconventions.org/Data/cf-conventions/cf-conventions-1.6/build/cf-conventions.html#units, http://cfconventions.org/Data/cf-conventions/cf-conventions-1.6/build/cf-conventions.html#long-name and http://cfconventions.org/Data/cf-conventions/cf-conventions-1.6/build/cf-conventions.html#standard-name. - the standard_name attributes should comply to the list with standard names, see http://cfconventions.org/26.html.
- a data variable for a 2D grid must depend on three dimensions, representing time and two spatial dimensions (in the order time, y, x). The names of the dimensions can be anything. For the time dimension there must be a corresponding 1D time coordinate variable with the same name. If the grid is regular or rectangular, then for each spatial dimension there must be a corresponding 1D coordinate variable with the same name. See http://cfconventions.org/Data/cf-conventions/cf-conventions-1.6/build/cf-conventions.html#idp5553648. If the grid is curvilinear, then the y and x coordinate variables are both 2D and depend on both spatial dimensions. In this case the data variable must have an attribute "coordinates" that refers to the y and x coordinate variables (in that order), e.g. coordinates="y x". See http://cfconventions.org/Data/cf-conventions/cf-conventions-1.6/build/cf-conventions.html#idp5559280. Each coordinate variable must have an attribute "axis" that indicates which axis is defined by that coordinate variable. For the time coordinate variable the axis must be "T", for the x coordinate variable the axis must be "X", for the y coordinate variable the axis must be "Y".
- if the data has a forecast reference time/analysis time/base time, then this time must be stored in a separate coordinate variable that contains only one value, which is the forecast reference time. This variable must have standard_name="forecast_reference_time". The name of this variable must be present in the coordinates attribute of the data variable, e.g. coordinates="analysis_time" or coordinates="y x analysis_time". If needed, different data variables can point to different forecast reference time variables. See http://cfconventions.org/Data/cf-conventions/cf-conventions-1.6/build/cf-conventions.html#scalar-coordinate-variables
- the time coordinate variable must have an attribute "units" with a valid unit of time and an attribute "axis" with value "T". A valid unit of time is a string formatted as per the recommendations in the Udunits package, e.g. units="minutes since 1970-01-01 00:00:00.0 +0000". See http://cfconventions.org/Data/cf-conventions/cf-conventions-1.6/build/cf-conventions.html#time-coordinate.
- the time and spatial coordinate variables must have values either in ascending or in descending order.
Compacted data
Netcf data variables with attributes scale_factor and add_offset will be treated as compacted data and the values of these attributes will be used to uncompact the values of this variable.
How to import a netCDF file into Delft-FEWS
To import data from a netCDF file into Delft-FEWS you must first create the relevant configuration.
Import configuration
To import data from a netCDF file into Delft-FEWS, either create an import run (see 03 Import Module) or an importNetcdfActivity in a general adapter (see 05 General Adapter Module#05GeneralAdapterModule-importNetcdfActivity).
To import scalar time series you also need to configure the corresponding internal locations (see 01 Locations) and an id mapping (see 01 ID Mapping).
To import grid time series you also need to configure the grid definition in Delft-FEWS, see the next section.
Creating the correct grid definition
For irregular or unstructured grids the cell center (and cell boundaries) must be defined for each grid cell. See 06 Grids#06Grids-Irregulargrids for ways to do this.
For a grid that is regular in a certain coordinate system, follow these steps (also see 06 Grids#06Grids-Regulargrids):
- In the grids.xml config file create an empty regular grid definition.
- Open the netCDF file in your favorite viewer and find the details of the coordinate system in which the grid is a regular grid. This coordinate system is usually described in a grid mapping variable, which has an attribute called "grid_mapping_name". In your grid definition choose the geoDatum or grid projection that corresponds to this coordinate system.
- In the netCDF file find the variables that contain the coordinates of the grid cells in the coordinate system in which the grid is regular. These are one-dimensional variables that usually have standard names like "projection_x_coordinate" and "projection_y_coordinate" or "grid_longitude" and "grid_latitude". According to the CF conventions these coordinates are cell centers. Find the lowest x coordinate and use that as firstCellCenter x in your grid definition. Find the highest y coordinate and use that as firstCellCenter y in your grid definition.
- In the netCDF file look at the spacings between the x and y coordinates. If these spacings are constant, then use them as xCellSize and yCellSize in your grid definition.
- In the netCDF file find the sizes of the dimensions of the coordinate variables. Use these as the number of rows and columns in your grid definition.
Error messages from the Delft-FEWS NetCDF import
Invalid URL
If you get the error "invalid URL" then either the URL is invalid, or the OpenDAP server does not return any data. The latter either means that the OpenDAP server itself does not work correctly, or that the user does not have access rights to download the file on the OpenDAP server. This can be easily verified by following these steps:
- copy the URL of the netcdf file or catalog from the FEWS configuration or from the error message.
- if the URL ends with .nc, then append .html to it, so that it ends with .nc.html
- go to the resulting URL in a browser, e.g. Mozilla Firefox.
- if the browser gives an error, then there is a problem with the URL, the OpenDAP server or user access rights to the server. In this case please contact the administrator of the OpenDAP server (and not the FEWS support team).
Errors from the Java NetCDF library
If you get an error like e.g. "Failed to open NetCdf file from url" or "Variable name (...) must be unique within Group", then the netcdf file is probably corrupt or not CF-1.6 compliant. If the error message is accompanied by a stacktrace (in the FEWS log.txt file) then look for the first part of the stacktrace. Error messages from the Java NetCDF library can be recognised by a line that starts with something like "ucar.nc2.", e.g.:
java.lang.IllegalArgumentException: Variable name (hgtpv34768) must be unique within Group
version: 2014.02, build: 54174 (patched from 50595) May21, type: stable, jre: 1.7.0_40, os:Windows 7, mx: 1037m, db: Firebird
at ucar.nc2.Group.addVariable(Group.java:458)
at ucar.nc2.dods.DODSNetcdfFile.addVariable(DODSNetcdfFile.java:646)
at ucar.nc2.dods.DODSNetcdfFile.constructConstructors(DODSNetcdfFile.java:634)
at ucar.nc2.dods.DODSNetcdfFile.<init>(DODSNetcdfFile.java:277)
at ucar.nc2.dataset.NetcdfDataset.acquireDODS(NetcdfDataset.java:805)
In this case please contact the data supplier that created the NetCDF file and ask them to make the NetCDF file CF-1.6 compliant. If the file is not CF-1.6 compliant then FEWS cannot import the file and there is no reason to contact the FEWS support team.
Related modules and documentation
Internal
- Import Module
- Import Module configuration options
- Import data using OPeNDAP
- Available data types
- NETCDF-CF_GRID
- NETCDF-CF_TIMESERIES
- NETCDF-CF_PROFILE
- NETCDF-CF_TRAJECTORY
External
- http://cfconventions.org/
- https://github.com/ugrid-conventions/ugrid-conventions
- OPeNDAP
- THREDDS
- https://www.unidata.ucar.edu/software/netcdf/docs/BestPractices.html
- https://www.unidata.ucar.edu/software/netcdf/workshops/2012/cf/index.html
- https://www.unidata.ucar.edu/software/netcdf/workshops/2012/index.html
- http://zandmotor.citg.tudelft.nl/netcdfkickstarter/
- netCDF kickstarter