in progress
netCDF/OPeNDAP are the perfect web file format/web service for gridded data. Gridded data exist in various types.
Single grid vs. tiled grids
- Single grid coverages: one file contains all the data of the entire space covered by the dataset.
- A set of tiled grid coverages: one file contains only the data of a subset of the entire space covered by the dataset. The entire space is . Example:
http://opendap.deltares.nl/thredds/catalog/opendap/rijkswaterstaat/jarkus/grids/catalog.html
http://opendap.deltares.nl/thredds/catalog/opendap/rijkswaterstaat/vaklodingen/catalog.html
http://opendap.deltares.nl/thredds/catalog/opendap/rijkswaterstaat/kustlidar/catalog.html
For Matlab Openearth developed a crawler that can gets the names of all netCDF files that comprise a tile by interpreting the THREDDS catalog.xml
>> url = 'http://opendap.deltares.nl/thredds/catalog/opendap/rijkswaterstaat/vaklodingen/catalog.xml' >> L = opendap_catalog(url) >> disp(L{1}) ans = http://opendap.deltares.nl/thredds/dodsC/opendap/rijkswaterstaat/vaklodingen/vaklodingenKB138_0706.nc >> nc_dump(L{1})
Grid topology
- Orthogonal grid
- An orthogonal lat-lon grid: climate data products often appear in this topology. There is a separate lat vector (e.g. -180:1:180 deg) and a separate lon vector (e.g. -90:1:90 deg) that span a part of the globe. Support for these grids is always available. Example:
http://opendap.deltares.nl/thredds/catalog/opendap/deltares/FEWS-IPCC/catalog.html
- An orthogonal x-y grid: national data products often appear in this topology. There is a separate x vector (e.g. 0:100:280000 m)
and a separate y vector (e.g. 624800:-100:300000) that span a part in a local projection. Support to plot these grids in lat-lon space is less often available, as it assumes that the client can deal with the two methods to prescribe lat-lon coordinates. The first, implicit option is assuming that the client can do the conversion from the local projection to the a WGS84 lat-lon projection. The CF convention describes how to add projection meta-data to the netCDF file (grid_mapping attribute). In addition, the data producer can transform the x and y vectors to complete lat and lon grids, prescribing the WGS84 position of each grid cell separately. The CF convention describes how to attach these global coordinates to the dataset (coordinates attribute). OpenEarth recommend to supply both options in netCDF files. This option is in fact the same as considering the orthogonal x-y grid as a curvi-linear lat-lon grid, as discussed next. Example:
http://opendap.deltares.nl/thredds/catalog/opendap/tno/ahn100m/catalog.html with clearly defined local projection (Dutch RD) and with full lat-lon matrices included.
http://opendap.deltares.nl/thredds/catalog/opendap/knmi/NOAA/mom/1990_mom/5/catalog.html without clearly defined local projection (a polar stereographic one) but with full lat-lon matrices included.
- An orthogonal lat-lon grid: climate data products often appear in this topology. There is a separate lat vector (e.g. -180:1:180 deg) and a separate lon vector (e.g. -90:1:90 deg) that span a part of the globe. Support for these grids is always available. Example:
- Curvi-linear grid: plotting support for curvi-linear grids depends on the client. Curvi-linear grids often exist, key examples are: individual satellite images that are orthogonal in pixel space, but fully warped in lat-lon space and (ii) curvi-linear coastal circulation models that are orthogonal in grid index space, but that have been warped to fit local coast lines.
- A curvi-linear lat-lon mesh: Example: remote sensing swath
http://data.nodc.noaa.gov/opendap/ghrsst/L2P/AVHRR19_L/NEODAAS/2010/348/20101214-AVHRR19_L-NEODAAS-L2P-14dec100129_wsst.8bit-v01.nc.bz2
- A curvi-linear x-y mesh: This kind of data is the regular output type of regional ocean circulation models. The CF convention allows to attach only one set of coordinates, either the lat-lon matrices or the x-y matrices. Example:
- A curvi-linear lat-lon mesh: Example: remote sensing swath
- Unstructured grid: Support for unstructured grids has not been standardized. A proposals can be found at the Deltares netCDF wiki
Time dependency
Most 'GIS'-minded people produce grids without a time dimension, but simply with some kind of time annotation. This does not easily allow to automatic processing of temporal data. Instead, model output, spatially binned remote sensing products and repeated depth sounding products do produce datasets with a clear and explicit time dimension.
- 'GIS' map. Example
http://opendap.deltares.nl/thredds/catalog/opendap/rijkswaterstaat/vaklodingen/catalog.html - 'GIS' map with added dummy time dimension. OpenEarth recommends that also 'GIS' maps always insert a dummy time dimension so 'GIS' data can be processed like model data or remote sensing products. Example: single remote sensing map valid at a specific moment in time.
http://opendap.deltares.nl/thredds/catalog/opendap/knmi/NOAA/mom/1990_mom/5/catalog.html
For datasets gathered over a longer period of time the CF convention allows to also to specify time bounds (begin and end time) rather than a specific 'snap-shot' time. - Full spatio-temporal datasets with explicit time dimension. Example:
http://opendap.deltares.nl/thredds/catalog/opendap/rijkswaterstaat/vaklodingen/catalog.html
Password restrictions
- without: example
http://opendap.deltares.nl/thredds/catalog/ - with password. Not all client support password protection. All internet browsers do, so you can always browse pass-word protected datasats. The snctools toolbox shipped with OpenEarthTools allows for password encryption also of direct data request from Matlab. A 26 line open source piece of java code is used for this.
http://matroos.deltares.nl:8080/thredds/