| Excerpt | ||
|---|---|---|
| ||
: Import |
...
data from an OPeNDAP server directly into Delft-FEWS |
Function: | Import data from an OPeNDAP server directly into Delft-FEWS |
|---|---|
Where to Use? | This can be used for importing NetCDF data into the Delft-FEWS system. |
Why to Use? | The advantage of importing NetCDF data directly from an OPeNDAP server, as opposed to importing local NetCDF files, is that the files do not have to be stored locally. Furthermore if only part of a file is needed, then only that part will be downloaded instead of the entire file. This can save a lot of network bandwidth (i.e. time) for large data files. |
Preconditions: | The data to import needs to be available on an OPeNDAP server that is accessible by the Delft-FEWS system. |
Outcome(s): | The imported data will be stored in the Delft-FEWS dataStore. |
Available since: | Delft-FEWS version 2011.02 |
Contents
| Table of Contents |
|---|
Overview
OPeNDAP (Open-source Project for a Network Data Access Protocol) can be used to import NetCDF or GRIB data from an OPeNDAP server directly into Delft-FEWS. For more information on OPeNDAP see http://opendap.org/. Currently only NetCDF files can be imported from an OPeNDAP server. Three types of NetCDF data can be imported: grid time series, scalar time series and profile time series. For more information on these specific import types see their individual pages: NETCDF-CF_GRID, NETCDF-CF_TIMESERIES and NETCDF-CF_PROFILE. Also see NetCDF formats that can be imported in Delft-FEWS and Available data types.
How to import data from an OPeNDAP server
...
import type | usage |
|---|---|
NETCDF-CF_GRID | Use this for importing grid time series that are stored in NetCDF format |
NETCDF-CF_TIMESERIES | Use this for importing scalar time series that are stored in NetCDF format |
NETCDF-CF_PROFILE | Use this for importing profile time series that are stored in NetCDF format |
| GRIB1 | Imports grid time series data from grib1 format used by meteorological institutes. |
| GRIB2 | Imports grid time series data from grib2 format used by meteorological institutes. |
To instruct the import to use OPeNDAP instead of importing local files, specify a server URL instead of a local import folder. Below is an example import configuration with a serverUrl element.
| Code Block | ||||
|---|---|---|---|---|
| ||||
<?xml version="1.0" encoding="UTF-8"?>
<timeSeriesImportRun xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/timeSeriesImportRun.xsd">
<import>
<general>
<importType>NETCDF-CF_GRID</importType>
<serverUrl>http://test.opendap.org/opendap/hyrax/data/nc/sst.mnmean.nc.gz</serverUrl>
<startDateTime date="2007-07-01" time="00:00:00"/>
<endDateTime date="2008-01-01" time="00:00:00"/>
<idMapId>OpendapImportIdMap</idMapId>
<missingValue>32767</missingValue>
</general>
<timeSeriesSet>
<moduleInstanceId>OpendapImport</moduleInstanceId>
<valueType>grid</valueType>
<parameterId>T.obs</parameterId>
<locationId>gridLocation1</locationId>
<timeSeriesType>external historical</timeSeriesType>
<timeStep unit="nonequidistant"/>
<readWriteMode>add originals</readWriteMode>
</timeSeriesSet>
</import>
</timeSeriesImportRun>
|
...
| Note |
|---|
The external parameter id is case sensitive. |
| Code Block | ||||
|---|---|---|---|---|
| ||||
<?xml version="1.0" encoding="UTF-8"?>
<idMap xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/idMap.xsd" version="1.1">
<parameter internal="T.obs" external="sst"/>
<location internal="gridLocation1" external="unknown"/>
</idMap>
|
...
For example to import data from the folder http://test.opendap.org/opendap/hyrax/data/nc/ use the catalog URL http://test.opendap.org/opendap/hyrax/data/nc/catalog.xml in the import configuration. For example:
| Code Block | ||||
|---|---|---|---|---|
| ||||
<import>
<general>
<importType>NETCDF-CF_GRID</importType>
<serverUrl>http://test.opendap.org/opendap/hyrax/data/nc/catalog.xml</serverUrl>
<startDateTime date="2007-07-01" time="00:00:00"/>
<endDateTime date="2008-01-01" time="00:00:00"/>
<idMapId>OpendapImportIdMap</idMapId>
<missingValue>32767</missingValue>
</general>
<timeSeriesSet>
<moduleInstanceId>OpendapImport</moduleInstanceId>
<valueType>grid</valueType>
<parameterId>T.obs</parameterId>
<locationId>gridLocation1</locationId>
<timeSeriesType>external historical</timeSeriesType>
<timeStep unit="nonequidistant"/>
<readWriteMode>add originals</readWriteMode>
</timeSeriesSet>
</import>
|
...
An import file (local or on an OPeNDAP server) can contain multiple variables. For each time series set in the import configuration the import uses the external parameter id from the id map configuration to search for the corresponding variable(s) in the file(s) to import. If a corresponding variable is found, then the data from that variable is imported. Only data for the found variables is downloaded and imported, all other data in the import file(s) is ignored.
For NetCDF files the external parameter id is by default matched to the names of the variables in the NetCDF file to find the required variable to import. There also is an option to use the standard_name attribute or long_name attribute of a variable in the NetCDF file as external parameter id. To use this option add the variable_identification_method property to the import configuration, just above the time series set(s). For example:
| Code Block | ||||
|---|---|---|---|---|
| ||||
<import>
<general>
<importType>NETCDF-CF_GRID</importType>
<serverUrl>http://test.opendap.org/opendap/hyrax/data/nc/sst.mnmean.nc.gz</serverUrl>
<startDateTime date="2007-07-01" time="00:00:00"/>
<endDateTime date="2008-01-01" time="00:00:00"/>
<idMapId>OpendapImportIdMap</idMapId>
<missingValue>32767</missingValue>
</general>
<properties>
<string key="variable_identification_method" value="long_name"/>
</properties>
<timeSeriesSet>
<moduleInstanceId>OpendapImport</moduleInstanceId>
<valueType>grid</valueType>
<parameterId>T.obs</parameterId>
<locationId>gridLocation1</locationId>
<timeSeriesType>external historical</timeSeriesType>
<timeStep unit="nonequidistant"/>
<readWriteMode>add originals</readWriteMode>
</timeSeriesSet>
</import>
|
...
Example: to import only data within the period from 2007-07-01 00:00:00 to 2008-01-01 00:00:00, add the following lines to the import configuration:
| Code Block | ||||
|---|---|---|---|---|
| ||||
<startDateTime date="2007-07-01" time="00:00:00"/>
<endDateTime date="2008-01-01" time="00:00:00"/>
|
...
Example of an import URL with TIME_ZERO tags:
| No Format |
|---|
<serverUrl>http://nomads.ncep.noaa.gov:9090/dods/gfs/gfs%TIME_ZERO(yyyyMMdd)%/gfs_%TIME_ZERO(HH)%z</serverUrl>
|
Example of an import URL with RELATIVE_TIME_IN_SECONDS tags:
| No Format |
|---|
<serverUrl>http://nomads.ncep.noaa.gov:9090/dods/gfs/gfs%RELATIVE_TIME_IN_SECONDS(yyyyMMdd, -18000 )%/gfs_%RELATIVE_TIME_IN_SECONDS(HH,-18000)%z</serverUrl>
|
...
| Note |
|---|
Importing data for a subgrid currently only works for regular grids from NetCDF files. |
This section only applies to the import of grid data. For data with a regular grid that is imported from a NetCDF file, it is in most cases not required to have a grid definition in the grids.xml configuration file. Because for regular grids the import reads the grid definition from the NetCDF file and stores the grid definition directly in the datastore of Delft-FEWS. If for the imported data there is no grid definition present in the grids.xml configuration file, then data for the entire grid is imported.
To import data for only part of the original grid, it is required to specify a grid definition in the grids.xml configuration file. The grid definition defines the part of the grid that needs to be imported. In other words the grid definition defines a subgrid of the original grid. In this case only data for the configured subgrid is downloaded and imported, the data for the rest of the original grid is ignored. The following restrictions apply:
...
For example to import data for a sub grid from the URL http://test.opendap.org/opendap/hyrax/data/nc/sst.mnmean.nc.gz use e.g. the following grid definition in the grids.xml file. In this example a subgrid of 5x5 cells is imported, where the cell center longitude coordinates range from 0 to 8 degrees and the cell center latitude coordinates range from 50 to 58 degrees.
| Code Block | ||||
|---|---|---|---|---|
| ||||
<regular locationId="gridLocation1">
<rows>5</rows>
<columns>5</columns>
<geoDatum>WGS 1984</geoDatum>
<firstCellCenter>
<x>0</x>
<y>58</y>
</firstCellCenter>
<xCellSize>2</xCellSize>
<yCellSize>2</yCellSize>
</regular>
|
...
Example of an import configuration with user and password elements:
| Code Block | ||||
|---|---|---|---|---|
| ||||
<import>
<general>
<importType>NETCDF-CF_GRID</importType>
<serverUrl>http://test.opendap.org/opendap/hyrax/data/nc/sst.mnmean.nc.gz</serverUrl>
<user>kermit</user>
<password>gr33n</password>
<startDateTime date="2007-07-01" time="00:00:00"/>
<endDateTime date="2008-01-01" time="00:00:00"/>
<idMapId>OpendapImportIdMap</idMapId>
<missingValue>32767</missingValue>
</general>
<timeSeriesSet>
<moduleInstanceId>OpendapImport</moduleInstanceId>
<valueType>grid</valueType>
<parameterId>T.obs</parameterId>
<locationId>gridLocation1</locationId>
<timeSeriesType>external historical</timeSeriesType>
<timeStep unit="nonequidistant"/>
<readWriteMode>add originals</readWriteMode>
</timeSeriesSet>
</import>
|
...