Page History

...

To instruct the import to use OPeNDAP instead of importing local files, specify a server URL instead of an a local import folder. Below is an example import configuration with a serverUrl element.

<?xml version="1.0" encoding="UTF-8"?>
<timeSeriesImportRun xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/timeSeriesImportRun.xsd">
	<import>
		<general>
			<importType>NETCDF-CF_GRID</importType>
			<serverUrl>http://test.opendap.org/opendap/hyrax/data/nc/sst.mnmean.nc.gz</serverUrl>
			<startDateTime date="2007-07-01" time="00:00:00"/>
			<endDateTime date="2008-01-01" time="00:00:00"/>
			<idMapId>OpendapImportIdMap</idMapId>
			<missingValue>32767</missingValue>
		</general>
		<timeSeriesSet>
			<moduleInstanceId>OpendapImport</moduleInstanceId>
			<valueType>grid</valueType>
			<parameterId>T.obs</parameterId>
			<locationId>gridLocation1</locationId>
			<timeSeriesType>external historical</timeSeriesType>
			<timeStep unit="nonequidistant"/>
			<readWriteMode>add originals</readWriteMode>
		</timeSeriesSet>
	</import>
</timeSeriesImportRun>

The Here the serverURL is the URL of a file on an OPeNDAP server. For details on specifying the URL see Import data from a single file below. The time series set(s) define what data should be imported into Delft-FEWS. Only data for the configured time series sets is downloaded and imported, all other data in the import file(s) is ignored. For more details see Import Module configuration options.

Id map configuration

The import also needs an id map configuration file, that contains a mapping between the time series sets in the import configuration and the variables in the file(s) to import. For each time series set in the import configuration the import searches for the corresponding variable(s) in the file(s) to import. If a corresponding variable is found, then the data from that variable is imported. Only data for the configured time series sets is downloaded and imported, all other data in the import file(s) is ignored. Below is an example id map configuration.

Below is an example id map configuration.

<?xml version="1.0" encoding="UTF-8"?>
<idMap xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/idMap.xsd" version="1.1">
	<parameter internal="T.obs" external="sst"/>
	<location internal="gridLocation1" external="unknown"/>
</idMap>

A NetCDF file can contain multiple variables. By default the external parameter id is matched to the names of the variables in the NetCDF file to find the required variable to import. There also is an option to use the standard_name attribute or long_name attribute of the variable in the NetCDF file as external parameter id. To use this option add the variable_identification_method property to the import configuration, just above the time series sets. For example:

...

Import data from a single file

To import data from a single file on an OPeNDAP server, the correct URL needs to be configured in the serverUrl element. To get the correct URL for a single file:
1. Use a browser to browse to a data file on an OPeNDAP server, e.g. http://test.opendap.org/opendap/hyrax/data/nc/sst.mnmean.nc

...

.gz.html
2. Copy the URL that is listed on the page after the keyword "Data URL:", e.g. http://test.opendap.org/opendap/hyrax/data/nc/sst.mnmean.nc.gz
3. Paste this URL in the serverUrl element in the import configuration file.

Import data from a catalog

Instead of specifying the URL of a single file on an OPeNDAP server, it is also possible to specify the URL of a catalog. The files on an OPeNDAP server are usually grouped in folders and for each folder there is a catalog file available. The catalog usually contains a list of files and subfolders, but can also refer to other catalog files. If the URL of a catalog file is specified for the import, then all files that are listed in the catalog will be imported. Other catalogs that are listed in the specified catalog are also imported recursively.

A catalog file is usually called catalog.xml. The URL of a catalog file can be obtained in the following way.

For example to import data from the folder http://test.opendap.org/opendap/hyrax/data/nc/ use the catalog file http://test.opendap.org/opendap/hyrax/data/nc/catalog.xml in the following import configuration:

<import>
	<general>
		<importType>NETCDF-CF_GRID</importType>
		<serverUrl>http

The value can be "standard_name", "long_name" or "variable_name" (default). If value is e.g. "standard_name", then the external parameter id from the id map configuration file is matched to the standard_name attributes of the variables in the NetCDF file to find the required variable to import. If the variable_identification_method property is not present, then "variable_name" is used by default. The variable_identification_method property currently only works for import types "NETCDF-CF_GRID", "NETCDF-CF_TIMESERIES" and "NETCDF-CF_PROFILE".

Import data from a single file

TODO

Import data from a catalog

Instead of specifying the URL of a single file on an OPeNDAP server, it is also possible to specify the URL of a catalog. The files on an OPeNDAP server are usually grouped in folders and for each folder there is a catalog file available. The catalog usually contains a list of files and subfolders, but can also refer to other catalog files. If the URL of a catalog file is specified for the import, then all files that are listed in the catalog will be imported. Other catalogs that are listed in the specified catalog are also imported recursively.

A catalog file is usually called catalog.xml. The URL of a catalog file can be obtained in the following way.

...

://test.opendap.org/opendap/hyrax/data/nc/

...

catalog

...

<import>
	<general>
		<importType>NETCDF-CF_GRID</importType>
		<serverUrl>http://test.opendap.org/opendap/hyrax/data/nc/catalog.xml</serverUrl>
		<startDateTime date="2007-07-01" time=".xml</serverUrl>
		<startDateTime date="2007-07-01" time="00:00:00"/>
		<endDateTime date="2008-01-01" time="00:00:00"/>
		<idMapId>OpendapImportIdMap</idMapId>
		<missingValue>32767</missingValue>
	</general>
	<timeSeriesSet>
		<moduleInstanceId>OpendapImport</moduleInstanceId>
		<valueType>grid</valueType>
		<parameterId>T.obs</parameterId>
		<locationId>gridLocation1</locationId>
		<timeSeriesType>external historical</timeSeriesType>
		<timeStep unit="nonequidistant"/>
		<readWriteMode>add originals</readWriteMode>
	</timeSeriesSet>
</import>

Import data for a given variable

An import file (local or on an OPeNDAP server) can contain multiple variables. For each time series set in the import configuration the import uses the external parameter id from the id map configuration to search for the corresponding variable(s) in the file(s) to import. If a corresponding variable is found, then the data from that variable is imported. Only data for the configured time series sets is downloaded and imported, all other data in the import file(s) is ignored.

For NetCDF files the external parameter id is by default matched to the names of the variables in the NetCDF file to find the required variable to import. There also is an option to use the standard_name attribute or long_name attribute of the variable in the NetCDF file as external parameter id. To use this option add the variable_identification_method property to the import configuration, just above the time series set(s). For example:

<import>
	<general>
		<importType>NETCDF-CF_GRID</importType>
		<serverUrl>http://test.opendap.org/opendap/hyrax/data/nc/sst.mnmean.nc.gz</serverUrl>
		<startDateTime date="2007-07-01" time="00:00:00"/>
		<endDateTime date="2008-01-01" time="00:00:00"/>
		<idMapId>OpendapImportIdMap</idMapId>
		<missingValue>32767</missingValue>
	</general>
	<properties>
		<string key="variable_identification_method" value="long_name"/>
	</properties>
	<timeSeriesSet>
		<moduleInstanceId>OpendapImport</moduleInstanceId>
		<valueType>grid</valueType>
		<parameterId>T.obs</parameterId>
		<locationId>gridLocation1</locationId>
		<timeSeriesType>external historical</timeSeriesType>
		<timeStep unit="nonequidistant"/>
		<readWriteMode>add originals</readWriteMode>
	</timeSeriesSet>
</import>

The variable_identification_method property can have the following values:

If the variable_identification_method property is not present, then the "variable_name" is used by default. The variable_identification_method property currently only works for import types "NETCDF-CF_GRID", "NETCDF-CF_TIMESERIES" and "NETCDF-CF_PROFILE".

Import data for a given period of time

...