...
Excerpt | ||
---|---|---|
| ||
: Import data from an OPeNDAP server directly into Delft-FEWS |
Function: | Import data from an OPeNDAP server directly into Delft-FEWS |
---|---|
Where to Use? | This can be used for importing data into the Delft-FEWS system. |
Why to Use? | The advantage of importing data directly from an OPeNDAP server, as opposed to importing local files, is that the files do not have to be stored locally. Furthermore if only part of a file is needed, then only that part will be downloaded instead of the entire file. This can save a lot of network bandwidth (i.e. time) for large data files. |
Preconditions: | The data to import needs to be available on an OPeNDAP server that is accessible by the Delft-FEWS system. |
Outcome(s): | The imported data will be stored in the Delft-FEWS dataStore. |
Available since: | Delft-FEWS version 2011.02 |
Contents
Table of Contents |
---|
...
Data can be imported into Delft-FEWS directly from an OPeNDAP server. This can be done using the Import Module. The following import types currently support import using OPeNDAP:
import type | usage |
---|---|
NETCDF-CF_GRID | Use this for importing grid time series that are stored in NetCDF format |
NETCDF-CF_TIMESERIES | Use this for importing scalar time series that are stored in NetCDF format |
NETCDF-CF_PROFILE | Use this for importing profile time series that are stored in NetCDF format |
GRIB1 | Imports grid time series data from grib1 format used by meteorological institutes. |
GRIB2 | Imports grid time series data from grib2 format used by meteorological institutes. |
To instruct the import to use OPeNDAP instead of importing local files, specify a server URL instead of a local import folder. Below is an example import configuration with a serverUrl element.
...
Import data from a single file
To import data from a single file on an OPeNDAP server, the correct URL needs to be configured in the serverUrl element. To get the correct URL for a single file:
- Use a browser to browse to a data file on an OPeNDAP server, e.g. http://test.opendap.org/opendap/hyrax/data/nc/sst.mnmean.nc.gz.html
- Copy the URL that is listed on the page after the keyword "Data URL:", e.g. http://test.opendap.org/opendap/hyrax/data/nc/sst.mnmean.nc.gz
- Paste this URL in the serverUrl element in the import configuration file.
Delft-FEWS version 2022.01 and later are also able to import compressed datasets with extension bz2 from the serverUrl
Import data from a catalog
...
A catalog file is usually called catalog.xml. The URL of a catalog file can be obtained in the following way.
For a THREDDS opendap server: | First browse to a folder on the server. Then copy the current URL from the address line and replace ".html" at the end of the url by ".xml". |
---|---|
For a HYRAX opendap server: | First browse to a folder on the server. Then click on the link "THREDDS Catalog XML" on the bottom of the page. Then copy the current URL from the address line. |
For example to import data from the folder http://test.opendap.org/opendap/hyrax/data/nc/ use the catalog URL http://test.opendap.org/opendap/hyrax/data/nc/catalog.xml in the import configuration. For example:
...
The variable_identification_method property can have the following values:
variable_identification_method | behaviour |
---|---|
standard_name | All external parameter ids are matched to the standard_name attributes of the variables in the NetCDF file to find the required variable(s) to import. |
long_name | All external parameter ids are matched to the long_name attributes of the variables in the NetCDF file to find the required variable(s) to import. |
variable_name | All external parameter ids are matched to the names of the variables in the NetCDF file to find the required variable(s) to import. |
If the variable_identification_method property is not present, then variable_name is used by default. The variable_identification_method property currently only works for the import types NETCDF-CF_GRID, NETCDF-CF_TIMESERIES and NETCDF-CF_PROFILE.
...
Code Block | ||||
---|---|---|---|---|
| ||||
<import> <general> <importType>NETCDF-CF_GRID</importType> <serverUrl>http://test.opendap.orgdummy_hostname/opendap/hyrax/data/nc/sst.mnmean.nc.gz</serverUrl> <user>kermit<<user>dummy_username</user> <password>gr33n<<password>dummy_password</password> <startDateTime date="2007-07-01" time="00:00:00"/> <endDateTime date="2008-01-01" time="00:00:00"/> <idMapId>OpendapImportIdMap</idMapId> <missingValue>32767</missingValue> </general> <timeSeriesSet> <moduleInstanceId>OpendapImport</moduleInstanceId> <valueType>grid</valueType> <parameterId>T.obs</parameterId> <locationId>gridLocation1</locationId> <timeSeriesType>external historical</timeSeriesType> <timeStep unit="nonequidistant"/> <readWriteMode>add originals</readWriteMode> </timeSeriesSet> </import> |
...
To import the certificate file into the truststore use e.g. the following command on the command line:an Operator Client or Stand Alone and press F12.
If it needs to be done via the command line, use the following command
No Format |
---|
GD:\java\jre6jdk11\bin\keytool.exe -import -v -alias aliasName -keystore GD:\FEWS\client.truststore -storepass d3lftf3wsdummy_password -file fileName |
where fileName is the pathname of the certificate file, aliasName is the alias to use for the certificate, G:\java\jre6\bin\keytool.exe is the pathname of the Java keytool.exe file (depends on your Java installation) and G:\FEWS is the path of the Delft-FEWS region home directory (depends on your Delft-FEWS installation). If the file client.truststore does not exist, then the above command will create it. After entering this command, the keytool will display details of the server certificate, type 'yes' to trust the certificate. If the above procedure was successful, then the keytool will display "Certificate was added to keystore". The truststore file called "client.truststore" in the Delft-FEWS region home directory is automatically read each time when Delft-FEWS starts, so Delft-FEWS may need to be restarted after the certificate has been added.
In the above command d3lftf3ws is dummy_password needs to be replaced with the default password for the client.truststore file. By default (obtainable via Delft-FEWS expects the truststore to have this default password. If another password is used for the truststore, then that password Support) or this password must be set as the value of the java system property "javax.net.ssl.trustStorePassword". This is needed so that Delft-FEWS is able to get the password to access the truststore.
Known issues
Export of data
...