scrollbar |
---|
Table of Contents |
---|
Ingest datasets in Delft-FEWS stand alone - ImportArchiveModule
Once downloaded, the data can be ingested by a Delft-FEWS stand alone with the aim to bring the local datastore back in the 'original state', i.e. with a similar time series set definition as used when Delft-FEWS ran the workflow to produce the data. This exact timeseries definition is included in the netcdf files during the archive export process. The importArchiveModule has been implemented to uses this definition included in the netcdf files to put the data back in Delft-FEWS local datastore
...
Table 4.10 importArchiveModule configuration
Element | Format | Description |
importSimulated/importFolder | string (path) | Full path where the simulated datasets are made available for import by Delft-FEWS |
importObserved/importFolder | string (path) | Full path where the observed datasets are made available for import by Delft-FEWS |
importMessages/importFolder | string (path) | Full path where the messages datasets are made available for import by Delft-FEWS |
importExternalForecast/importFolder | string (path) | Full path where the external forecasts are made available for import by Delft-FEWS |
importRatingCurves/importFolder | string (path) | Full path where the rating curve datasets are made available for import by Delft-FEWS |
...
IdMapping of locations that have changed id over time
If the id of locations used in the Delft-FEWS system have changed over time (e.g. from locationId_old to locationId_new), the data can still be imported from the archive. For this you need to configure a idMap, just like your used to with the normal import Module. However, in this case you will need to make sure that the old id (locationId_old) is still known as a location in the configuration. You can either add the old id to Locations.xml or create a csv file with this (and potentially other) deprecated locationId's. The benefit of the second method is that you can add the new location id to this file, so you can use a function to map the id's instead of defining them by hand in the idMap configuration file.
Ingest of Historic events in Delft-FEWS (FSS)
...
Historic events are a special datatype to Delft-FEWS as they can be used to overlay on an existing time series graph. Historic events are the only datasets which can be ingested in a Delft-FEWS client server system. Normally, the archive server has a process running (the HistoricEventsExporter) which extracts historic events data from the archive and pushes them to the Forecasting Shell Server import directories for ingest in the operational database.
...
Table 4.11 Historical events import configuration (importArchiveModule)
Element | Format | Description |
importHistoricalEvents | importHistoricalEvents ComplexType | root element for import of Historical events into Delft-FEWS database |
importHistoricalEvents ComplexType |
importFolder | string (path) | Full path where the historical events datasets are made available for import by Delft-FEWS |
failedFolder | string (path | Full path where Delft-FEWS can put any dataset which failed on import |
backupFolder | string (path | Full path where Delft-FEWS can back up any dataset which is being imported |
idMapId | string | Idmap to translate NetCDF variables to FEWS parameters/locations |
Import data from the Open Archive in a workflow
It is possible to schedule a workflow which imports data from the archive based on a list of data sets which is provided in a file. Below an example of the configuration of such a file.
Code Block |
---|
<importArchiveModule xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/importArchiveModule.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://www.wldelft.nl/fews">
<importRequestedDataSets>
<importFolder>$IMPORT_FOLDER$</importFolder>
<dataFolderArchive>$ARCHIVE_DOWNLOAD_FOLDER$</dataFolderArchive>
</importRequestedDataSets>
</importArchiveModule>
|
The importFolder tag should contain the file which define which datasets should be imported. The dataFolderArchive tag defines the location of the root of the data folder of the archive.
Below an example of how the files which define which datasets should be imported should look like.
Code Block |
---|
<?xml version="1.0" encoding="UTF-8"?>
<dataSets xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.wldelft.nl/fews/archive" xsi:schemaLocation="http://www.wldelft.nl/fews/archive http://fews.wldelft.nl/schemas//version1.0/archive-schemas/dataImport.xsd">
<dataSet>dataSet1\metaData.xml</dataSet>
<dataSet>dataSet2\metaData.xml</dataSet>
</dataSets> |
In the Open Archive each dataset has its own metaData.xml file. The import task will import each dataset which is defined in the list entirely. The import folder can contain multiple files. If that is the case the files will be imported one by one in a random order.
The files can be provided by writing them directly into the defined import folder. Another option is to upload the files. For this a dedicated webservice is available in the archive.