Page History

...

Within Delft-FEWS, the datasets are exported to the archive in a workflow using the ExportArchiveModule. This workflow needs to be scheduled on a regular interval to archive all relevant data. Simulated data sets are preferably exported to the archive from the workflow which that created the simulation. This is especially the case when modifiers are used in the simulation. When a simulation is not exported to the archive directly, it might be that some modifiers which are that were used in the simulation are deleted are or changed in between. This will cause that it won't be possible to archived all the used modifiers correctlythe mean time. This can then result in archived simulation data without the modifiers that were used to produce them.

To prevent dependencies on other processes, the ExportArchiveModule is envisioned to write directly into the archive file storage. The FSS thus needs to be able to have write access to those disks.

For each kind of dataset, the ExportArchiveModule checks the database for changes over a (configured) relative period. It exports any all data which that meets the export instructions and that has changed within this period. Datasets are archived in a pre-defined predefined directory structure, which is based on areaId, date and dataset name.

The schema of the associated configuration file (Figure 4.1) is defined at:
http://fews.wldelft.nl/schemas/version1.0/exportArchiveModule.xsd.

Figure 4.1 Top level of Delft-FEWS exportArchiveModule.xsd

...

Archiving by using time series sets or workflow selection

The FEWS archive export exporting module has several exports in which data can be archivedoptions for exporting data. In all those exports, time series can be selected by either specifying the time series by using time series sets or it is possible to define by specifying a workflow in combination with a time series filter. In this case When specifying a workflow, all the time series which that comply to with the defined filter will be exported. This filter can be used to select the desired data type, parameter, location, qualifier, etc. for exporting. By default the filter will be applied to the time series of the current workflow. If a workflow id workflowId is defined then the data from the last or current run will be used. Or if One can also define a period is defined then , whereby all simulations or external forecasts for the defined that period will are be exported. 

Generated file name

The path paths of the folders to which the data will be archived will not differ between the two options. 

Generated file name

written is the same for both options, following the default folder structure for the respective data type. 

When exports are setup by specifying timeseries, one must configure the filename of the exported file.  If the workflow selection option is used to select which time series will be archived, it is not necessary to define the file names of the netcdf files in which the data will be archived. These names will be generated based on the timeseries which will be that are stored in the file.   The generated filename will be determined as follows.: 

<parameterid>_<qualifierIds>_<timestep>_<valuetype>.nc

A file containing scalar data for parameter Hobs with qualifier Q and time step SETS60 will be stored in file Hobs_Q_SETS66SETS60_scalar.nc.  It is also possible to configure a fixed prefix for each generated file name.

If timeseries only differ by module instance id then the filename is extended with the module instance id.  

Example

Below an example of an export for a workflow with a time series selection through a filter. In the workflowTimeSeriesSelection section, one must specify at least one filter and one workflowId (optional).

<exportArchiveModule xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/exportArchiveModule.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.wldelft.nl/fews">
	<exportExternalForecast>
		<general>
			<archiveFolder>$ARCHIVE_DIR$</archiveFolder>
			<idMapId>IdExportArchiveExternalForecasts</idMapId>
		</general>
		<activities>
			<netcdfExportActivities>
				<netcdfExportActivity>
					<fileNamePrefix>Deltares</fileNamePrefix>
					<areaId>test</areaId>
					<sourceId>test</sourceId>
					<includeComments>false</includeComments>
					<includeFlags>false</includeFlags>
					<useModuleInstanceIdAsSourceId>true</useModuleInstanceIdAsSourceId>
					<workflowTimeSeriesSelection>
						<timeSeriesFilter>
							<parameterId>Q</parameterId>
						</timeSeriesFilter>
					    <workflowId>some_sub_workflow</workflowId>      
					</workflowTimeSeriesSelection>
				</netcdfExportActivity>
			</netcdfExportActivities>
		</activities>
	</exportExternalForecast>
</exportArchiveModule>

...

...