24. ImportAmalgamate

Description

Workflows may produce inefficient blobs that only span a few time steps. These blobs will be amalgamated. After the amalgamate is finished, the original import runs with all its imported external data is scheduled for deletion. Large grids, external forecasts and samples should be imported in a separate workflow, that are not handled through this module.

Configuration

workflowId

One or more work flow ids that import external historical time series over a short time span (scheduled frequently).

expiry time.

The original (import) runs are deleted after successful amalgamation. To make sure the run is successful amalgamated and not removed by the rolling barrel it is best to configure a large expiry time for the amalgamated workflows in the workflowDescriptors.xml. Configure an explicit expiry time for the time series sets in the amalgamated workflows. The expiry time of the individual time series is preserved while amalgamated.

importRunMinimalAge

Import runs younger than the specified age are skipped. After the amalgamate has runned it is no longer possible to create an archive with the exact original data available during the run.

Example

<moduleDescriptor id="ImportAmalgamate">
  <className>nl.wldelft.fews.system.plugin.amalgamate.ImportAmalgamate</className>
</moduleDescriptor>

<moduleInstanceDescriptor id="Amalgamate">
  <moduleId>ImportAmalgamate</moduleId>
</moduleInstanceDescriptor>

<?xml version="1.0" encoding="UTF-8"?>
<importAmalgamate xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/importAmalgamate.xsd">
  <workflowId>Import_Data</workflowId>
  <workflowId>Procesoverzicht_Update</workflowId>
  <importRunMinimalAge unit="hour"/>
</importAmalgamate>