...
Workflows that import or process external historical data may produce small time series objects (stored as binary objects,blobs, in the time series database table). These small blobs may only cover a few time steps. If the time series table contains many of these small blobs, the number of records in the database can grow considerable and become very large (>1 mln records). In those cases it is advised to run a database maintenance workflow that amalgamates the small blobs database records in the time series table into larger blobs database records that cover larger time periods and store the time series data in larger blobs. In Delft-FEWS this can be done with the ImportAmalgamate module; this module amalgamates all time series records that are created in specific workflows. After the amalgamate module is finishedcompleted the amalgamate task, the original amalgamated workflows with all its imported external data is scheduled for deletion. The amalgamated time series will be stored in the database as time series that are created by the database maintenance workflow.
...
When the database maintenance workflow (with the amalgamate module) is executed for the first time the workflow can take some time to complete. In those cases it is advised to change the aloowable allowable run time of a workflow run. The fews.master.mc.conf file contains a chaser grace time property, that defines the maximum time that this workflow is allowed to run. Please consider whether this value needs to be temporarily increased.
...
| Code Block | ||||||
|---|---|---|---|---|---|---|
| ||||||
<moduleInstanceDescriptor id="Amalgamate"> <moduleId>ImportAmalgamate</moduleId> </moduleInstanceDescriptor> |
In the module instance configuration the wirkflow ID's can be specified or a workflow pattern can be used when many workflows need to be amalgamated.
| Code Block | ||||||
|---|---|---|---|---|---|---|
| ||||||
<?xml version="1.0" encoding="UTF-8"?>
<importAmalgamate xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/importAmalgamate.xsd">
<workflowId>Import_Data</workflowId>
<workflowId>Procesoverzicht_Update</workflowId>
<workflowIdPattern>*_Process_Observations</workflowIdPattern>
<importRunMinimalAge unit="hour"/>
<amalgamateOrphans>false</amalgamateOrphans>
</importAmalgamate> |
...
| Code Block | ||
|---|---|---|
| ||
<?xml version="1.0" encoding="UTF-8"?> <importAmalgamate xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/importAmalgamate.xsd"> <workflowId>Database_Maintenance</workflowId> <importRunMinimalAge unit="day" multiplier="30"/> <amalgamateOrphans>false</amalgamateOrphans> </importAmalgamate> |
Amalgamate with multiple MC's
Special attention must be given to Delft-FEWS systems that have a duty-standby set-up. The amalgamate module will only amalgamate workflows that are run on the MC where the amalgamate module is scheduled on. In standard duty-dtandby systems the import and data processing tasks are run on both MC's. Also the amalgamate modules (database maintenance workflows) must be scheduled to run on both MC's. This is specially relevant when some of the imported data is synchronised between the MC's.
There has been a change in the Delft-FEWS code since April 2016, when the check on MC's is introduced.