Page History

...

When available on the file system, the name of the XML file for configuring a workflow called for example ImportExternal may be:

ImportExternal 1.00 default.xml

ImportExternal Chosen name for this workflow.

1.00 Version number

default Flag to indicate the version is the default configuration (otherwise omitted)(e.g ImportExternal.xml) needs to be registered in the WorkflowDescriptors with Id ImportExternal.

Each processing step in the workflow is referred to as an activity. In defining workflows, several levels of complexity in defining these activities are available;

Simple activities
Activities with a fallback activity;
Activities to be run as an ensemble.
Parallel activities, i.e. a group of activities that can be executed in parallel when multiple cores are available
Sequence activities, i.e. a group of activities that have to be executed in sequence

Activities

Each activity can either be a single moduleInstance or another workflow.

A single moduleInstance can have a one-to-one relation to a module config file (i.e. file name corresponds to moduleInstanceId) or it use a unique moduleInstanceId while referencing a module config file name which is shared as a template by various module instances. In the latter case, properties can be defined in the workflow to make the timeSeriesSets used within each moduleInstanceId unique. A property has a key and a value. When a module config file holds a property-key ($key$) the property-value, as defined in e.g. the workflow, will replace the $key$ when the configuration file is read.

As can be seen in the following images, properties can be defined at varies levels in a workflow. In general the property definition at the lowest level overrules a property defined at a higher level. E.g. a property defined at the module instance level overrules a property defined at the (sub)workflow level overrules a property at the global level (i.e. global.properties file).

Image RemovedImage Added

Figure 142 Elements of the Workflow configuration.

Image Added

Figure 143 Elements of an Activity in the workflow.

activity

Root element for the definition of a workflow activity. Multiple entries can be defined.

...

The ID of the moduleInstance to run as the workflow activity. This module instance must be defined in the moduleInstanceDescriptors (see Regional Configuration) and a suitable module configuration file must be available (see Module configurations). This moduleInsatnceId is used to administer the data generated.

moduleConfigFileName

Optional reference to a module config file that holds the processing logic for a module instance. Typically this module config file holds property-keys such that it can acts as template for multiple module instances, each module instacne having a unique set of property-values.

workflowId

The ID of the (nested) workflow to run as the workflow activity. This workflow must be defined in the worfklowDescriptors (see Regional Configuration) and a suitable workflow must be available (see Module configurations).

properties

Root element for the definition of one or more properties. Multiple entries can be defined. Properties can be externalize portions of a timeSeriesSet definitions (e.g. locationSetId) or parameter values from the module configuration file to the level of the workflow definition.

properties:string

property definition for a string data type. Holds a key-attribute and a value-attribute, where the attribute value is replacing the $key$ as defined in a module config file.

properties:int

property definition for a integer data type. Holds a key-attribute and a value-attribute, where the attribute value is replacing the $key$ as defined in a module config file.

properties:float

property definition for a float data type. Holds a key-attribute and a value-attribute, where the attribute value is replacing the $key$ as defined in a module config file.

properties:double

property definition for a double data type. Holds a key-attribute and a value-attribute, where the attribute value is replacing the $key$ as defined in a module config file.

properties:bool

property definition for a boolean data type. Holds a key-attribute and a value-attribute, where the attribute value is replacing the $key$ as defined in a module config file.

properties:dateTime

property definition for a dateTime data type. Holds a key-attribute while the value is based on the combination of the date-attribute and the time-attribute.

Image Added

fallbackActivity

A fallback activity may be defined to run if the activity under which it is defined fails. This can be used to run a simple module if the more complex method used in preference fails, and ensures continuity of the forecasting process. The definition of the fallbackActivity is the same as the definition of an activity.

...

Info

When running activities as an ensemble that request time series sets from the database that are not a part of that ensemble, the default ensembleId should be added to the TimeSeriesSets definition. The default ensemble Id is "main".

All time series sets written when running in ensemble mode will have the ensembleId as specified in the workflow ensembleId element, unless overruled by a local ensembleId defined in the timeSeriesSet on writing.

workflow:parallel

Grouping to accommodate parallel execution of multiple activities when multiple CPU available. Parallel activities should only be configured when the underlying activities have no data dependency. If data dependency exists, activities should be executed in sequence. This may be forced by embedding them as part of a sequence group.

workflow:sequence

Grouping to enforce sequential execution of multiple activities as they generally have data dependencies. Typically used in workflows where parallel activities are defined to clarify which activities have be to executed in sequence.

Image Added

Figure 143 Elements of an Parallel Activity in the workflow.

Examples

The workflow below runs seven moduleInstances. If the first moduleInstance fails in this example all other processing is stopped. If any of the other activities fail the processing will continue.

Code Block

	xml
	xml


<?xml version="1.0" encoding="UTF-8"?>
<workflow version="1.1" xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews
E:\schemas\http://fews.wldelft.nl/schemas/version1.0/workflow.xsd">
	<activity>
		<runIndependent>false</runIndependent>
		<moduleInstanceId>Astronomical</moduleInstanceId>
	</activity>
	<activity>
		<runIndependent>true</runIndependent>
		<moduleInstanceId>BackupPrecipitation_Forecast</moduleInstanceId>
	</activity>

		<runIndependent>true</runIndependent>
		<moduleInstanceId>PrecipitationGaugeToGrid_Forecast</moduleInstanceId>
	</activity>
	<activity>
		<runIndependent>true</runIndependent>
		<moduleInstanceId>Spatial_Interpolation_Precipitation_Forecast</moduleInstanceId>
	</activity>
	<activity>
		<runIndependent>true</runIndependent>
		<moduleInstanceId>MergePrecipitation_Forecast</moduleInstanceId>
	</activity>
	<activity>
		<runIndependent>true</runIndependent>
		<moduleInstanceId>GridToCatchments_Forecast</moduleInstanceId>
	</activity>
	<activity>
		<runIndependent>true</runIndependent>
		<moduleInstanceId>Singapore_Sobek_Forecast</moduleInstanceId>
	</activity>
</workflow>

The example below is more complex and includes several modules that are run in ensemble mode.

Code Block

	xml
	xml


<?xml version="1.0" encoding="UTF-8"?>
<workflow xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
xsi:schemaLocation="http://www.wldelft.nl/fews 
http://fews.wldelft.nl/schemas/version1.0/workflow.xsd" version="1.1">
	<!--Run Rhein Interpolation -->
	<activity>
		<runIndependent>true</runIndependent>
		<workflowId>Rhein_Interpolate</workflowId>
	</activity>
	<!--Spatial interpolation from grid to HBV-centroids-->
	<activity>
		<runIndependent>true</runIndependent>
		<moduleInstanceId>Rhein_SpatialInterpolationCOSMO-LEPS</moduleInstanceId>
		<ensemble>
			<ensembleId>COSMO-LEPS</ensembleId>
			<runInLoop>true</runInLoop>
		</ensemble>
	</activity>
	<!--Aggregate forecast data for display -->
	<activity>
		<runIndependent>true</runIndependent>
		<moduleInstanceId>Rhein_AggregateForecast_COSMO-LEPS</moduleInstanceId>
		<ensemble>
			<ensembleId>COSMO-LEPS</ensembleId>
			<runInLoop>true</runInLoop>
		</ensemble>
	</activity>
	<!--Disaggregate timeseries at HBV-centroids -->
	<activity>
		<runIndependent>true</runIndependent>
		<moduleInstanceId>Rhein_DisaggregateSeriesCOSMO-LEPS</moduleInstanceId>
		<ensemble>
			<ensembleId>COSMO-LEPS</ensembleId>
			<runInLoop>true</runInLoop>
		</ensemble>
	</activity>
	<!--Merge timeseries from historical run and forecast run -->
	<activity>
		<runIndependent>true</runIndependent>
		<moduleInstanceId>HBV_Rhein_Merge_COSMO-LEPS</moduleInstanceId>
		<ensemble>
			<ensembleId>COSMO-LEPS</ensembleId>
			<runInLoop>true</runInLoop>
		</ensemble>
	</activity>
	<!--Aggregate inputs for display -->
	<activity>
		<runIndependent>true</runIndependent>
		<moduleInstanceId>HBV_Rhein_AggregateInputs_COSMO-LEPS</moduleInstanceId>
		<ensemble>
			<ensembleId>COSMO-LEPS</ensembleId>
			<runInLoop>true</runInLoop>
		</ensemble>
	</activity>
	<!--Interpolate timeseries from historical run and forecast run -->
	<activity>
		<runIndependent>true</runIndependent>
		<moduleInstanceId>HBV_Rhein_Interpolate_COSMO-LEPS</moduleInstanceId>
		<ensemble>
			<ensembleId>COSMO-LEPS</ensembleId>
			<runInLoop>true</runInLoop>
		</ensemble>
	</activity>
	<!--Run HBV-model for forecast period-->
	<activity>
		<runIndependent>true</runIndependent>
		<moduleInstanceId>HBV_Rhein_COSMO-LEPS</moduleInstanceId>
		<ensemble>
			<ensembleId>COSMO-LEPS</ensembleId>
			<runInLoop>true</runInLoop>
		</ensemble>
	</activity>
		<!--Run ErrorModule for forecast period-->
	<activity>
		<runIndependent>true</runIndependent>
		<moduleInstanceId>HBV_Rhein_AR_COSMO-LEPS</moduleInstanceId>
		<ensemble>
			<ensembleId>COSMO-LEPS</ensembleId>
			<runInLoop>true</runInLoop>
		</ensemble>
	</activity>
	<!--Calculate Statistics-->
	<activity>
		<runIndependent>true</runIndependent>
		<workflowId>Statistics_COSMO-LEPS</workflowId>
	</activity>
	<!--Export forecast data to wavos format -->
	<activity>
		<runIndependent>true</runIndependent>
		<moduleInstanceId>Rhein_ExportForecast_COSMO-LEPS</moduleInstanceId>
	</activity>
</workflow>

The example below is complex in a different way as it uses properties to hand over catchment specific information to a report.

Code Block

title	Usage of properties to add catchment specific information to a web-report

<?xml version="1.0" encoding="UTF-8"?>
<workflow version="1.1" xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/workflow.xsd">
    <!--Create Calibration reports-->
    <properties>
        <string key="CATCHMENT" value="armidale"/>
        <string key="CATCHMENT_NAME" value="Armidale"/>
        <string key="BASIN" value="macleay"/>
        <string key="STATE" value="NSW"/>
        <string key="TIMEZONE" value="AET"/>
        <string key="ENSEMBLE" value="OFFICIAL"/>
        <string key="ENSMEMBER" value="URBS Final"/>
    </properties>
    <!--Compute -sub-catchment averaged data as sub-catchment data is not archived-->
    <activity>
        <runIndependent>true</runIndependent>
        <moduleInstanceId>Processing_15m_Site_Calibration</moduleInstanceId>
    </activity>
    <activity>
        <runIndependent>true</runIndependent>
        <moduleInstanceId>Performance_15m_Calibration</moduleInstanceId>
    </activity>
    <activity>
        <runIndependent>true</runIndependent>
        <moduleInstanceId>Report_15m_Site_Calibration</moduleInstanceId>
    </activity>
     <activity>
        <runIndependent>true</runIndependent>
        <moduleInstanceId>Report_Catchment_Calibration</moduleInstanceId>
    </activity> 
</workflow>

The following workflow example uses properties to make a apply the logic, encapsulated in general module config file, to a specific catchment. The second code block shows how some of the properties are used to make timeseriessets in the Rainfall_15m_Forecast module config file unique for the locations of a certain catchment.

Code Block

title	Workflow using properties to apply general processing logic to a specific catchment

<?xml version="1.0" encoding="UTF-8"?>
<workflow xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/workflow.xsd" version="1.1">

    <properties>
        <string key="RADAR" value="noradar"/>
        <string key="CATCHMENT" value="armidale"/>
        <string key="BASIN" value="macleay"/>
        <string key="ENSEMBLE" value="IFD"/>
        <string key="ENSMEMBER" value="URBS Local"/>
    </properties>
    <activity>
        <runIndependent>true</runIndependent>
        <moduleInstanceId>armidale_Rainfall_Forecast</moduleInstanceId>
        <moduleConfigFileName>Rainfall_15m_Forecast</moduleConfigFileName>
    </activity>
    <activity>
        <runIndependent>true</runIndependent>
        <workflowId>armidale_URBS_Catchment_Forecast</workflowId>
        <ensemble>
            <ensembleId>IFD</ensembleId>
            <runInLoop>true</runInLoop>
        </ensemble>
    </activity>
</workflow>

Code Block

title	Usage of properties in timeseries definition of the Rainfall_15m_Forecast module config file

    <variable>
        <variableId>ACCESS_G_subarea_3h</variableId>
        <timeSeriesSet>
            <moduleInstanceId>$CATCHMENT$_Rainfall_Forecast</moduleInstanceId>
            <valueType>scalar</valueType>
            <parameterId>P.nwp.fcst</parameterId>
            <qualifierId>ACCESS_G</qualifierId>
            <locationSetId>URBS_subareas.$CATCHMENT$</locationSetId>
            <timeSeriesType>temporary</timeSeriesType>
            <timeStep unit="hour" multiplier="3"/>
            <readWriteMode>read complete forecast</readWriteMode>
        </timeSeriesSet>
    </variable>

Page tree

Versions Compared

Old Version 19

New Version 20

Key

Activities

activity

moduleConfigFileName

workflowId

properties

properties:string

properties:int

properties:float

properties:double

properties:bool

properties:dateTime

fallbackActivity

workflow:parallel

workflow:sequence

Examples