05 General Adapter Module

General Adapter Configuration

A key feature of DELFT-FEWS is its ability to run external modules to provide essential forecasting functionality. These modules may be developed by Deltares as well as by other companies or institutions. The DELFT-FEWS system does not have any knowledge of the specific implementation of these modules. It is rather the central philosophy to have an open system, that is able to treat external modules as plug-ins that can be used if needed.

The General Adapter is the part of the DELFT-FEWS system that implements this feature. It is responsible for the data exchange with the modules and for executing the modules and their adapters. The central philosophy of the General Adapter is that it knows as little as possible of module specific details. Module specific intelligence is strictly separated from the DELFT-FEWS system. In this way an open system can be guaranteed. Module specific intelligence required by the module to run is vested in the module adapters.

Communication between the General Adapter and a module is established through the published interface (PI). The PI is an XML based data interchange format. The General Adapter is configured to provide the data required for a module to run in the PI format. A module adapter is then used to translate the data from the PI to the module native format. Vice versa, results will first be exported to the PI format by a module adapter before the General Adapter imports them back into DELFT-FEWS.

The General Adapter module can be configured to carry out a sequence of five types of tasks;

• Startup Activities. These activities are run prior to a module run and any export import of data. The activities defined are generally used to remove files from previous runs that may implicate the current run.
• Export Activities. These activities define all items to be exported through the published interface XML formats to the external module, prior to the module or the module adapters being initialised.
• Execute Activities. The execute activities define the external executables or Java classes to be run. Tracking of diagnostics from these external activities is included in this section.
• Import Activities: These activities define all items to be imported following successful completion of the module run.
• Shutdown Activities. These activities are run following completion of all other activities. The activities defined are generally used to remove files no longer required.

Figure 65 Schematic interaction between the General Adapter and an external module

When available as configuration on the file system, the name of the XML file for configuring an instance of the general adapter module called for example HBV_Maas_Forecast may be:

HBV_Maas_Forecast 1.00 default.xml

Figure 66 Elements of the General Adapter configuration

general

Root element of general settings.

burnInProfile

Burn-in period and initial value for cold state starts.

activities

Root element for the activities to be defined. The activities are defined in a fixed order;

• startUpActivities
• exportActivities
• executeActivities
• importActivities
• shutDownActivities

General settings

Figure 67 Elements of the general section of the general adapter configuration

description

Optional description of the configuration. Used for reference purposes only.

piVersion

Version of the PI specification that is supported by the pre and post adapter.

rootDir

Root directory for the external module. Other directories can be defined relative to this rootDir using predefined tags (see comment box below).

workDir

Working directory to be used by the external module. When started this directory will be the current directory.

exportDir

Directory to export data from DELFT-FEWS to the external module. All Published Interface files will be written to this directory (unless overruled in naming the specific export files).

exportDataSetDir

Directory to export module datasets from DELFT-FEWS to the external module. A module dataset is a ZIP file, which will be unzipped using this directory as the root directory. If the zip file contains full path information, this will be included as a tree of subdirectories under this directory.

exportIdMap

ID of the IdMap used to convert internal parameterId's and locationId's to external parameter and location Id's. See section on configuration for Mapping Id's units and flags.

exportUnitConversionsId

Id of UnitConversions to be used for export unit mapping

importDir

Directory to import result data from the external module to DELFT-FEWS. All Published Interface files will be read from this directory (unless overruled in naming the specific export files).

importIdMap

ID of the IdMap used to convert external parameterId's and locationId's to internal parameter and location Id's. This may be defined to be the same as the import directory, but may also contain different mappings. See section on configuration for Mapping Id's units and flags.

importUnitConversionsId

Id of UnitConversions to be used for import unit mapping

dumpFileDir

Directory for writing dump files to. Dump Files are created when one of the execute activities fails. A dump file is a ZIP file which includes all the dumpDir directories defined. The dump file is created immediately on failure, meaning that all data and files are available as they are at the time of failure and can be used for analysis purposes. The ZIP file name is time stamped to indicate when it was created.

dumpDir

Directory to be included in the dump file. All contents of the directory will be zipped. Multiple dumpDir's may be defined.

NOTE: ensure that the dumpDir does not include the dumpFileDir. This creates a circular reference and may result in corrupted ZIP files.

diagnosticFile

File name and path of diagnostic files created in running modules. This file should be formatted using the Published Interface diagnostics file specification.

missVal

Optional specification of missing value identifier to be used in PI-XML exported to modules and imported from modules.

NOTE: it is assumed an external uses the same missing value identification for both import and export data.

convertDatum

Optional Boolean flag to indicate level data is used and produced by the module at a global rather than a local datum. The convention in DELFT-FEWS is that data is stored at a local datum. If set to true data in parameter groups supporting datum conversion will be converted on export to the global datum by adding the z coordinate of the location. (see definition of parameters and locations in Regional Configuration).

timeZone

Time zone with reference to UTC (equivalent to GMT) for all time dependent data communicated with the module. If not defined, UTC+0 (GMT) will be used. This time zone is used when importing pi files and the time zone in not available in the pi file.

timeZoneOffset

The offset of the time zone with reference to UTC (equivalent to GMT). Entries should define the number of hours (or fraction of hours) offset. (e.g. +01:00). This time zone is used when importing pi files and the time zone in not available in the pi file.

timeZoneName

Enumeration of supported time zones. See appendix B for list of supported time zones. This time zone is used when importing pi files and the time zone is not available in the pi file.

time0Format

The date time format of the %TIME0% variable
yyyy Year
M Month in year
d Day in month
H Hour in day (0-23)
m Minute in hour
s Second in minute

ensembleMemberCount

Defines if ensembles are read from or written to a number of sub directories.

Burn-In Profile

Burn-in profile for cold state starts. Used to replace first part of a timeseries.

For time series with matching parameter-location ids, the first value is replaced by the initialValue. Element length defines the length of timeseries beginning that is to be replaced using linear interpolation.

length

Length of time series beginning that is to be replaced.

timeSeries

Initial value (which should match cold state), location and parameter should be specified.

Startup Activities

Figure 68 Elements of the startUpActivities section of the General Adapter configuration.

purgeActivity

Root element of a purge activity used to delete files from previous runs. Multiple purge activities may be defined.

filter

Filter specifying files to be removed. Wildcards may be used.

Deleting a whole directory can be achieved by defining the directory path in the filter without any file filter options (.).
eg: %ROOT_DIR%/exportDir/purgeDirectory

A directory can only be removed if it is a sub directory of the General Adapter root directory!

Example (not the use of tags to define the directory names):

unzipActivity

Root element of an unzip activity used to unpack a zip file and put the contained files in the directory of choice. Multiple unzip activities may be defined.

Each activity has the following elements:

• description - optional description of the activity (for documentation only)
• sourceZipFile - the name of the zip file to be unzipped
• destinationDir - the name of the directory where the files will be put

zipActivity

Root element of a zip activity used to pack all files and subdirectories of an indicated directory to a zip file for later use/inspection. Multiple zip activities may be defined.

Each activity has the following elements:

• description - optional description of the activity (for documentation only)
• sourceDir - the name of the directory containing the files to be zipped
• destinationZipFile - the name of the zip file to be created

Example:

<startupActivities>
    <unzipActivity>
        <sourceZipFile>extra_files.zip</sourceZipFile>
        <destinationDir>%ROOT_DIR%/work</destinationDir>
    </unzipActivity>
</startupActivities>
...
<shutdownActivities>
    <zipActivity>
        <sourceDir>%ROOT_DIR%/work</sourceDir>
        <destinationZipFile>%ROOT_DIR%/inspection/saved.zip</destinationZipFile>
    </zipActivity>
</shutdownActivities>

Export Activities

Figure 69 Elements of the ExportActivity section

Export activities are defined to allow exporting various data objects from DELFT-FEWS to the external modules. The list of objects that can be exported (see figure above) includes;

• exportStateActivity to export module states
• exportTimeSeriesActivity to export time series for scalar or polygon time series
• exportMapStacksActivity to export time series for grid time series
• exportProfilesActivity to export time series for longitudinal profile time series
• exportDataSetActivity to export module datasets
• exportParameterActivity to export module parameters
• exportTableActivity to export table (e.g. rating table)
• exportNetcdfActivity to export grid time series in Netcdf format
• exportRunFileActivity to export a run file (The run file contains general information that is used by the pre and post adapter)
• exportAreaSelectionActivity exports bitmasks for the model adapter with a polygon embedded in the TaskProperties as input (is currently only used within NGMS)
• exportLocationAreaActivity exports bitmasks for the model adapter with location ids from the TaskProperties as input (is currently only used within NGMS)

For most types of exportActivity, multiple entries may exist.

NB. See also the exportPlaceholderFile option in the several importActivities, which is another form of preparing input for the module adapter.

exportStateActivity

Figure 70 Elements of the ExportStatesActivity section.

description

Optional description for the export states configuration. Used for reference purposes only.

moduleInstanceId

Id of the moduleInstance that has written the state to be exported. Generally this will be the same as the Id of the current instance of the General Adapter. This can also be the ID of another instance of the General Adapter. The latter is the case when using a state in a forecast run that has been written in an historical run.

stateExportDir

Directory to export the states to. This is the export location for the (binary) state files.

stateConfigFile

Name (and location) of the PI-XML file describing the states. If the directory location is not explicitly specified the file will be written in the exportDir defined in the general section.

stateLocations

Root element for the description of the state. Both a read location and a write location will need to be defined. This allows the name of the file(s)/directory to be different on read and write. Multiple locations may be defined, but these must all be of the same type.

• Attributes type: indication of type of state to be imported. This may either be "directory" or "file". Note that multiple locations are supported only if type is "file".
• stateLocation - Root element of a state location
• readLocation - Location where the external module will read the state. This is the location (and name of file/directory) where the General Adapter writes the state.
• writeLocation - Location where the external module is expected to write the state. This is the location (and name of file/directory) where the General Adapter expects to read the state.
  

<stateLocations type="file">
	<stateLocation>
		<readLocation>state.inp</readLocation>
		<writeLocation>state.out</writeLocation>
	</stateLocation>
</stateLocations>

stateSelection

Root element to specify how a state to be exported to the external module is to be selected. Two main groups are available, cold states and warm states. Only one of these types can be specified. Note that if a warm state selection is specified and an appropriate warm state cannot be found, a cold state will be exported by default.

• coldState - Root element for defining the stateSelection method to always export a cold state.
• groupId - Id of the group of cold states to be used. This must be a groupId as defined in the ColdModuleInstanceStateGroups configuration (see Regional Configuration).
• coldState:startDate - Definition of the start date of the external module run when using the cold state. This startDate is specified relative to the start time of the forecast run. A positive startDate means it is before the start time of the forecast run.
• coldState:fixedStartTime - (Since 2012_02) the start date can be configured as an fixed time
• warmState - Root element for defining the stateSelection method to search for the most suitable warm state.
• stateSearchPeriod - Definition of the search period to be used in selecting a warm state. The database will return the most recent suitable warm state found within this search period.
• coldStateTime - Definition of the start time to use for a cold state if a suitable state is not found within the warm state search period.
• insertColdState - When you set insertColdState to true, the defaultColdState is inserted into the WarmStates when no WarmState is found inside the stateSearchPeriod. By default the cold state is not inserted as warm state

<stateSelection>
        <warmState>
          <stateSearchPeriod unit="hour" start="-48" end="0"/>
	  <coldStateTime unit="hour" value="-48"/>
	  <insertColdState>true</insertColdState>"
	</warmState>
</stateSelection>

loopTimeStep

When specified, all activities are run in a loop to ensure that a state is produced on every cardinal time step between the time of the exported state and T0. This has two advantages:

• states are distributed over time equally and frequently. It is possible to start an update run from every point, also half way of a cold udpate run that spans several days.
• restriction of memory consumption. You can run an update run over months without going out of RAM.

writeIntermediateState

When specified, an extra state is written at the end of the state search period. Note that the run is then split in two. E.g my state search period is -10 to -4 days, then there are two update runs, one from the time where a state was found to -4 and one from -4 to T0. A state is written at the end of both runs (T0 and T0 - 4days). You can additionally define a minimum run length. This is necessary for some runs that need a minimum run length for e.g. PT updating. The run is then only split in two if both runs can be run over the minimum run length. If not, there is only one run and the state is written to the end of this run (T0), no intermediate state is written.

see example configuration below and this figure

<exportStateActivity>
	<moduleInstanceId>HBV_AareBrugg_Hist</moduleInstanceId>
	<stateExportDir>%ROOT_DIR%/FEWS/states</stateExportDir>
	<stateConfigFile>%ROOT_DIR%/FEWS/states/states.xml</stateConfigFile>
	<stateLocations type="file">
		<stateLocation>
			<readLocation>HBV_States.zip</readLocation>
			<writeLocation>HBV_States.zip</writeLocation>
		</stateLocation>
	</stateLocations>
	<stateSelection>
		<warmState>
			<stateSearchPeriod unit="hour" start="-240" end="-96"/>
		</warmState>
	</stateSelection>
	<writeIntermediateState>true</writeIntermediateState>
	<minimumRunLength unit="day" multiplier="4"/>
</exportStateActivity>

ExportTimeSeriesActivity

Figure 71 Elements of the exportTimeSeries section

description

Optional description of the timeSeries export configuration.

exportFile

Name (and location) of the PI-XML file with exported time series. If the directory location is not explicitly specified the file will be written in the exportDir defined in the general section.

exportBinFile

When true the events in the PI time series file are written to a binary file instead of the xml file. The written xml file will only contain the time series headers and optionally a time zone. The binary file has the same name as the xml file only the extension is "bin" instead of "xml". During PI time series import the bin file is automatically read when available. The byte order in the bin file is always Intel x86.

ignoreRunPeriod

When true the run period, written in the pi run file, will not be extended.

includeThresholds

When true any thresholds for the exported timeseries will be written in the timeserie headers

timeSerieSets

Root element for defining timeSerieSets to be exported.

timeSerieSets: timeSerieSet

TimeSeriesSets to be exported. These may contain either a (list of) locations or a locationSet. Multiple entries may be defined.

omitMissingValues

Includes missing values in the export file or leave them out.

omitEmptyTimeSeries

When true, a series is not exported when the time series is empty (or when omitMissingValues = true, when the time series is empty after removing the missing values.)

forecastSelectionPeriod

Can be used to select all approved forecasts with a forecast start time lying within this period

ExportMapStacksActivity

Figure 72 Elements of the ExportMapStacksActivity.

description

Optional description of the timeSeries export configuration.

exportFile

Name (and location) of the PI-XML file describing the map stack of the exported grid time series. If the directory location is not explicitly specified the file will be written in the exportDir defined in the general section.

gridFile

Root element for defining file name of grid to be exported

locationId

LocationId for grid to be exported.

gridName

Name of the files for the grid to be exported. For grid files where each time slice is stored in a different file, this name is the prefix for the full file name. The final file name is created using an index of files exported (e.g the file name for the 4^th^ time step is grid00000.004).

gridFormat

Format of the exported grid. Enumeration of options include;

• asc : for exporting to ARC-INFO ASCII grid format
• pcrgrid : for exporting to PCRaster native grid file format
• usgs : for exporting to USGS DEM format (BIL)
  

  timeSerieSet

TimeSeriesSets to be exported. These should contain only one locationId. For exporting multiple grids, multiple exportMapStack activities should be defined.

exportProfilesActivity

Configuration of the exportProfiles activity is identical to the exportTimeSeries Activity.

ExportDataSetActvity

Figure 73 Elements of the exportDataSets section

description

Optional description of the module dataset export configuration.

moduleInstanceId

Optional reference to the moduleInstanceId of the moduleDataSet to be exported. If not defined the moduleInstanceId of the current module instance is taken as a default (see section on Module Datasets and Parameters).

ExportParameterActivity

Figure 74 Elements of the exportParameter section

description

Optional description of the module parameter configuration.

moduleInstanceId

Optional reference to the moduleInstanceId of the moduleParameter to be exported. If not defined the moduleInstanceId of the current module instance is taken as a default (see section on Module Datasets and Parameters)

fileName

Name (and location) of the PI-XML file with exported parameters. If the directory location is not explicitly specified the file will be written in the exportDir defined in the general section.

ExportTableActivity

Figure 1 Elements of the ExportTableActivity configuration

description

Optional description of the ExportTableActivity configuration.

exportFile

File to which the table will be exported. This file is always placed in exportDir.

tableType

Type of table to be exported. Currently enumeration must be "ratingCurve"

operation

ID of the table to be exported.

parameters

Parameters for the convertEquation operation. Must include minimumLevel, maximumLevel and stepSize

locationId/locationSetId

location id to select the rating curve

exportNetcdfActivity

Figure 2 Elements of the ExportNetcdfActivity configuration

description

Optional description of the ExportNetcdfActivity configuration.

exportFile

File to which the data will be exported. This file is always placed in exportDir.

timeSeriesSets

TimeSeriesSet that defines what data is to be exported.

omitMissingValues

Include missing values in the export file or leave them out.

omitEmptyTimeSeries

The time series is not exported when the time series is empty or when; omitMissingValues = true and the time series is empty after removing the missing values.

ExportRunFileActivity

Figure 3 Elements of the ExportRunFileActivity configuration

description

Optional description of the ExportRunFileActivity configuration.

exportFile

File to which the data will be exported. This file is always placed in exportDir.

properties

Kind of environment variables for the pre and post adapters. These properties are copied to the run file. This is also a convenient way to pass global properties to a pre or post adapter. An adapter is not allowed to access the FEWS global.properties directly. Global properties (between $) are replace by there literal values before copied to the run file. These extra options makes an additional pre or post adapter configuration file unnecessary.

Options:

• string
• int
• float
• bool

example of exported run file

<?xml version="1.0" encoding="UTF-8"?>
<Run xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.wldelft.nl/fews/PI" xsi:schemaLocation="http://www.wldelft.nl/fews/PI http://fews.wldelft.nl/schemas/version1.0/pi-schemas/pi_run.xsd" version="1.5">
	<logLevel>info</logLevel>
    <timeZone>0.0</timeZone>
    <startDateTime date="1900-01-01" time="00:00:00"/>
    <endDateTime date="2100-01-01" time="00:00:00"/>
    <time0 date="2000-01-01" time="00:00:00"/>
    <workDir>workdir</workDir>
    <outputDiagnosticFile>diagnostic</outputDiagnosticFile>
</Run>

ExportAreaSelectionActivity (currently NGMS only)

Exports an area selection mask as a PiMapStack. This activity requires an embedded polygon to be specified within the areaSelectionShapeFileBase64 field of the TaskProperties. This file is then available to be used by the pre and post adapter.

description

Optional description for the activity. Used for reference purposes only.

exportFile

File name of the file to be exported. Always placed into the export dir.

gridLocationId

Locations that contains the reference grid information.

gridFormat

Format of the PiMapStack file. Can be

• asc : for exporting to ARC-INFO ASCII grid format
• pcrgrid : for exporting to PCRaster native grid file format
• usgs : for exporting to USGS DEM format (BIL)

ExportLocationAreaActivity (currently NGMS only)

Exports an area selection mask in the form of a PiMapStack. This activity requires selectedLocationIds to be specified within the TaskProperties. This selection is then available to be used by the pre and post adapter.

description

Optional description for the activity. Used for reference purposes only.

exportFile

File name of the file to be exported. Always placed into the export dir.

gridLocationId

Locations that contains the reference grid information.

gridFormat

Format of the PiMapStack file. Can be

• asc : for exporting to ARC-INFO ASCII grid format
• pcrgrid : for exporting to PCRaster native grid file format
• usgs : for exporting to USGS DEM format (BIL)

polygonLocationId

Identifier for locationset for which to generate grid files. The locations should have polygons associated with them.

Execute Activities

Figure 75 Elements of the ExecuteActivity configuration

executeActivity

Root element for the definition of an execute activity. For each external executable or Java class to run, an executeActivity must be defined. Multiple entries may exist.

description

Optional description for the activity. Used for reference purposes only.

command

Root element to define command to execute.

• executable - File name and location of the executable to run the command is an executable. The file name may include environment variables, as well as tags defined in the general adapter or on the global.properties.
• className - Name of Java Class to run if the command defined as a Java class. This class may be made available to DELFT-FEWS in a separate JAR file in the \Bin directory.
• binDir - Directory with jar files and optionally native dlls. When not specified the bin dir and classloader of FEWS is used. When specified the java class is executed in a private class loader, it will not use any jar in the FEWS bin dir. Only one class loader is created per binDir, adapters should still not use static variables. All dependencies should also be in this configured bin dir.
  

arguments

Root element for defining arguments to be passed to the executable/Java class

• argument - Definition of an argument to be passed to the executable/Java Class
  

environmentVariables

Root element for defining environment variable prior to running the executable/Java class

• environmentVariable - Definition of an environment variable prior to running the executable/Java class
• environmentVariable.name - Name of environment variable
• environmentVariable.value - Value of environment variable
  

timeOut

Optional timeout to be used when running module (in milliseconds). If run time exceeds timeout it will be terminated and the run considered as having failed.

overrulingDiagnosticFile

File containing diagnostic information about activity. This file always is located in the importDir and overrules the global diagnostic file.

ignoreDiagnostics

For this activity no check should be done whether the diagnostics file is present or not.

GA Variables

Several variables are available to be used as an argument to an external program. These are:

Import Activities

Figure 76 Elements of the ImportActivities configuration

description

Optional description of import activity. Used for reference purposes only

importStateActivity

Root element for importing modules states resulting from the run of the external modules. Multiple elements may be defined. If no state is to be imported (for example in forecast run as opposed to state run), then the element should not be defined.

• stateConfigFile - Fully qualifying name of the XML file containing the state import configuration
• expiryTime - When the state is an intermediate result in a forecast run you can let the state expire. By default the expiry time is the same as the module instance run.
• synchLevel - Optional synch level for state. Defaults to 0 is not specified (i.e. same as data generated by the forecast run)
  

stateConfigFile

Name (and location) of the PI-XML file describing the states to be imported. If the directory location is not explicitly specified the file will be expected to be read from the importDir defined in the general section. This file contains all necessary information to define state type and location. The moduleInstanceId of the state imported is per definition the current module instance.

importTimeSeriesActivity

Root element for importing scalar and polygon time series resulting from the run of the external modules. Multiple elements may be defined. importFile and timeSeriesSet should be defined.

• importFile - PI-XML file describing the time series to be imported. The file contains all information on type of data to be imported (scalar, longitudinal, grid, polygon). For all data types except the grid the file also contains the time series data If the directory location is not explicitly specified the file will be expected to be read from the importDir defined in the general section.

importMapStacksActivity

Root element for importing grid time series resulting from the run of the external modules. Multiple elements may be defined. importFile and timeSeriesSet should be defined.

importPiNetcdfActivity

Root element for importing grid time series in Pi-Netcdf format. importFile and timeSeriesSet should be defined.

importProfilesActivity

Root element for importing longitudinal profile time series resulting from the run of the external modules. Multiple elements may be defined. importFile and timeSeriesSet should be defined.

exportPlaceholdFile

If <exportPlaceholderFile>true</exportPlaceholderFile>, then the General adapter will generate placeholder files. A placeholder file is a file with headers only, without timeseries. Its name is the same as filename configured for the ImportActivity and this placeholderfile is written to the import directory. The placeholder files are written before any execute activity is started. The models cq model adapters should read this placeholder files to see which data should be provided to import in FEWS.

• <importTimeSeriesActivity> writes headers to pi_timeseries.xml
• <importMapStacksActivity> writes headers to pi_mapstacks.xml
• <importPiNetcdfActivity> writes headers to NC file
• <importProfilesActivity> writes headers to pi_profiles.xml

The intention behind this exportPlaceholderFile functionality is that the model cq modeladapter reads the placeholders to see which timeseries are required by Fews.
After simulation, the model cq modeladapter overwrites these files with its own data over the placeholder files ready to be imported by the import activity.

<importTimeSeriesActivity>
	<exportPlaceholderFile>true</exportPlaceholderFile>
	<importFile>output.xml</importFile>
	<timeSeriesSets>
        ....
	</timeSeriesSets>
</importTimeSeriesActivity>

It is available in the same way for importTimeSeriesActivity, importMapStacksActivity, importPiNetcdfActivity and importProfilesActivity.

Shutdown Activities

Elements of the Shutdown Activities configuration
This activity is the identical to the startUpActivities. The only difference is that these are carried out after the module run and import of data. See definition of StartUp activities for configuration.