What | nameofinstance.xml |
|---|---|
Description | Configuration for the general adapter module |
schema location | http://fews.wldelft.nl/schemas/version1.0/generalAdapterRun.xsd |
Entry in ModuleDescriptors | <moduleDescriptor id="GeneralAdapter"> |
A key feature of Delft-FEWS is its ability to run external modules to provide essential forecasting functionality. These modules may be developed by Deltares as well as by other companies or institutions. The Delft-FEWS system does not have any knowledge of the specific implementation of these modules. It is rather the central philosophy to have an open system, that is able to treat external modules as plug-ins that can be used if needed.
The General Adapter is the part of the Delft-FEWS system that implements this feature. It is responsible for the data exchange with the modules and for executing the modules and their adapters. The central philosophy of the General Adapter is that it knows as little as possible of module specific details. Module specific intelligence is strictly separated from the Delft-FEWS system. In this way an open system can be guaranteed. Module specific intelligence required by the module to run is vested in the module adapters.
Communication between the General Adapter and a module is established through the published interface (PI). The PI is an XML based data interchange format. The General Adapter is configured to provide the data required for a module to run in the PI format. A module adapter is then used to translate the data from the PI to the module native format. Vice versa, results will first be exported to the PI format by a module adapter before the General Adapter imports them back into Delft-FEWS.
Grid and scalar timeseries can also be exchanged using netCDF format.
This exchange format is intended for the modules that obtain their input from netCDF files and/or write their output to netCDF files.
The General Adapter module can be configured to carry out a sequence of five types of tasks;
Figure 65 Schematic interaction between the General Adapter and an external module
When available as configuration on the file system, the name of the XML file for configuring an instance of the general adapter module called for example HBV_Maas_Forecast may be:
HBV_Maas_Forecast 1.00 default.xml
HBV_Maas_Forecast | File name for the HBV_Maas_Forecast configuration. |
1.00 | Version number |
default | Flag to indicate the version is the default configuration (otherwise omitted). |
Figure 66 Elements of the General Adapter configuration
Root element of general settings.
Burn-in period and initial value for cold state starts.
Root element for the activities to be defined. The activities are defined in a fixed order;
Figure 67 Elements of the general section of the general adapter configuration
Optional description of the configuration. Used for reference purposes only.
Version of the PI specification that is supported by the pre and post adapter.
Root directory for the external module. Other directories can be defined relative to this rootDir using predefined tags (see comment box below).
Working directory to be used by the external module. When started this directory will be the current directory.
Directory to export data from Delft-FEWS to the external module. All Published Interface files will be written to this directory (unless overruled in naming the specific export files).
Directory to export module datasets from Delft-FEWS to the external module. A module dataset is a ZIP file, which will be unzipped using this directory as the root directory. If the zip file contains full path information, this will be included as a tree of subdirectories under this directory.
If set to 'true' datasets are only updated when they have changed. Change is detected by conparing the timestamp in the dataset info file with the modification time of the dataset configuration file. By default this value is 'false' and datasets are always updated.
If a dataset has been detected as updated then this option allows user to purge content of existing moduledataset directory before new moduledataset is exported. By default the value is 'false' and all existing dataset content is overwritten.
ID of the IdMap used to convert internal parameterId's and locationId's to external parameter and location Id's. See section on configuration for Mapping Id's units and flags.
Id of UnitConversions to be used for export unit mapping
Directory to import result data from the external module to Delft-FEWS. All Published Interface files will be read from this directory (unless overruled in naming the specific export files).
ID of the IdMap used to convert external parameterId's and locationId's to internal parameter and location Id's. This may be defined to be the same as the import directory, but may also contain different mappings. See section on configuration for Mapping Id's units and flags.
Id of UnitConversions to be used for import unit mapping
Directory for writing dump files to. Dump Files are created when one of the execute activities fails. A dump file is a ZIP file which includes all the dumpDir directories defined. The dump file is created immediately on failure, meaning that all data and files are available as they are at the time of failure and can be used for analysis purposes. The ZIP file name is time stamped to indicate when it was created.
Directory to be included in the dump file. All contents of the directory will be zipped. Multiple dumpDir's may be defined.
NOTE: ensure that the dumpDir does not include the dumpFileDir. This creates a circular reference and may result in corrupted ZIP files.
File name and path of diagnostic files created in running modules. This file should be formatted using the Published Interface diagnostics file specification.
Optional specification of missing value identifier to be used in PI-XML exported to modules and imported from modules.
NOTE: it is assumed an external module uses the same missing value identification for both import and export data.
Optional Boolean flag to indicate level data is used and produced by the module at a global rather than a local datum. The convention in Delft-FEWS is that data is stored at a local datum. If set to true data in parameter groups supporting datum conversion will be converted on export to the global datum by adding the z coordinate of the location. (see definition of parameters and locations in Regional Configuration).
Time zone with reference to UTC (equivalent to GMT) for all time dependent data communicated with the module. If not defined, UTC+0 (GMT) will be used. This time zone is used when importing pi files and the time zone is not available in the pi file.
The offset of the time zone with reference to UTC (equivalent to GMT). Entries should define the number of hours (or fraction of hours) offset. (e.g. +01:00). This time zone is used when importing pi files and the time zone is not available in the pi file.
Enumeration of supported time zones. See appendix B for list of supported time zones. This time zone is used when importing pi files and the time zone is not available in the pi file.
The date time format of the %TIME0% variable and the %CURRENT_TIME% variable. This date time format is only used if there is no date time format specified in the arguments for the variable. yyyy = Year, MM = Month in year, dd = Day in month, HH = Hour in day (0-23), mm = Minute in hour, ss = Second in minute.
The date time format of the %START_DATE_TIME% variable. This date time format is only used if there is no date time format specified in the arguments for the variable. yyyy = Year, MM = Month in year, dd = Day in month, HH = Hour in day (0-23), mm = Minute in hour, ss = Second in minute. The %START_DATE_TIME% variable only works in an importStateActivity (since 2014.02) and inside a template file for an exportCustomFormatRunFileActivity (since 2013.02).
The date time format of the %END_DATE_TIME% variable. This date time format is only used if there is no date time format specified in the arguments for the variable. yyyy = Year, MM = Month in year, dd = Day in month, HH = Hour in day (0-23), mm = Minute in hour, ss = Second in minute. The %END_DATE_TIME% variable only works in an importStateActivity (since 2014.02) and inside a template file for an exportCustomFormatRunFileActivity (since 2013.02).
Since stable build 2015.01. Definitions of date time formats that can be used as arguments for the %TIME0%, %START_DATE_TIME% and %END_DATE_TIME% variables, e.g.: %START_DATE_TIME(idOfADateFormat)% %END_DATE_TIME(idOfAnotherDateFormat)%. Date time formats specified as arguments overrule the startDateTimeFormat and endDateTimeFormat specified above. The %START_DATE_TIME% and %END_DATE_TIME% variables only work in an importStateActivity (since 2014.02) and inside a template file for an exportCustomFormatRunFileActivity (since 2013.02).
Defines if ensembles are read from or written to a number of sub directories.
The model time step adjusts the end time when necessary of the run period for this model and for all models/modules that will run after this model in the same task run.
Burn-in profile for cold state starts. Used to replace first part of a timeseries.
For time series with matching parameter-location ids, the first value is replaced by the initialValue. Element length defines the length of timeseries beginning that is to be replaced using linear interpolation.
<burnInProfile>
<length multiplier="4" unit="hour"/>
<timeSeries>
<parameterId>H.obs</parameterId>
<locationSetId>locationInitialValueAttSet</locationSetId>
<initialValueAttributeId>initialValue</initialValueAttributeId>
</timeSeries>
</burnInProfile>
|
Length of time series beginning that is to be replaced.
parameterId and locationId or locationSetId => to match timeSeries this burn in profile should apply to
initialValue or initialValueAttributeId => to specify the start value hardcoded or via a location attribute
Figure 68 Elements of the startUpActivities section of the General Adapter configuration.
Root element of a purge activity used to delete files from previous runs. Multiple purge activities may be defined.
Filter specifying files to be removed. Wildcards may be used.
Deleting a whole directory can be achieved by defining the directory path in the filter without any file filter options (.).
eg: %ROOT_DIR%/exportDir/purgeDirectory
Sub directories will not be removed. If the configured directory contains subdirectories it can not be removed, this will result in an error that is logged: "GA.Error: Can not delete an existing directory "
A directory can only be removed if it is a sub directory of the General Adapter root directory!
Example (not the use of tags to define the directory names):
Root element of an unzip activity used to unpack a zip file and put the contained files in the directory of choice. Multiple unzip activities may be defined.
Each activity has the following elements:
Root element of a zip activity used to pack all files and subdirectories of an indicated directory to a zip file for later use/inspection. Multiple zip activities may be defined. The file name may include environment variables, as well as tags defined in the general adapter or on the global.properties. See the environment variables section for a list of internal variables.
Each activity has the following elements:
Example:
<startupActivities>
<unzipActivity>
<sourceZipFile>extra_files.zip</sourceZipFile>
<destinationDir>%ROOT_DIR%/work</destinationDir>
</unzipActivity>
</startupActivities>
...
<shutdownActivities>
<zipActivity>
<sourceDir>%ROOT_DIR%/work</sourceDir>
<destinationZipFile>%ROOT_DIR%/inspection/%CURRENT_TIME%_%TIME0%_saved.zip</destinationZipFile>
</zipActivity>
</shutdownActivities>
|
Figure 69 Elements of the ExportActivity section
Export activities are defined to allow exporting various data objects from Delft-FEWS to the external modules. The list of objects that can be exported (see figure above) includes;
For most types of exportActivity, multiple entries may exist.
NB. See also the exportPlaceholderFile option in the several importActivities, which is another form of preparing input for the module adapter.
Figure 70 Elements of the ExportStateActivity section.
Optional description for the export states configuration. Used for reference purposes only.
Id of the moduleInstance that has written the state to be exported. Generally this will be the same as the Id of the current instance of the General Adapter. This can also be the ID of another instance of the General Adapter. The latter is the case when using a state in a forecast run that has been written in an historical run.
Export state exported by this ensemble.
Use a sub string of the active ensemble member id as member id when resolving the state. Fore example use (...) to use the first three characters of the active ensemble member id.
Directory to which the module instance state is to be exported. This should be the path where the model expects to find the state file(s). This should be either an absolute path or a path relative to the exportDir defined in the general section. All contents of the stored state are exported to this directory.
Optional. Deprecated, do not use this if a pi state description xml file is not needed. Name (and location) of the PI-XML file describing the states. If the directory location is not explicitly specified the file will be written in the exportDir defined in the general section. This should be either an absolute path or a path relative to the exportDir defined in the general section. This option writes the input state file paths to a pi state description xml file.
Optional. Deprecated, do not use this if a pi state description xml file is not needed. Root element for the description of the state. Both a read location and a write location will need to be defined. This allows the name of the file(s)/directory to be different on read and write. Multiple locations may be defined, but these must all be of the same type. This option writes the input state file paths to a pi state description xml file.
<stateLocations type="file"> <stateLocation> <readLocation>state.inp</readLocation> <writeLocation>state.out</writeLocation> </stateLocation> </stateLocations> |
Root element to specify how a state to be exported to the external module is to be selected. Two main groups are available, cold states and warm states. Only one of these types can be specified. Note that if a warm state selection is specified and an appropriate warm state cannot be found, a cold state will be exported by default.
<stateSelection>
<warmState>
<stateSearchPeriod unit="hour" start="-48" end="0"/>
<coldStateTime unit="hour" value="-48"/>
<insertColdState>true</insertColdState>"
</warmState>
<overrulingColdStateModuleInstanceId>ExportStateActivityOverrulingColdState</overrulingColdStateModuleInstanceId>
</stateSelection>
|
Different module instance id used for finding cold state (when warm state is configured but can't be found, or always when cold state is configured)
Optional. Adjusts the start time of time series to the exported state time within this general adapter run when start is overrulable or not specified. Default is true.
When specified, all activities are run in a loop to ensure that a state is produced on every cardinal time step between the time of the exported state and T0. This has two advantages:
Do not specify a relative view period for all time series sets in the export activity |
When specified, an extra state is written at the end of the state search period. Note that the run is then split in two. E.g my state search period is -10 to -4 days, then there are two update runs, one from the time where a state was found to -4 and one from -4 to T0. A state is written at the end of both runs (T0 and T0 - 4days). You can additionally define a minimum run length. This is necessary for some runs that need a minimum run length for e.g. PT updating. The run is then only split in two if both runs can be run over the minimum run length. If not, there is only one run and the state is written to the end of this run (T0), no intermediate state is written.
Optional. Some (HBV) models need a minimum run length to perform corrections at the end of the run.
see example configuration below and this figure
New example that does not use a pi state description xml file. In this case, if the model adapter needs to know the input state file paths, then these can be read from the netcdf version of the run info file (which can be exported using the new ExportNetcdfRunFileActivity described below):
<exportStateActivity> <moduleInstanceId>HBV_AareBrugg_Hist</moduleInstanceId> <stateExportDir>%ROOT_DIR%/FEWS/states</stateExportDir> <stateSelection> <warmState> <stateSearchPeriod unit="hour" start="-240" end="-96"/> </warmState> </stateSelection> <writeIntermediateState>true</writeIntermediateState> <minimumRunLength unit="day" multiplier="4"/> </exportStateActivity> |
Old example that writes the input state file paths to a pi state description xml file. Do not use this if a pi state description xml file is not needed:
<exportStateActivity> <moduleInstanceId>HBV_AareBrugg_Hist</moduleInstanceId> <stateExportDir>%ROOT_DIR%/FEWS/states</stateExportDir> <stateConfigFile>%ROOT_DIR%/FEWS/states/states.xml</stateConfigFile> <stateLocations type="file"> <stateLocation> <readLocation>HBV_States.zip</readLocation> <writeLocation>HBV_States.zip</writeLocation> </stateLocation> </stateLocations> <stateSelection> <warmState> <stateSearchPeriod unit="hour" start="-240" end="-96"/> </warmState> </stateSelection> <writeIntermediateState>true</writeIntermediateState> <minimumRunLength unit="day" multiplier="4"/> </exportStateActivity> |
Figure 71 Elements of the exportTimeSeries section
Optional description of the timeSeries export configuration.
Name (and location) of the PI-XML file with exported time series. If the directory location is not explicitly specified the file will be written in the exportDir defined in the general section.
When true the events in the PI time series file are written to a binary file instead of the xml file. The written xml file will only contain the time series headers and optionally a time zone. The binary file has the same name as the xml file only the extension is "bin" instead of "xml". During PI time series import the bin file is automatically read when available. The byte order in the bin file is always Intel x86.
When true the run period, written in the pi run file, will not be extended.
When true any thresholds for the exported timeseries will be written in the timeserie headers
Root element for defining timeSerieSets to be exported.
TimeSeriesSets to be exported. These may contain either a (list of) locations or a locationSet. Multiple entries may be defined.
Includes missing values in the export file or leave them out.
Option to check the timeseries for missing values. This option works in combination with the Parameter element 'allowMissing'. If that allowMissing is set to false, you can check for missing values. So if checkMissingValues = true and allowMissing=false, an error will occur in case there are missing values to be exported. As such, you can prevent to run a model with missing values and let the General Adapter stop with an error.
When true, a series is not exported when the time series is empty (or when omitMissingValues = true, when the time series is empty after removing the missing values.)
Can be used to select all approved forecasts with a forecast start time lying within this period
Option to skip time series sets if the location set does not exist. Useful when module or workflow is run in loop with tags being translated. Default is 'false'.
Figure 72 Elements of the ExportMapStacksActivity.
Optional description of the timeSeries export configuration.
Name (and location) of the PI-XML file describing the map stack of the exported grid time series. If the directory location is not explicitly specified the file will be written in the exportDir defined in the general section.
Root element for defining file name of grid to be exported
LocationId for grid to be exported.
Name of the files for the grid to be exported. For grid files where each time slice is stored in a different file, this name is the prefix for the full file name. The final file name is created using an index of files exported (e.g the file name for the 4^th^ time step is grid00000.004).
Format of the exported grid. Enumeration of options include;
TimeSeriesSets to be exported. These should contain only one locationId. For exporting multiple grids, multiple exportMapStack activities should be defined.
Configuration of the exportProfiles activity is identical to the exportTimeSeries Activity.
Figure 73 Elements of the exportDataSets section
Optional description of the module dataset export configuration.
Optional reference to the moduleInstanceId of the moduleDataSet to be exported. If not defined the moduleInstanceId of the current module instance is taken as a default (see section on Module Datasets and Parameters).
Figure 74 Elements of the exportParameter section
Optional description of the module parameter configuration.
Optional reference to the moduleInstanceId of the moduleParameter to be exported. If not defined the moduleInstanceId of the current module instance is taken as a default (see section on Module Datasets and Parameters) . The parameterfile should be stored in the folder config\ModuleParFiles.
Name (and location) of the PI-XML file with exported parameters. If the directory location is not explicitly specified the file will be written in the exportDir defined in the general section.
It is possible to have the parameterfile filled with location attributes. When exporting the parameter file, FEWS will loop over all the locations in the given locationSet, or through the given locationIds, and add the configured attributes to the parameterfile. For an example of a template parameterfile, see 02 Module Parameters. Notice that the element "model" references to the model in the template parameter file.
Figure 1 Elements of the ExportTableActivity configuration
Optional description of the ExportTableActivity configuration.
File to which the table will be exported. This file is always placed in exportDir.
Type of table to be exported. Currently enumeration must be "ratingCurve"
ID of the table to be exported.
Parameters for the convertEquation operation. Must include minimumLevel, maximumLevel and stepSize
location id to select the rating curve
Exports scalar, grid or 1d/2d spectra time series to a NetCDF file. All time series specified inside one exportNetcdfActivity must have the same value type (grid, scalar, 1d spectra or 2d spectra). For this export it is required that all ensembles have exactly the same ensemble member indices. The usage of ensemble member id's (strings) is not supported yet.
Compacting netCDF variables with scale factor and offset into a smaller integer typed variable is not available in contrast to the archive export (applied whenever possible) and time series export (applied when possible if property tryCompactingNetCDFData is set to true).
Figure 2 Elements of the ExportNetcdfActivity configuration
Optional description of the ExportNetcdfActivity configuration.
File to which the data will be exported. This file is always placed in exportDir.
TimeSeriesSet that defines what data is to be exported.
Include missing values in the export file or leave them out.
The time series is not exported when the time series is empty or when; omitMissingValues = true and the time series is empty after removing the missing values.
Option to skip time series sets if the location set does not exist. Useful when module or workflow is run in loop with tags being translated. Default is 'false'.
Figure 3 Elements of the ExportRunFileActivity configuration
Optional description of the ExportRunFileActivity configuration.
File to which the data will be exported. This file is always placed in exportDir.
Kind of environment variables for the pre and post adapters. These properties are copied to the run file. This is also a convenient way to pass global properties to a pre or post adapter. An adapter is not allowed to access the FEWS global.properties directly. Global properties (between $) are replace by there literal values before copied to the run file. These extra options makes an additional pre or post adapter configuration file unnecessary.
Options:
example of exported run file
<Run xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.wldelft.nl/fews/PI" xsi:schemaLocation="http://www.wldelft.nl/fews/PI http://fews.wldelft.nl/schemas/version1.0/pi-schemas/pi_run.xsd" version="1.5">
<logLevel>info</logLevel>
<timeZone>0.0</timeZone>
<startDateTime date="1900-01-01" time="00:00:00"/>
<endDateTime date="2100-01-01" time="00:00:00"/>
<time0 date="2000-01-01" time="00:00:00"/>
<workDir>workdir</workDir>
<outputDiagnosticFile>diagnostic</outputDiagnosticFile>
</Run>
|
Exports a run file in netcdf format. The netcdf run file contains general information about the run executed by the general adapter that can be used by the pre and post adapter.
Elements of the ExportNetcdfRunFileActivity configuration
Optional description for this activity.
Netcdf run file to which the general run information will be exported. This should be either an absolute path or a path relative to the exportDir defined in the general section.
Optional. Properties in the form of key-value pairs that can be used to pass information to the pre and post adapter. These properties will be copied to the netcdf run file. These properties can be used as configuration for the pre and post adapter, so it is not needed to have a separate configuration file for the pre and post adapter. This is also a convenient way to pass global properties from FEWS to a pre or post adapter, since an adapter is not allowed to access the FEWS global.properties file directly. Global properties (between '$' signs) in these key-value pairs will be replaced by their literal values before they are copied to the netcdf run file.
Options:
<exportNetcdfRunFileActivity> <description>This run file is passed as argument to XBeachPreAdapter</description> <exportFile>run_info.nc</exportFile> <properties> <bool key="use_friction" value="true"/> </properties> </exportNetcdfRunFileActivity> |
A netcdf run file will be generated by FEWS and contains information that is needed by the pre/post adapter.
The variables input_netcdf_files and input_state_files are optional and list the files that are exported from FEWS. The variables output_netcdf_files and output_state_files are optional and list the files that need to be imported by FEWS.
The attributes of the properties variable form the configuration for the pre/post adapter and can be configured in FEWS (both the attribute names and attribute values are configurable).
All paths in the netcdf run file are relative to the parent directory of the netcdf run file.
An example of a netcdf run file can be downloaded here: run_info_example.nc. Below is the same example netcdf run file, converted to text format:
netcdf run_info_example {
dimensions:
path_length = 255 ;
input_netcdf_file_count = 2 ;
input_state_file_count = 1 ;
output_netcdf_file_count = 2 ;
variables:
double start_time ;
start_time:long_name = "start_time" ;
start_time:standard_name = "time" ;
start_time:units = "minutes since 2002-11-26 00:00:00.0 +0000" ;
double end_time ;
end_time:long_name = "end_time" ;
end_time:standard_name = "time" ;
end_time:units = "minutes since 2002-11-26 00:00:00.0 +0000" ;
char work_dir(path_length) ;
char input_netcdf_files(input_netcdf_file_count, path_length) ;
char input_state_files(input_state_file_count, path_length) ;
char output_netcdf_files(output_netcdf_file_count, path_length) ;
char properties ;
properties:string_property = "zs_0 with spaces" ;
properties:INT_property = 3 ;
properties:float_PROPERTY = 3.5f ;
properties:DOUBLE_PROPERTY_321 = 0.123456789 ;
properties:logical_property_1 = "true" ;
properties:logical_property_2 = "false" ;
// global attributes:
:title = "Run file" ;
:institution = "Deltares" ;
:source = "Run file from Delft-FEWS" ;
:history = "2014-03-18 12:22:55 GMT: exported from Delft-FEWS to D:\\fews\\workspace\\fews\\junit_test_output\\nl\\wldelft\\fews\\system\\plugin\\generaladapter\\exportDir\\run_info.nc" ;
:references = "http://www.delft-fews.com" ;
data:
start_time = 0 ;
end_time = 5880 ;
work_dir = "..\\work" ;
input_netcdf_files =
"..\\work\\boundary_data.nc",
"..\\work\\more_boundary_data.nc" ;
input_state_files =
"..\\input_state\\state.inp" ;
output_netcdf_files =
"..\\work\\model_output_data1.nc",
"..\\work\\model_output_data2.nc" ;
properties = " " ;
}
|
Activity to create a custom run info file. This activity replaces tags in a template file with actual (modified) values. The template file can contain location attribute tags between '@' signs, general adapter tags between '%' signs and global properties between '$' signs. The specified template file will be copied to the specified export file before the tags are replaced. So the template file itself is not changed. If the export file already exists, then it will be overwritten.
Elements of the ExportCustomFormatRunFileActivity configuration
Note: this feature is deprecated. Better use the export(Netcdf)RunFileActivity in combination with a model pre-adapter instead.
Optional description for the activity.
The pathname of the template file relative to the rootDir (or absolute path). This file can contain location attribute tags between '@' signs, general adapter tags between '%' signs and global properties between '$' signs.
In addition to the tags specified in the documentation on the general section, also tags %START_DATE_TIME% and %END_DATE_TIME% are accommodated
The pathname of the export file relative to the exportDir (or absolute path). This file is created during the workflow run and will be a copy of the template file in which the tags are replaced with actual (modified) values. If this file already exists, then it will be overwritten.
Optional. Location attribute tags in the template file will be replaced with the corresponding (modified) attribute values of the location with this locationId. If this is not specified, then location attribute tags will not be replaced.
Available since 2017.02. Optional field used to specify the width (length in characters) the location attribute tags and global properties ('@' and '$') should be replaced with. The result will be padded on the left with the required number of spaces. Note that if no numberOfDecimals is specified, all number attributes will be rounded based on the number of significant digits that will fit in the configured fixedWidth. If a numberOfDecimals is specified, the number attributes will be rounded to the specified number of decimals instead. If the text for any attribute or global property contains more than the specified number of characters, a warning is logged and the value is outputted as is.
Available since 2017.02. Optional field used to specify the number of decimals all number attributes ('@') should be rounded to. Note that when a fixedWidth is also specified, the rounded values must always fit in the number of characters specified.
<generalAdapterRun xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/generalAdapterRun.xsd"> <general> ... <startDateTimeFormat>yyyy MM dd HH mm</startDateTimeFormat> <endDateTimeFormat>yyyy MM dd HH mm</endDateTimeFormat> </general> <activities> <exportActivities> <exportCustomFormatRunFileActivity> <templateFile>%ROOT_DIR%/templatefiles/event_tox2_template.inp</templateFile> <exportFile>event_tox2.inp</exportFile> <locationId>locationWithAttributes</locationId> <fixedWidth>10</fixedWidth> <numberOfDecimals>5</numberOfDecimals> </exportCustomFormatRunFileActivity> </exportActivities> </activities> </generalAdapterRun> |
Example of template file:
%START_DATE_TIME% ! MODEL START : ISYEAR,ISMONTH,ISDATE,ISHR,ISMN [year,month,day,hour,minute] %END_DATE_TIME% ! MODEL STOP : ISYEAR,ISMONTH,ISDATE,ISHR,ISMN [year,month,day,hour,minute] 2 ! TIME STEP :[sec] @TOXIC_ID@ ! TOXIC ID : CAS NO.-TOXIC, 444#-OI @ACCIDENT_LOC_X@ @ACCIDENT_LOC_Y@ ! ACCIDENT LOC. : X Y TM coordinate (referred in LXLY.INP) @ACCIDENT_TIME@ ! ACCIDENT TIME : IEVYEAR,IEVMONTH,IEVDATE,IEVHR,IEVMN [year,month,day,hour,minute] @spill_duration@ ! spill duration : [MIN] @spill_material_mass@ ! spill material mass : [G] 10. ! CHLA : CHLA CONC. [UG/L] FOR TOXIC CALCULATION 5. ! TEM : WATER TEMPERATURE [DEGREE C] FOR TOXIC CALCULATION 5. ! SSC : Suspended Solid Conc. [MG/L] FOR TOXIC CALCULATION 0.1 ! DOC : DOC CONC. [MG/L] FOR TOXIC CALCULATION 100. ! I : IRADIATION [LY/DAY] 5. ! WSPD : WINDSPEED [M/S] $SIMULATION_NUMBER$ ! simulation number %TIME0%$Identifier$%TIME0% $Identifier$ ! test |
Example of exported file:
2002 11 30 09 00 ! MODEL START : ISYEAR,ISMONTH,ISDATE,ISHR,ISMN [year,month,day,hour,minute] 2002 11 30 11 00 ! MODEL STOP : ISYEAR,ISMONTH,ISDATE,ISHR,ISMN [year,month,day,hour,minute] 2 ! TIME STEP :[sec] 131-52-2 ! TOXIC ID : CAS NO.-TOXIC, 444#-OI 235875.9 329956.4 ! ACCIDENT LOC. : X Y TM coordinate (referred in LXLY.INP) 2012 01 10 09 00 ! ACCIDENT TIME : IEVYEAR,IEVMONTH,IEVDATE,IEVHR,IEVMN [year,month,day,hour,minute] 10.0 ! spill duration : [MIN] 500000.0 ! spill material mass : [G] 10. ! CHLA : CHLA CONC. [UG/L] FOR TOXIC CALCULATION 5. ! TEM : WATER TEMPERATURE [DEGREE C] FOR TOXIC CALCULATION 5. ! SSC : Suspended Solid Conc. [MG/L] FOR TOXIC CALCULATION 0.1 ! DOC : DOC CONC. [MG/L] FOR TOXIC CALCULATION 100. ! I : IRADIATION [LY/DAY] 5. ! WSPD : WINDSPEED [M/S] 73 ! simulation number 200211300900FSS200211300900 FSS ! test |
Exports an area selection mask as a PiMapStack. This activity requires an embedded polygon to be specified within the areaSelectionShapeFileBase64 field of the TaskProperties. This file is then available to be used by the pre and post adapter.
Optional description for the activity. Used for reference purposes only.
File name of the file to be exported. Always placed into the export dir.
Locations that contains the reference grid information.
Format of the PiMapStack file. Can be
Exports an area selection mask in the form of a PiMapStack. This activity requires selectedLocationIds to be specified within the TaskProperties. This selection is then available to be used by the pre and post adapter.
Optional description for the activity. Used for reference purposes only.
File name of the file to be exported. Always placed into the export dir.
Locations that contains the reference grid information.
Format of the PiMapStack file. Can be
Identifier for locationset for which to generate grid files. The locations should have polygons associated with them.
Exports rating curves to pi_ratingcurves.xml file. The rating curves with the start time within the GeneralAdapter run period will be exported.
Figure 1 Elements of the exportRatingCurveActivity configuration
Optional description for the activity. Used for reference purposes only.
File name of the file to be exported. Always placed into the export dir.
Stage resolution. Use unit configured with 'displayUnit' in stage parameter group. If there is no 'displayUnit' , m are used. Every resolution 'step' (e.g. 0.5 ft) stage/flow pair are exported. The exporting stage/value pairs starts at rating curve minStage and goes up to the ratingCurve maxStage, if any available, otherwise up to the last rating curve stage value. The logarithmic interpolation in a rating curve needs a complex algorithm. For this reason it's it is convenient when we can convert to a more detailed table where we can use linear interpolation
Locations the rating curves will be exported for.
Locations the rating curves will be exported for.
Elements of the ExecuteActivity configuration
Root element for the definition of an execute activity. Executes a module or its adapters. For each external executable or Java class to run, an executeActivity must be defined. Multiple entries may exist.
In case of a failed run, Delft-FEWS attempts to terminate any child process afterwards. In order to be able to do this reliably, it is important to only create processes with a proper process tree. For instance, an unsupported option is to execute a Windows batch file starting executables in the background where the batch file terminates itself, leaving the child processes orphaned. This is illegal because Delft-FEWS cannot detect these as child processes and therefore not terminate the child processes. This will lead to model executables from dead taskruns to remain while possible locking files, blocking new runs with the same executable. |
Optional description for the activity. Used for reference purposes only.
Root element to define command to execute. This can be either an executable or a Java class.
<command> <className>nl.wldelft.fews.adapter.urbsiniadapter.UrbsIniAdapter</className> <customJreDir>%ROOT_DIR%/jre</customJreDir> <jvmArg>-mx512m</jvmArg> </command> |
Optional. Root element for defining arguments to be passed to the executable/Java class
Optional. Root element for defining environment variables prior to running the executable/Java class
For example, to append some directory to the Windows PATH environment variable use:
<environmentVariables> <environmentVariable> <name>path</name> <value>%PATH%;%ROOT_DIR%\wflow_bin;</value> </environmentVariable> </environmentVariables> |
Optional. One or more text files that contain log information from the module that is executed. After the module has executed, the specified lines from these files will be imported in FEWS as log messages with the specified log level.
Configuration example:
<logFile> <file>XBerror.txt</file> <!-- Import every line as a separate FEWS error log message. --> <errorLinePattern>*</errorLinePattern> </logFile> <logFile> <file>XBwarning.txt</file> <!-- Import every line that contains "warning" as a separate FEWS info log message. --> <infoLinePattern>*warning*</infoLinePattern> </logFile> <logFile> <file>XBlog.txt</file> <!-- Import every line that contains "ERROR" as a separate FEWS debug log message. --> <debugLinePattern>*ERROR*</debugLinePattern> </logFile> |
Timeout to be used when running module (in milliseconds). If run time exceeds timeout it will be terminated and the run considered as having failed.
Optional. Since 2013.01. FEWS-8674. Some models only allow a limited number of runs simultaneously, sometimes limited by license server. maxNumberOfSimultaneousRuns = 1 is exactly the same as waitForOtherRun = true. By default the number of runs is limited by the workflow and runInLoopParallelProcessorCount in the global properties.
Optional. Since 2009.02. When an adapter is using shared writable resources (e.g. common blocks in a linked fortran library or writable static java fields) it will not allow multiple runs of this java class at the same time. When waitForOtherRun is true it will wait until any other run of the specified class is finished. When the executable only allows one instance set this option to true. Some executables only allow one instance due to licensing issues or temporary files that are used.
Optional. File containing diagnostic information about activity. This file always is located in the importDir and overrules the global diagnostic file.
Optional. For this activity no check should be done whether the diagnostics file is present or not.
Optional. For this activity no check should be done whether the executable returns a non zero exit code. A non zero exit code can prevent the post adapter from running. When the post adapter parses a log file the exit code can be ignored. The error in the log file is more descriptive than the error code.
Several variables are available to be used as an argument to an external program or in the exportCustomFormatRunFileActivity. You can use in any filename or directory the properties from the global.properties file or the next internal variables:
The colon characters ":" will be replaced by an underscore "_". Use the internal variables with % characters (like %TEMP_DIR%) and the global.properties variables with $ characters (like e.g. $DUMP_DIR$).
Figure 76 Elements of the ImportActivities configuration
Option exportPlaceholderFile is not supported for spectra data. |
This option can be used for all import activities, except for importStateActivity.
If <exportPlaceholderFile>true</exportPlaceholderFile>, then the General adapter will generate placeholder files. A placeholder file is a file with headers only, without timeseries. Its name is the same as the filename configured for the ImportActivity and this placeholderfile is written to the import directory. The placeholder files are written before any execute activity is started. The models cq model adapters should read this placeholder files to see which data should be provided to import in FEWS.
The intention of this exportPlaceholderFile functionality is that the model cq modeladapter reads the placeholders to see which timeseries are required by FEWS.
After simulation, the model cq modeladapter overwrites these files with its own data over the placeholder files ready to be imported by the import activity.
<importTimeSeriesActivity>
<exportPlaceholderFile>true</exportPlaceholderFile>
<importFile>output.xml</importFile>
<timeSeriesSets>
....
</timeSeriesSets>
</importTimeSeriesActivity>
|
It is available in the same way for importTimeSeriesActivity, importMapStacksActivity, importNetcdfActivity (only for scalar and grid data, not for spectra data) and importProfilesActivity.
Optional description of import activity. Used for reference purposes only
Elements of the ImportStateActivity section.
Root element for importing modules states resulting from the run of the external modules. Multiple elements may be defined. If no state is to be imported (for example in forecast run as opposed to state run), then the element should not be defined.
Configuration example:
New example that does not use a pi state description xml file. In this case, the output state file paths are configured directly in the importStateActivity as absolute paths:
<importStateActivity> <stateFile> <importFile>%WORK_DIR%/state.out</importFile> <!-- Rename imported state file so that it can be used as input state for a future model run. --> <relativeExportFile>state.inp</relativeExportFile> </stateFile> <stateFile> <importFile>%WORK_DIR%/state2.out</importFile> <!-- Rename imported state file so that it can be used as input state for a future model run. --> <relativeExportFile>state2.inp</relativeExportFile> </stateFile> </importStateActivity> |
New example that does not use a pi state description xml file. In this case, the output state file paths are configured directly in the importStateActivity as paths relative to a stateImportDir:
<importStateActivity> <stateImportDir>%WORK_DIR%</stateImportDir> <stateFile> <importFile>state.out</importFile> <!-- Rename imported state file so that it can be used as input state for a future model run. --> <relativeExportFile>state.inp</relativeExportFile> </stateFile> <stateFile> <importFile>state2.out</importFile> <!-- Rename imported state file so that it can be used as input state for a future model run. --> <relativeExportFile>state2.inp</relativeExportFile> </stateFile> </importStateActivity> |
Since 2017.02. Import a dynamic number of states for different state times at once. Pi state description xml file is not used. When exported the file is named "start.bin":
<importStateActivity> <stateImportDir>states</stateImportDir> <stateFileDateTimePattern>'state'yyyyMMddHHmm'.bin'</stateFileDateTimePattern> <relativeExportFile>start.bin</relativeExportFile> </importStateActivity> |
Old example that reads the input state file paths from a pi state description xml file. Do not use this if a pi state description xml file is not needed:
<importStateActivity> <stateConfigFile>pi_output_state_description.xml</stateConfigFile> </importStateActivity> |
Example of a client config for stand alone when importing to the warm states directory. The meta data is still stored in the data base. Files in this directory that don't have meta data will be automatically deleted by the rolling barrel.
<?xml version="1.0" encoding="UTF-8"?> <clientConfiguration xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/clientConfig.xsd"> <localDataStoreFormat>Firebird</localDataStoreFormat> <warmStatesDirectory>%REGION_HOME%/warmStates</warmStatesDirectory> </clientConfiguration> |
Root element for importing scalar and polygon time series resulting from the run of the external modules. Multiple elements may be defined. importFile and timeSeriesSet should be defined.
Root element for importing grid time series resulting from the run of the external modules. Multiple elements may be defined. importFile and timeSeriesSet should be defined.
Imports scalar, grid or 1d/2d spectra time series from a NetCDF file resulting from the run of the external modules. All time series specified inside one importNetcdfActivity must have the same value type (grid, scalar, 1d spectra or 2d spectra). For this activity the exportPlaceholderFile option only works for scalar and grid data, not for spectra data. Note: this activity is exactly the same as importPiNetcdfActivity, only the name is better, since the NetCDF format has nothing to do with the PI format.
When an importNetcdfActivity is run within an ensemble loop defined in the workflow configuration and the %ENSEMBLE_MEMBER_ID% tag is used in the <importDir> or the <importFile> it will loop over the ensemble members and use the ensemble member index for the imported scalar time series (since 2017.02).
<activity> <moduleInstanceId>ImportNetcdfScalarEnsembles</moduleInstanceId> <ensemble> <ensembleId>Ens</ensembleId> <ensembleMemberIndexRange start="0" end="4"/> <runInLoop>false</runInLoop> </ensemble> </activity> |
When it is not desired that all other activities are also run for each ensemble member, make sure <runInLoop> is set to false.
This is useful when 1 general adapter run generates many similar ensemble files that do not contain data from which their ensemble id can be determined besides their file name or their parent directories. Like a run from OpenDA.
Deprecated, do not use. Please use importNetcdfActivity instead. Imports scalar, grid or 1d/2d spectra time series from a NetCDF file resulting from the run of the external modules. All time series specified inside one importPiNetcdfActivity must have the same value type (grid, scalar, 1d spectra or 2d spectra). For this activity the exportPlaceholderFile option only works for scalar and grid data, not for spectra data. For more documentation see importNetcdfActivity.
Root element for importing longitudinal profile time series resulting from the run of the external modules. Multiple elements may be defined. importFile and timeSeriesSet should be defined.
Optional description of import activity. Used for reference purposes only
Root element for importing shape files resulting from the run of the external modules.
A <shapeFileImportDir> can be defined which is either relative (to the general <importDir>) or absolute. When it is not defined the general <importDir> will be used. If the defined directory does not exist, an exception will be thrown. The import directory can be empty.
A <fileDateTimePattern> should be defined to filter out the shape files that need to be imported and to extract the time for the shape in the time series. If the directory is not empty, but none of the files match the defined pattern, an exception will be thrown.
A <geoDatum> can be defined if the shape file is not in WGS84. When this is defined, a conversion to WGS84 will take place before storing the time series into the database.
The <timeSeriesSet> will automatically be used for the imported shape, no idmapping will take place.
<importActivities> <importShapeFileActivity> <shapeFileImportDir>import</shapeFileImportDir> <fileDateTimePattern>'ImportShapeFileActivity_'dd MMM yyyy HH mm ss'.shp'</fileDateTimePattern> <geoDatum>EPSG:102736</geoDatum> <timeSeriesSet> <moduleInstanceId>ImportShapeActivity</moduleInstanceId> <valueType>polygon</valueType> <parameterId>par</parameterId> <locationId>Polygon</locationId> <timeSeriesType>external historical</timeSeriesType> <timeStep unit="nonequidistant"/> <readWriteMode>read only</readWriteMode> </timeSeriesSet> </importShapeFileActivity> </importActivities> |
Root element for importing csv files that can be displayed using the Module Run Table Display component.
Elements of the importCsvModuleRunTablesActivity section.
Only columns that are mapped will be imported. The imported values will be stored in the order of the configured column mappings.
Configuration example:
In the following example one importCsvModuleRunTablesActivity is specified for the import file import_csv_module_test1.csv. In case more than one csv file has to be imported, for each csv file an activity has to be configured. Note that in this example the csv column 'Float' hasn't been mapped. This will result in not importing the 'Float' column values.
<importActivities> <importCsvModuleRunTablesActivity> <charset>UTF-8</charset> <description>Csv Module Data Activity Test case.</description> <importFile>import_csv_module_test1.csv</importFile> <displayName>Import CSV Module Test1</displayName> <table> <!-- Map de following CSV definition: DateTime;Status;Boolean;Integer;Float;Double;String 2003-02-22 00:00;Ok;yes;14;1.4;1.45;Hello world 1 2003-02-23 00:00;Not Ok;;true;24;2.4;2.45;Hello world 2 --> <dateTimeColumn name="DateTime" displayName="datum" pattern="yyyy-MM-dd HH:ss"/> <stringColumn name="Status" displayName="status field"/> <booleanColumn name="Boolean" displayName="Yes or No"/> <integerColumn name="Integer" displayName="Int field"/> <doubleColumn name="Double" displayName="Double field"/> <stringColumn name="String" displayName="String field"/> </table> </importCsvModuleRunTablesActivity> </importActivities> |
If errors have been made in the mapping of the csv table, one warning will be logged per column to avoid flooding the log file with warnings. The import will always continue and a empty value will be set. The exceptions are Booleans that will be set to false and dateTimes will be set to 0.
Elements of the Shutdown Activities configuration
This activity is the identical to the startUpActivities. The only difference is that these are carried out after the module run and import of data. See definition of StartUp activities for configuration.
ignoreNonExistingLocationSets<command> <className>nl.wldelft.fews.adapter.urbsiniadapter.UrbsIniAdapter</className> <customJreDir>%ROOT_DIR%/jre</customJreDir> <jvmArg>-mx512m</jvmArg> </command>