Page History

...

Since 2022.02. When true all exported and imported files (XML, NetCdf) are transferred in memory between FEWS and execute activities. An execute activity can read the file content from

...

All execute activities that transfer files to and from FEWS should be java activities that don't use a custom jre and check the System.getProperties().get(file) before they fall back to read/write the file on disk. Only NetCDF3 All files can be transferred in memory, except NetCDF4. For NetCDF NetCDF3 files the System.getProperties().get(file) contains a chunked byte array (byte[][]). An empty byte[][] array is set to tell an execute activity to save the nc file to memory as chunked byte array. The diagnostic file is also transferred in memory. Execute activities should always fall back on the files on disk when the System.getProperties().get(file) == null

rootDir

Root directory for the external module. Other directories can be defined relative to this rootDir using predefined tags (see comment box below).

workDir

Working directory to be used by the external module. When started this directory will be the current directory.

exportDir

Directory to export data from Delft-FEWS to the external module. All Published Interface files will be written to this directory (unless overruled in naming the specific export files).

exportDataSetDir

Directory to export module datasets from Delft-FEWS to the external module. A module dataset is a ZIP file, which will be unzipped using this directory as the root directory. If the zip file contains full path information, this will be included as a tree of subdirectories under this directory.

updateExportDataSetDirOnlyOnChange

If set to 'true' datasets are only updated when they have changed. Change is detected by comparing the timestamp in the dataset info file with the modification time of the dataset configuration file. By default this value is 'false' and datasets are always updated. Since the 2018.02 the dataset info file is written to $REGION_HOME$/temp/moduleDataSetsCheckSums when the export dir is sub dir (any level) of the region home.

purgeExportDataSetDirOnUpdate

If a dataset has been detected as updated then this option allows user to purge content of existing moduledataset directory before new moduledataset is exported. By default the value is 'false' and all existing dataset content is overwritten.

exportIdMap

ID of the IdMap used to convert internal parameterId's and locationId's to external parameter and location Id's. See section on configuration for Mapping Id's units and flags.

exportUnitConversionsId

Id of UnitConversions to be used for export unit mapping

importDir

Adapters that consistently use the Delf-PI.jar classes for I/O of XML data, configuration and diagnostics do not need to be modified to specifically make use of this feature, as the switching between in-memory or on-disk file transfers will be handled by Delf-PI.jar automatically. The feature has also been implemented for NetCDF files in NetCdfUtils.jar but this is considered experimental and requires further testing before using it in production.

Fortran/C models has to be recompiled to so/dll to allow in memory transfer. From the adapter this so/dll can be invoked with JNA (https://github.com/java-native-access/jna)

The NetCDF3 bytes can be passed to the so/dll  use System.getProperties().get/put(file) to get/put the bytes of the netcdf file. The Fortran/C code can use the nc_open_mem/nc_create_mem to open/create the bytes

rootDir

Root directory for the external module. Other directories can be defined relative to this rootDir using predefined tags (see comment box below).

workDir

Working directory to be used by the external module. When started this directory will be the current directory.

exportDir

Directory to export data from Delft-FEWS to the external module. All Published Interface files will be written to Directory to import result data from the external module to Delft-FEWS. All Published Interface files will be read from this directory (unless overruled in naming the specific export files).

importIdMap

ID of the IdMap used to convert external parameterId's and locationId's to internal parameter and location Id's. This may be defined to be the same as the import directory, but may also contain different mappings. See section on configuration for Mapping Id's units and flags.

importUnitConversionsId

Id of UnitConversions to be used for import unit mapping

dumpFileDir

Directory for writing dump files to. Dump Files are created when one of the execute activities fails. A dump file is a ZIP file which includes all the dumpDir directories defined. The dump file is created immediately on failure, meaning that all data and files are available as they are at the time of failure and can be used for analysis purposes. The ZIP file name is time stamped to indicate when it was created.

dumpDir

Directory to be included in the dump file. All contents of the directory will be zipped. Multiple dumpDir's may be defined.

NOTE: ensure that the dumpDir does not include the dumpFileDir. This creates a circular reference and may result in corrupted ZIP files.

diagnosticFile

File name and path of diagnostic files created in running modules. This file should be formatted using the Published Interface diagnostics file specification.

missVal

Optional specification of missing value identifier to be used in PI-XML exported to modules and imported from modules.

NOTE: it is assumed an external module uses the same missing value identification for both import and export data.

convertDatum

Optional Boolean flag to indicate level data is used and produced by the module at a global rather than a local datum. The convention in Delft-FEWS is that data is stored at a local datum. If set to true data in parameter groups supporting datum conversion will be converted on export to the global datum by adding the z coordinate of the location. (see definition of parameters and locations in Regional Configuration).

timeZone

Time zone with reference to UTC (equivalent to GMT) for all time dependent data communicated with the module. If not defined, UTC+0 (GMT) will be used. This time zone is used when importing pi files and the time zone is not available in the pi file.

timeZoneOffset

The offset of the time zone with reference to UTC (equivalent to GMT). Entries should define the number of hours (or fraction of hours) offset. (e.g. +01:00). This time zone is used when importing pi files and the time zone is not available in the pi file.

timeZoneName

Enumeration of supported time zones. See appendix B for list of supported time zones. This time zone is used when importing pi files and the time zone is not available in the pi file.

time0Format

The date time format of the %TIME0% variable and the %CURRENT_TIME% variable. This date time format is only used if there is no date time format specified in the arguments for the variable. yyyy = Year, MM = Month in year, dd = Day in month, HH = Hour in day (0-23), mm = Minute in hour, ss = Second in minute.

startDateTimeFormat

The date time format of the %START_DATE_TIME% variable. This date time format is only used if there is no date time format specified in the arguments for the variable. yyyy = Year, MM = Month in year, dd = Day in month, HH = Hour in day (0-23), mm = Minute in hour, ss = Second in minute. The %START_DATE_TIME% variable only works in an importStateActivity (since 2014.02) and inside a template file for an exportCustomFormatRunFileActivity (since 2013.02).

endDateTimeFormat

exportDataSetDir

Directory to export module datasets from Delft-FEWS to the external module. A module dataset is a ZIP file, which will be unzipped using this directory as the root directory. If the zip file contains full path information, this will be included as a tree of subdirectories under this directory.

updateExportDataSetDirOnlyOnChange

If set to 'true' datasets are only updated when they have changed. Change is detected by comparing the timestamp in the dataset info file with the modification time of the dataset configuration file. By default this value is 'false' and datasets are always updated. Since the 2018.02 the dataset info file is written to $REGION_HOME$/temp/moduleDataSetsCheckSums when the export dir is sub dir (any level) of the region home.

purgeExportDataSetDirOnUpdate

If a dataset has been detected as updated then this option allows user to purge content of existing moduledataset directory before new moduledataset is exported. By default the value is 'false' and all existing dataset content is overwritten.

exportIdMap

ID of the IdMap used to convert internal parameterId's and locationId's to external parameter and location Id's. See section on configuration for Mapping Id's units and flags.

exportUnitConversionsId

Id of UnitConversions to be used for export unit mapping

importDir

Directory to import result data from the external module to Delft-FEWS. All Published Interface files will be read from this directory (unless overruled in naming the specific export files).

importIdMap

ID of the IdMap used to convert external parameterId's and locationId's to internal parameter and location Id's. This may be defined to be the same as the import directory, but may also contain different mappings. See section on configuration for Mapping Id's units and flags.

importUnitConversionsId

Id of UnitConversions to be used for import unit mapping

dumpFileDir

Directory for writing dump files to. Dump Files are created when one of the execute activities fails. A dump file is a ZIP file which includes all the dumpDir directories defined. The dump file is created immediately on failure, meaning that all data and files are available as they are at the time of failure and can be used for analysis purposes. The ZIP file name is time stamped to indicate when it was created.

dumpDir

Directory to be included in the dump file. All contents of the directory will be zipped. Multiple dumpDir's may be defined.

NOTE: ensure that the dumpDir does not include the dumpFileDir. This creates a circular reference and may result in corrupted ZIP files.

diagnosticFile

File name and path of diagnostic files created in running modules. This file should be formatted using the Published Interface diagnostics file specification.

missVal

Optional specification of missing value identifier to be used in PI-XML exported to modules and imported from modules.

NOTE: it is assumed an external module uses the same missing value identification for both import and export data.

convertDatum

Optional Boolean flag to indicate level data is used and produced by the module at a global rather than a local datum. The convention in Delft-FEWS is that data is stored at a local datum. If set to true data in parameter groups supporting datum conversion will be converted on export to the global datum by adding the z coordinate of the location. (see definition of parameters and locations in Regional Configuration).

timeZone

Time zone with reference to UTC (equivalent to GMT) for all time dependent data communicated with the module. If not defined, UTC+0 (GMT) will be used. This time zone is used when importing pi files and the time zone is not available in the pi file.

timeZoneOffset

The offset of the time zone with reference to UTC (equivalent to GMT). Entries should define the number of hours (or fraction of hours) offset. (e.g. +01:00). This time zone is used when importing pi files and the time zone is not available in the pi file.

timeZoneName

Enumeration of supported time zones. See appendix B for list of supported time zones. This time zone is used when importing pi files and the time zone is not available in the pi file.

time0Format

The date time format of the %TIME0% variable and the %CURRENT_TIME% variable. This date time format The date time format of the %END_DATE_TIME% variable. This date time format is only used if there is no date time format specified in the arguments for the variable. yyyy = Year, MM = Month in year, dd = Day in month, HH = Hour in day (0-23), mm = Minute in hour, ss = Second in minute.

startDateTimeFormat

The date time format of the %START The %END_DATE_TIME% variable only works in an importStateActivity (since 2014.02) and inside a template file for an exportCustomFormatRunFileActivity (since 2013.02).

dateTimeFormat

Since stable build 2015.01. Definitions of date time formats that can be used as arguments for the %TIME0%, %START_DATE_TIME% and %END_DATE_TIME% variables, e.g.: %START_DATE_TIME(idOfADateFormat)% %END_DATE_TIME(idOfAnotherDateFormat)%. Date time formats specified as arguments overrule the startDateTimeFormat and endDateTimeFormat specified above. . This date time format is only used if there is no date time format specified in the arguments for the variable. yyyy = Year, MM = Month in year, dd = Day in month, HH = Hour in day (0-23), mm = Minute in hour, ss = Second in minute. The %START_DATE_TIME% and %END_DATE_TIME% variables only work variable only works in an importStateActivity (since 2014.02) and inside a template file for an exportCustomFormatRunFileActivity (since 2013.02).

...

endDateTimeFormat

The date time format

...

of the %END_DATE_TIME% variable. This date time format is only used if there is no date time format specified in the

...

arguments for the variable. yyyy = Year, MM = Month in year, dd = Day in month, HH = Hour in day (0-23), mm = Minute in hour, ss = Second in minute.

...

ensembleMemberCount

Defines if ensembles are read from or written to a number of sub directories.

modelTimeStep

The model time step adjusts the end time when necessary of the run period for this model and for all models/modules that will run after this model in the same task run.

Burn-In Profile

Burn-in profile for cold state starts. Used to replace first part of a timeseries.

For time series with matching parameter-location ids, the first value is replaced by the initialValue. Element length defines the length of timeseries beginning that is to be replaced using linear interpolation.

...

<burnInProfile>
   <length multiplier="4" unit="hour"/>
   <timeSeries>
      <parameterId>H.obs</parameterId>
      <locationSetId>locationInitialValueAttSet</locationSetId>
      <initialValueAttributeId>initialValue</initialValueAttributeId>
   </timeSeries>
</burnInProfile>

The %END_DATE_TIME% variable only works in an importStateActivity (since 2014.02) and inside a template file for an exportCustomFormatRunFileActivity (since 2013.02).

dateTimeFormat

Since stable build 2015.01. Definitions of date time formats that can be used as arguments for the %TIME0%, %START_DATE_TIME% and %END_DATE_TIME% variables, e.g.: %START_DATE_TIME(idOfADateFormat)% %END_DATE_TIME(idOfAnotherDateFormat)%. Date time formats specified as arguments overrule the startDateTimeFormat and endDateTimeFormat specified above. The %START_DATE_TIME% and %END_DATE_TIME% variables only work in an importStateActivity (since 2014.02) and inside a template file for an exportCustomFormatRunFileActivity (since 2013.02).

• id: Identifier for this date time format. Use this identifier to refer to this date time format in the argument of a variable.
• dateTimePattern: Pattern for this date time format. This can be e.g. "yyyy-MM-dd HH:mm:ss". yyyy = Year, MM = Month in year, dd = Day in month, HH = Hour in day (0-23), mm = Minute in hour, ss = Second in minute.

ensembleMemberCount

Defines if ensembles are read from or written to a number of sub directories.

modelTimeStep

The model time step adjusts the end time when necessary of the run period for this model and for all models/modules that will run aft05GeneralAdapterModule-importShapeFileActivityer this model in the same task run.

Burn-In Profile

Burn-in profile for cold state starts. Used to replace first part of a timeseries.

For time series with matching parameter-location ids, the first value is replaced by the initialValue. Element length defines the length of timeseries beginning that is to be replaced using linear interpolation.

<burnInProfile>
   <length multiplier="4" unit="hour"/>
   <timeSeries>
      <parameterId>H.obs</parameterId>
      <locationSetId>locationInitialValueAttSet</locationSetId>
      <initialValueAttributeId>initialValue</initialValueAttributeId>
   </timeSeries>
</burnInProfile>

length

Length of time series beginning that is to be replaced.

timeSeries

parameterId and locationId or locationSetId =>

length

Length of time series beginning that is to be replaced.

timeSeries

parameterId and locationId or locationSetId => to match timeSeries this burn in profile should apply to

...

Please note that it is possible to delete files outside the general adapter root folder, so it is recommended to always start the filter path with %ROOT_DIR% to prevent this from happening accidentally.

Minimum file age can optionally be used to only purge files over a certain age.

Example (note the use of tags to define the root directory name):

	<startUpActivities>
		<purgeActivity>
			<filter>%ROOT_DIR%/temp</filter>
		</purgeActivity>
		<purgeActivity>
			<filter>%ROOT_DIR%/logs/*.*</filter>
		</purgeActivity>
            <minimumFileAge  <purgeActivity>
			<includeSubdirectories>true</includeSubdirectories>
unit="day" multiplier="2"/>
        </purgeActivity>
        <purgeActivity>
			<includeSubdirectories>true</includeSubdirectories>
            <filter>%ROOT_DIR%/importDir/*.*</filter>
        </purgeActivity>
    </startUpActivities>

...

Name (and location) of the PI-XML file with exported time series. If the directory location is not explicitly specified the file will be written in the exportDir defined in the general section.

ensembleMemberFormat

Available since 2019.02. Canhave value 'name', 'index' or 'hide'.  If 'name' is configured, the ensemble member Id is written, with 'index' the ensemble member index is written. When using 'hide', the ensemble member information is omitted from the exported timeseries.

exportBinFile

When true the events in the PI time series file are written to a binary file instead of the xml file. The written xml file will only contain the time series headers and optionally a time zone. The binary file has the same name as the xml file only the extension is "bin" instead of "xml". During PI time series import the bin file is automatically read when available. The byte order in the bin file is always Intel x86.

...

Configuration of the exportProfiles activity is identical to the exportTimeSeries Activity.

...

exportDataSetActivity

Figure 73 Elements of the exportDataSets section

...

exportLocationAttributesCsvActivity (since 2020.01)

Exports location attributes attributes of a location set or a single location to csv format, including multi-value attributes. Skips by default when encountering empty or non-existing location sets.

...

Optional. Default value is netcdf3. Currently supported other option is netcdf4.

metadata

Since 2023.01 metadata can be added to the netcdf global attributes, just like the implementation for normal netcdf exports.

geoDatum

Optional. To configure reprojection of location coordinates you can specify the required output geodatum here. This option is available only for scalar datasets.

...

• description - Optional description for the activity.
• stateConfigFile - Optional, do not use this if a pi state description xml file is not needed. Name (and location) of the PI-XML file describing the states to be imported. This file contains all necessary information to define state type and location. The moduleInstanceId of the state imported is per definition the current module instance. This should be either an absolute path or a path relative to the importDir defined in the general section. This option reads the output state file paths from a pi state description xml file.
• stateImportDir - Optional state import directory. This should be either an absolute path or a path relative to the importDir defined in the general section. If this stateImportDir is specified, then the importFile paths in this activity can be paths relative to this stateImportDir instead of absolute paths. This stateImportDir is only needed when you want to use relative importFile paths. If all importFile paths in this activity are absolute, then this stateImportDir is not needed and will not be used. This option does not use a pi state description xml file.
• stateFile - One or more output state files from the model. The specified files will be imported. This option does not use a pi state description xml file.
• importFile - Path and name of an output state file from the model. This file will be imported. This should be either an absolute path or a path relative to the stateImportDir defined in this activity. The %END_DATE_TIME% tag can be used here (since 2014.02), in case the state file name contains a timestamp of the end time of the run. Important: the state time will be saved at the time of the last timesteps exported from the GA. For example, if you want the state to be saved at T0 but you export a timeseries from the GA which is exceeds T0, the time at which the state is saved will be the last available timestamp in that particular timeseries. Use the <ignoreRunPeriod> option the export timeseries activity to ignore timeseries to be included in the check at which time the state is saved!
• relativeExportFile - This relativeExportFile - This relative path and name are used to store the imported file in FEWS. This path should be relative to the stateExportDir in the exportStateActivity that will be used to export this state file again for a future model run. If the imported output state file needs to be renamed before it can be used as input state file for a future model run, then the name of the relativeExportFile can be different from the name of the importFile. The %END_DATE_TIME% tag can be used here (since 2014.02), in case the state file name contains a timestamp of the end time of the run.
• compressedStateLocation - Optional. By default the warm state is zipped and stored as a blob in the database. For large states (>50MB) it is recommended to store the data outside the database in a directory. This directory is configured in the clientConfig.xml (optional element warmStatesDirectory). When using a stand alone system or using oracle and a single MC system it is possible to store larger grids inside the database. Expired states are removed automatically from this directory by the CompactCacheFiles workflow. The state description is still written to the database, the blob field will be empty.
• expiryTime - Optional. When the state is an intermediate result in a forecast run you can make the state expire. By default the expiry time is the same as for the module instance run.
• synchLevel - Optional synch level for state. Defaults to 0 is not specified (i.e. same as data generated by the forecast run)

...

• maximumSnapDistance Since 2014.01. FEWS-10771. Optional maximum horizontal snap distance in meters. When the parser provides horizontal location coordinates (x,y) and no locationIds, then the location mapping will be done by matching the horizontal coordinates. The horizontal snap distance is the tolerance used to detect which internal and external horizontal coordinates are the same. This only works when the input format provides the coordinate system for the coordinates of the locations. When the parser does not provide the coordinates for a time series an error is logged. Note: this option has no effect for grid data. Note 2: it is not possible to import data using horizontal coordinates and using locationIds in the same importNetcdfActivity, need to define separate import activities for that (one with maximumSnapDistance and one without maximumSnapDistance).
• maximumVerticalSnapDistance Since 2014.02. Optional maximum vertical snap distance in meters. When the parser provides vertical location coordinates (z) and no locationIds, then the location mapping will be done by matching the vertical coordinates. The vertical snap distance is the tolerance used to detect which internal and external vertical coordinates are the same. This only works when the input format provides the coordinates of the locations. When the parser does not provide the vertical coordinates for a time series an error is logged. Note: this option currently only works for importing horizontal layers from netcdf 3D grid data. Note 2: it is not possible to import data with z-coordinates (layers from 3D grids) and data without z-coordinates (2D grids) in the same importNetcdfActivity, need to define separate import activities for that (one with maximumVerticalSnapDistance and one without maximumVerticalSnapDistance).
• startWhileRunningExecuteActivities Default is false. If this is true, then this importActivity will run continuously during the configured execute activities. Additionally this importActivity will also run as part of the configured import activities as normal. This way it is possible to import data that is produced by an execute activity, while it is being produced. For instance if a model run writes new output data to an existing file after each timeStep, then the continuously running importActivity will immediately import the file, including the new data. This way the new data can be viewed in FEWS as soon as it becomes available, i.e. already during the model run. Currently the data that is imported during the execute activities can only be viewed after selecting "open most recent running forecast and adjust system time" from the debug menu in the FEWS Explorer window. If the running forecast is opened and selected in the data viewer, then the displays are updated each time when new data becomes available during the run. This feature only has effect for stand alone FEWS systems and for FEWS systems that use direct database access.FEWS systems and for FEWS systems that use direct database access.
  

import multiple files at once

• folder Import all file in the specified folder.
• fileNamePatternFilter Only import files that match the pattern. (e.g. *.nc)
• fileNameLocationIdPattern. Since 2023.02. Regular Expression. When a match of the pattern in the filename is found, this will overrule the external grid location Id for the time series being imported A simple pattern is (.*) which matches the whole filename. (.*)\.nc extracts the file name without the .nc extension. An other simple pattern is .{2}(.*).{4} that removes the first 2 and last 4 character of the filename to get the id More complicated expressions can be found at http://en.wikipedia.org/wiki/Regular_expression

 <importNetcdfActivity>
	<folder>%ROOT_DIR%/importDir</folder>
	<fileNamePatternFilter>*.nc</fileNamePatternFilter>
	<fileNameLocationIdPattern>(.*)\.nc</fileNameLocationIdPattern>
	<timeSeriesSets>
		<timeSeriesSet>
			<moduleInstanceId>GeneralAdapterRun</moduleInstanceId>
			<valueType>scalar</valueType>
			<parameterId>WaterLevel</parameterId>
			<locationId>H-2001</locationId>
			<timeSeriesType>external historical</timeSeriesType>
			<timeStep unit="minute" divider="1" multiplier="15"/>
			<readWriteMode>add originals</readWriteMode>
			<ensembleId>prognose</ensembleId>
		</timeSeriesSet>
	</timeSeriesSets>
	<ignoreNonExistingLocationSets>true</ignoreNonExistingLocationSets>
</importNetcdfActivity>

Combined with workflow ensemble loop

...

A <fileDateTimePattern> should be defined to filter out the shape files that need to be imported and to extract the time for the shape in the time series. If the directory is not empty, but none of the files match the defined pattern, an exception will be thrown. Since 2023.01 the fileDateTimePattern is optional. When there is no fileDateTimePattern the *.shp file in the import directory will be imported. The time zero is used as time stamp. Only one shp-file should exist in the import directory. Files without the shp/dbf extension are ignored

A <geoDatum> can be defined if the shape file is not in WGS84. When this is defined, a conversion to WGS84 will take place before storing the time series into the database.

...

<importActivities>
	<importShapeFileActivity>
		<shapeFileImportDir>importshapefileactivityimportdir</shapeFileImportDir>
		<fileDateTimePattern>'ImportShapeFileActivity_'ddMMMyyyyHHmmss'.shp'</fileDateTimePattern>
		<geoDatum>$GEODATUM$</geoDatum>
		<charset>ISO-8859-1</charset>
		<shapeFileAttribute attributeId="Range Min" propertyKey="Min"/>
		<shapeFileAttribute attributeId="Range Max" propertyKey="Max"/>
		<timeSeriesSet>
			<moduleInstanceId>ImportActivityNoneEquidistantProfile</moduleInstanceId>
			<valueType>polygon</valueType>
			<parameterId>H.obs</parameterId>
			<locationId>SX.E7842</locationId>
			<timeSeriesType>external historical</timeSeriesType>
			<timeStep unit="nonequidistant"/>
			<relativeViewPeriod unit="hour" start="0" end="2"/>
			<readWriteMode>read only</readWriteMode>
		</timeSeriesSet>
	</importShapeFileActivity>
</importActivities>

Since 2024.01 it also to reference a single import file instead of a directory with pattern. The time0 will be used as timestamp

<activities>
		<importActivities>
			<importShapeFileActivity>
				<importFile>importshapefileactivityimportdir/test.shp</importFile>

</importActivities>

The (placeholder) location used for this polygon is defined in Locations.xls in the same way you define a placeholder location for a grid. 

...