Wiki Markup |
---|
scrollbar |
Table of Contents |
---|
Introduction
The Data Conversion Module (DCM) is a generic application with which timeseries data can be converted. The DCM is a ‘stripped’ version of Delft FEWS and therefore supports the same import and export file formats. It is also possible to perform some transformations that are available in the transformation module of Delft-FEWS.
...
Figure 1: Illustration of interaction between Data Interface Module (DiM) and Data Conversion Module (DCM).
Configuration
File structure of DCM after installation
Figure 2: Overview of directory structure after installation of the DCM.
After installation of the DCM your system will contain the following files (see Figure 2):
- bin Directory containing a subset of the Delft-FEWS jars.
- DataConversion_SA FEWS Region Home directory for the DCM (contains the XML-configuration files)
- jre7 Java Runtime Environment
- DataConversion.bat Script file needed to run DCM on a Windows machine
- DataConversion.sh Script file needed to run DCM on a Linux machine
- lcp.bat Helper file for DataConversion.bat
This is the complete set of files needed to run the DCM. In the directory DataConversion_SA one can find all configuration files of the DCM (see Figure 3). The DCM configuration is based upon the Delft-FEWS configuration; this implies that its configuration is distributed over several configuration files. All configuration files are written in XML format.
Figure 3: Example of DataConversion_SA directory. Note the dataconversion.xml in the root configuration.
DCM configuration file
In Figure 3 the dataconversion file can be distinguished, this file is the DataConversion configuration file. The DCM configuration file contains all required instructions in order to run the DCM. This configuration file is an XML file that is structured according to the schema file ‘dataconversion.xsd’ which can be found in the release package.
Figure 4: DataConversion schema
The DataConversion configuration file contains a list of activities which will be run sequentially. There is no limit to the number and order of the activities. It is possible to choose from the following list of activities:
PurgeActivity:
With this activity it is possible to delete both files and directories. This activity supports the wildcards ‘?’ and ‘*’.
This activity contains the following fields:
- id: required idendifier.
- description: optional information
- purgeFilePath: path to files that are to be deleted. Wildcards accepted.
Figure 5 DataConversion purge activity
CopyActivity:
With this activity it is possible to copy files from one location to another. This activity supports the wildcards ‘?’ and ‘*’.
This activity contains the following fields:
- id: required idendifier.
- description: optional information
- srcFilePath: path to files that are to be copied. Wildcards accepted.
- destFilePath: path to destination directory. No wildcards allowed.
- prefix: optional text that can be added before the name of the destination file.
- suffix: optional text that can be added after the name of the destination file.
- setCurrentFileNameProperty: option that sets the name of the file currently being copied as a global property. This property can then be used in the configuration of other activities. This option only works when this activity is nested in a ‘runInLoopActivityRunner’ activity.
Binaries / installation
For FEWS versions 2018 and earlier The DCM had a separate bin folder and its own patches and was using the installed Java runtime version 1.8 (or earlier) These can still be requested through FEWS Support.
Starting with FEWS release 2019.02 rather then distributing a separate bin folder (which would need to include Java 11 runtime files) the choice was made to include DCM in the main FEWS distribution. The code of the data conversion module is now part of the main Delft_FEWS.jar and is run using the Delft-FEWSc.exe executable. The deliverable package for DCM that can be requested through FEWs support now contains a standalone configuration to use as a template for you project and a startup script (DataConversion.sh / DataConversion.bat) that you can use to start the DCM.
To install the DCM, the procedure is pretty much like any FEWS installation, the main difference difference is that after installing a base-build, you request from the same branch the data conversion package, with the file name dataconversion-stable-NNNN,NN.zip (where NNNN.NN is the FEWS version number) and unzip this as your region home folder. Then finally you get the latest FEWS patch for the given branch, however instead of putting the patch in the region home folder you need to put it in the main DCM folder and rename it to patch.jar.
Configuration
File structure of DCM after installation
Figure 2: Overview of directory structure after installation of the DCM.
After installation of the DCM your system will contain the following files (see Figure 2):
- bin Directory containing the FEWS distribution (2019.02 or newer)
- patch.jar latest patch file for this FEWS distribution
- DataConversion_SA FEWS Region Home directory for the DCM (contains the XML-configuration files)
- DataConversion.bat Script file needed to run DCM on a Windows machine
- DataConversion.sh Script file needed to run DCM on a Linux machine
This is the complete set of files needed to run the DCM, the FEWS . In the directory DataConversion_SA one can find all configuration files of the DCM (see Figure 3). The DCM configuration is based upon the Delft-FEWS configuration; this implies that its configuration is distributed over several configuration files. All configuration files are written in XML format.
Figure 3: Example of DataConversion_SA directory. Note the dataconversion.xml in the root configuration.
DCM configuration file
In Figure 3 the dataconversion file can be distinguished, this file is the DataConversion configuration file. The DCM configuration file contains all required instructions in order to run the DCM. This configuration file is an XML file that is structured according to the schema file ‘dataconversion.xsd’ which can be found in the release package.
Code Block | ||
---|---|---|
<complexType name="DataConversionComplexType | ||
Code Block | ||
| ||
<runInLoopActivityRunner id="runInLoop"> <activities><sequence> <element <!— This CopyActivity will copy all XML files form the Import/aquo directory. However because The activity is being run in a loop each file will be copied individually. By setting the property 'setCurrentFileNameProperty' the name of the file being copied will be mapped to the key CURRENT_FILENAME. This key can then be used by the other activities as shown below. --> name="clearOnStartup" type="boolean" default="true" minOccurs="0"> <annotation> <documentation>Control if local datastore is cleared on startup. By default eacht new run of the DCM will start with an empty datastore.</documentation> <copyActivity id="loopedCopy"> </annotation> <srcFilePath>%REGION_HOME%/TestData/Import/aquo/*.xml</srcFilePath> </element> <element name="activities" type="fews:ActivitiesComplexType"/> <destFilePath>$IMPORT_FOLDER_ROOT$/aquo/</destFilePath>sequence> </complexType> <!--WftrActivitiesComplexType --> <complexType name="ActivitiesComplexType"> <setCurrentFileNameProperty>CURRENT_FILENAME</setCurrentFileNameProperty> <sequence> </copyActivity> <choice minOccurs="0" maxOccurs="unbounded"> <element <moveActivity idname="purgeActivity" type="loopedMovefews:PurgeActivityComplexType"> <srcFilePath>$EXPORT_FOLDER_ROOT $/aquo/*</srcFilePath> <annotation> <destFilePath>$BACKUP_FOLDER_ROOT$/aquo/$CURRENT_FILENAME$.xml</destFilePath> <documentation>Purges a single file or a set of files within a directory.</documentation> </moveActivity>annotation> ... |
...
WorkflowActivity:
With this activity FEWS workflows can be run.
This activity contains the following fields:
- description: optional information
- workflowId: id of workflow to be run
Figure 7 DataConversion workflow activity
MoveActivity:
With this activity it is possible to move files from one location to another. This activity supports the wildcards ‘?’ and ‘*’.
This activity contains the following fields:
- id: required idendifier.
- description: optional information
- srcFilePath: path to files that are to be copied. Wildcards accepted.
- destFilePath: path to destination directory. No wildcards allowed.
- setCurrentFileNameProperty: option that sets the name of the file currently being moved as a global property. This property can then be accessed by other activities. This option only works when this activity is nested in a ‘runInLoopActivityRunner’ activity.
Figure 8 DataConversion move activity
ImportStatusActivity:
With this activity the content of the ImportStatus table can be exported to CSV file. The ImportStatus is information produced by the Delft FEWS import modules, describing basic information about the last import run.
This activity contains the following fields:
...
</element>
<element name="copyActivity" type="fews:CopyActivityComplexType">
<annotation>
<documentation>Copies single file or a set of files from source directory to a destination
directory. It is possible to add prefix or suffix to original file names.</documentation>
</annotation>
</element>
<element name="workflowActivity" type="fews:WorkflowActivityComplexType">
<annotation>
<documentation>Define FEWS workflow to run.</documentation>
</annotation>
</element>
<element name="moveActivity" type="fews:MoveActivityComplexType">
<annotation>
<documentation>Move single file or a set of files from source directory to a destination
directory.</documentation>
</annotation>
</element>
<element name="importStatusActivity" type="fews:ImportStatusActivityComplexType">
<annotation>
<documentation>Exports the import status to file. When running import in loop import status
must be run</documentation>
</annotation>
</element>
<element name="runInLoopActivityRunner" type="fews:RunInLoopActivityRunnerComplexType">
<annotation>
<documentation>Repeats the sub activities until their run method returns 'false'.</documentation>
</annotation>
</element>
</choice>
</sequence>
</complexType> |
Figure 4: DataConversion schema
As of version 2015.02 the configuration file offers the option 'clearOnStartup' which makes the deletion of the local datastore at startup configurable.
The DataConversion configuration file contains a list of activities which will be run sequentially. There is no limit to the number and order of the activities. It is possible to choose from the following list of activities:
PurgeActivity:
With this activity it is possible to delete both files and directories. This activity supports the wildcards ‘?’ and ‘*’.
This activity contains the following fields:
- id: required idendifier.
- description: optional information
- purgeFilePath: path to files that are to be deleted. Wildcards accepted.
Figure 5 DataConversion purge activity
CopyActivity:
With this activity it is possible to copy files from one location to another. This activity supports the wildcards ‘?’ and ‘*’.
This activity contains the following fields:
- id: required idendifier.
- description: optional information
- srcFilePath: path to files that are to be copied. Wildcards accepted.
- destFilePath: path to destination directory. No wildcards allowed.
- prefix: optional text that can be added before the name of the destination file.
- suffix: optional text that can be added after the name of the destination file.
- setCurrentFileNameProperty: option that sets the name of the file currently being copied as a global property. This property can then be used in the configuration of other activities. This option only works when this activity is nested in a ‘runInLoopActivityRunner’ activity.
Code Block | ||
---|---|---|
| ||
<runInLoopActivityRunner id="runInLoop">
<activities>
<!— This CopyActivity will copy all XML files form the Import/aquo directory. However because
The activity is being run in a loop each file will be copied individually. By setting the property 'setCurrentFileNameProperty' the name of the file being copied will be mapped to the key CURRENT_FILENAME. This key can then be used by the other activities as shown below. -->
<copyActivity id="loopedCopy">
<srcFilePath>%REGION_HOME%/TestData/Import/aquo/*.xml</srcFilePath>
<destFilePath>$IMPORT_FOLDER_ROOT$/aquo/</destFilePath>
<setCurrentFileNameProperty>CURRENT_FILENAME</setCurrentFileNameProperty>
</copyActivity>
<moveActivity id="loopedMove">
<srcFilePath>$EXPORT_FOLDER_ROOT $/aquo/*</srcFilePath>
<destFilePath>$BACKUP_FOLDER_ROOT$/aquo/$CURRENT_FILENAME$.xml</destFilePath>
</moveActivity>
...
|
Figure 6 DataConversion copy activity
WorkflowActivity:
With this activity FEWS workflows can be run.
This activity contains the following fields:
- description: optional information
- workflowId: id of workflow to be run
Figure 7 DataConversion workflow activity
MoveActivity:
With this activity it is possible to move files from one location to another. This activity supports the wildcards ‘?’ and ‘*’.
This activity contains the following fields:
- id: required idendifier.
- description: optional information
- srcFilePath: path to files that are to be copied. Wildcards accepted.
- destFilePath: path to destination directory. No wildcards allowed.
- setCurrentFileNameProperty: option that sets the name of the file currently being moved as a global property. This property can then be accessed by other activities. This option only works when this activity is nested in a ‘runInLoopActivityRunner’ activity.
Figure 8 DataConversion move activity
ImportStatusActivity:
With this activity the content of the ImportStatus table can be exported to CSV file. The ImportStatus is information produced by the Delft FEWS import modules, describing basic information about the last import run.
This activity contains the following fields:
- id: required idendifier.
- description: optional information
- importStatusPath: path to the directory in which status file is to be generated.
- dataFeedId: option to filter the ‘dataFeedId’ values that are to be exported to file. If omitted all available datafeed information will be exported.
Figure 9 DataConversion import status
RunInLoopActivity:
With this activity one or more of the above can be run repeatedly. This activity will keep looping over the sub activities until one of the sub activities indicates that it has completed all its loops or until the ‘timeOut’ has been exceeded.
This activity contains the following fields:
- id: required idendifier
- description: optional information
- activities: list of activities over which to loop
- timeOutSeconds: option timeout in seconds after which looping is terminated.
Figure 10 DataConversion run in loop activity
Configuration of DCM: conversion task variants
The DCM makes use of the file based Delft-FEWS configuration. Hence, all modules that are available in Delft-FEWS can also be used for the DCM. The configuration is located in the Config directory of the 'region_home' directory (see Figure 2). In this section some general remarks about the Delft-FEWs configuration in relation to the DCM can be found. We do this by defining three 'conversion tasks' with increasing complexity:
- Variant 1: workflow with import and export module.
- Variant 2: workflow with import, transformation (only on new data) and export module.
- Variant 3: workflow with import, transformation (on new data and on data of previous DCM runs) and export module .
Variant 1: Import, Export.
This is the simplest workflow variant. The workflow only consists out of an import and an export module. This variant can be used when the original import files already have the correct timestep and when location specific transformations are not needed. In this case the DCM is mainly used to convert the file format of the import files. In the Figure below a schematic illustration of this workflow variant is given.
Figure 11 Simplest workflow variant in DCM. The DCM is only used for conversion of the file format, there is no need for transformations.
The DCM support the TimeSeriesImportRun module. This implies that all import types available within Delft-FEWS are also available to the DCM. To minimize the amount of configuration, it is possible to configure imports without having to specify the timeseries information in the import module. Here is an example for importing PI-Xml:
Code Block | ||
---|---|---|
| ||
<?xml version="1.0" encoding="UTF-8"?>
<timeSeriesImportRun xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:
schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/timeSeriesImportRun.xsd">
<import>
<general>
<importType>LmwWeatherdata</importType>
<folder>$IMPORT_FOLDER_ROOT$/LMW_maasaa</folder>
<idMapId>IdImport_LMWmaasaa</idMapId>
</general>
<temporary>true</temporary>
</import>
</timeSeriesImportRun>
|
In the above example it can be seen that the configuration of TimeSeriesSets has been replaced by the element ‘temporary’. This way it is no longer necessary to configure locations and parameters in the RegionConfig directory. These are generated on-the-fly during the import process. All imported data is stored as temporary data and will only be available to other tasks that are being executed within the same workflow as the import module. However, if you want to make use of the locationSets than it is necessary to configure both the locations and the locationSets in the RegionConfig directory.
As is the case with the import run, the export no longer requires the configuration of TimeSeriesSets. The export run will automatically export all data imported in the previous import actions of the same workflow run.
Code Block | ||
---|---|---|
| ||
<timeSeriesExportRun xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews
http://fews.wldelft.nl/schemas/version1.0/timeSeriesExportRun.xsd">
<export>
<general>
<exportType>NETCDF-CF_TIMESERIES</exportType>
<folder>$EXPORT_FOLDER_ROOT$/LMW_maasaa</folder>
<exportFileName>
<name>_LMWmaasaa.nc</name>
<prefix>
<timeZeroFormattingString>yyyy-MM-dd'T'HHmmss</timeZeroFormattingString>
</prefix>
</exportFileName>
<unitConversionsId>ExportUnitConversions</unitConversionsId>
<exportMissingValueString>-9999.0</exportMissingValueString>
<exportTimeZone>
<timeZoneOffset>+01:00</timeZoneOffset>
</exportTimeZone>
</general>
<properties>
<bool value="false" key="includeFlags"/>
<bool value="false" key="includeComments"/>
</properties>
<metadata>
<title>Export of LMW_maasaa datafeed</title>
<institution>Deltares</institution>
</metadata>
</export>
</timeSeriesExportRun> |
Variant 2: Import, Transformation, Export.
The second workflow variant is similar to the first workflow variant. Here, the timestep of the original timeseries is also correct but additional transformations are required. A transformation module is added to the workflow to allow for location specific conversions (e.g. correction of water level) and/or the derivation of new timeseries. For the transformation module it remains necessary to explicitly configure TimeSeriesSet information. However it is not necessary to pre configure parameters and locations in the RegionConfig directory. As long as the required timeseries have been imported during a prior import run they will be available for the transformation.
In the Figure below a schematic illustration of workflow variant 2 is given:
Figure 12 Using Transformation functionality to change timeseries.
It is possible to filter the data that is to be exported by configuring TimeSeriesSet information. But in that case the locations and parameter information must be added to the export configuration (see export example in Variant 2). Here is an example of a PI-export that includes a filter:
Code Block | ||
---|---|---|
| ||
<?xml version="1.0" encoding="UTF-8"?>
<timeSeriesExportRun xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews
http://fews.wldelft.nl/schemas/version1.0/timeSeriesExportRun.xsd">
<export>
<general>
<exportType>PI</exportType>
<folder>$EXPORT_FOLDER_ROOT$/LMW_lixhe</folder>
<exportFileName>
<name>_LMWlixhe.txt</name>
<prefix>
<timeZeroFormattingString>yyyy-MM-dd'T'HHmmss</timeZeroFormattingString>
</prefix>
</exportFileName>
<unitConversionsId>ExportUnitConversions</unitConversionsId>
<exportMissingValueString>-9999.0</exportMissingValueString>
<exportTimeZone>
<timeZoneOffset>+01:00</timeZoneOffset>
</exportTimeZone>
</general>
<properties>
<bool value="false" key="includeFlags"/>
<bool value="false" key="includeComments"/>
</properties>
<metadata>
<title>Export of LMW_lixhe datafeed</title>
<institution>Deltares</institution>
</metadata>
<timeSeriesSet>
<moduleInstanceId>LMWlixhe_interpolation</moduleInstanceId>
<valueType>scalar</valueType>
<parameterId>C10P</parameterId>
<locationSetId>LMWlixhe</locationSetId>
<timeSeriesType>temporary</timeSeriesType>
<timeStep unit="minute" multiplier="10"/>
<relativeViewPeriod unit="hour" start="-3" end="0"/>
<readWriteMode>read complete forecast</readWriteMode>
</timeSeriesSet>
</export>
</timeSeriesExportRun> |
Variant 3: Import, (dis)Aggregation, Export.
The third workflow variant is used when the original import data does not have the appropriate timestep. In Delft-FEWS it is relatively simple to convert the original import data to the appropriate timestep by using a (dis-)aggregation function.
However, in order to perform a (dis-)aggregation it is necessary to have data for multiple timesteps. In an operational system the following aspects have to be taken into account:
- It is likely that external data is delivered per timestep. Hence, these files only contain data for one single timestep.
- One of the characteristics of the DCM is that it removes all information from its localDataStore when Delft-FEWS is closed. Consequently, import files from previous DCM runs are not available in the current DCM run.
This means that simply adding a (dis-)aggregation transformation will not work because there is only information for a single timestep. In order to prevent this problem several modules are added to the workflow which enable the ‘storage’ of information from previous timesteps.
In Figure 13 a schematic illustration of this third workflow variant is given. In this workflow an additional import from the back-up folder is added. The files from the back-up folder contain the import data from the previous DCM runs. In the second step of this workflow the files from both imports are merged. The merge transformation contains a hierarchy which states that the (new) import data is more important than the (old) back-up import data. The result of this merge transformation is a new timeseries which contains both new and old import data. Part of this new timeseries is exported to the back-up folder to serve as back-up timeseries in the next DCM run.
This approach guarantees the availability of a timeseries that contains data for multiple timesteps. It is now possible to carry out a (dis-)aggregation transformation to convert the import data to the appropriate timestep.
Figure 13 This construction is needed whenever functionalities are used that need data from previous DCM runs. The extra export/import routine ensures that the user has data from previous DCM runs.
Run the DCM
The DCM can be executed as a script from the command line or as a scheduled task (see Figure 2 for the location of these files).
- On a Windows machine the DCM is run using the BATCH file: DataConversion.bat
- On a Linux machine the DCM is run using the bash script file: DataConversion.sh
Both scripts require the same arguments:
- regionpath=<name or path> : Name of region directory containing all configuration files (DataConversion_SA). If the region directory is not located in the same directory as the script then a path is required.
- configfile=<name or path> : Name of the DCM configuration file (here we use the dataconversion.xml). If the DCM configuration file is not located in the root of the region directory then a path is required.
- workflowid=<identifier> : Identifier of workflow that is to be run. This argument is required when the 'configfile' is not provided. Can be comma separated array of workflow ids.
- importpath=<path> : Path to folder containing import files. The value of this argument will overrule global property IMPORT_FOLDER_ROOT. It is therefore necessary that this property is configured in the global.properties file. Furthermore this argument only works in combination with 'workflowid' argument.
- exportpath=<path> : Path to where export files will be written. The value of this argument will overrule global property EXPORT_FOLDER_ROOT. It is therefore necessary that this property is configured in the global.properties file. Furthermore this argument only works in combination with 'workflowid' argument.
- systemtime=<timestamp> : Timestamp value (yyyy-MM-dd HH:mm:ssZ) to be used as System Time when running workflows. (note: Z for non-GMT time zones needs to be replaced by the timezone offset, i.e. +1000 for CET)
- binpath=<name or path> : Path to the directory containing the Delft-FEWS binaries (bin). This argument is configured in the script files.
- clearonstart=<true or false> : Option to turn off deletion of local datastore on startup. default is true.
- compact=<true or false> : Option to compact the datastore on startup using a 'rolling barrel' algorithm. default is false.
- loglevel=<error, warn, info or debug> : option to set the log level, default is info. You can set it to 'debug' for troubleshooting, 'warn' or even 'error' to filter out non-critical information.
...
Figure 9 DataConversion import status
RunInLoopActivity:
With this activity one or more of the above can be run repeatedly. This activity will keep looping over the sub activities until one of the sub activities indicates that it has completed all its loops or until the ‘timeOut’ has been exceeded.
This activity contains the following fields:
- id: required idendifier
- description: optional information
- activities: list of activities over which to loop
- timeOutSeconds: option timeout in seconds after which looping is terminated.
Figure 10 DataConversion run in loop activity
Configuration of DCM: conversion task variants
The DCM makes use of the file based Delft-FEWS configuration. Hence, all modules that are available in Delft-FEWS can also be used for the DCM. The configuration is located in the Config directory of the 'region_home' directory (see Figure 2). In this section some general remarks about the Delft-FEWs configuration in relation to the DCM can be found. We do this by defining three 'conversion tasks' with increasing complexity:
- Variant 1: workflow with import and export module.
- Variant 2: workflow with import, transformation (only on new data) and export module.
- Variant 3: workflow with import, transformation (on new data and on data of previous DCM runs) and export module .
Variant 1: Import, Export.
This is the simplest workflow variant. The workflow only consists out of an import and an export module. This variant can be used when the original import files already have the correct timestep and when location specific transformations are not needed. In this case the DCM is mainly used to convert the file format of the import files. In the Figure below a schematic illustration of this workflow variant is given.
Figure 11 Simplest workflow variant in DCM. The DCM is only used for conversion of the file format, there is no need for transformations.
The DCM support the TimeSeriesImportRun module. This implies that all import types available within Delft-FEWS are also available to the DCM. To minimize the amount of configuration, it is possible to configure imports without having to specify the timeseries information in the import module. Here is an example for importing PI-Xml:
Code Block | ||
---|---|---|
| ||
<?xml version="1.0" encoding="UTF-8"?>
<timeSeriesImportRun xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:
schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/timeSeriesImportRun.xsd">
<import>
<general>
<importType>LmwWeatherdata</importType>
<folder>$IMPORT_FOLDER_ROOT$/LMW_maasaa</folder>
<idMapId>IdImport_LMWmaasaa</idMapId>
</general>
<temporary>true</temporary>
</import>
</timeSeriesImportRun>
|
In the above example it can be seen that the configuration of TimeSeriesSets has been replaced by the element ‘temporary’. This way it is no longer necessary to configure locations and parameters in the RegionConfig directory. These are generated on-the-fly during the import process. All imported data is stored as temporary data and will only be available to other tasks that are being executed within the same workflow as the import module. However, if you want to make use of the locationSets than it is necessary to configure both the locations and the locationSets in the RegionConfig directory.
As is the case with the import run, the export no longer requires the configuration of TimeSeriesSets. The export run will automatically export all data imported in the previous import actions of the same workflow run.
Code Block | ||
---|---|---|
| ||
<timeSeriesExportRun xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews
http://fews.wldelft.nl/schemas/version1.0/timeSeriesExportRun.xsd">
<export>
<general>
<exportType>NETCDF-CF_TIMESERIES</exportType>
<folder>$EXPORT_FOLDER_ROOT$/LMW_maasaa</folder>
<exportFileName>
<name>_LMWmaasaa.nc</name>
<prefix>
<timeZeroFormattingString>yyyy-MM-dd'T'HHmmss</timeZeroFormattingString>
</prefix>
</exportFileName>
<unitConversionsId>ExportUnitConversions</unitConversionsId>
<exportMissingValueString>-9999.0</exportMissingValueString>
<exportTimeZone>
<timeZoneOffset>+01:00</timeZoneOffset>
</exportTimeZone>
</general>
<properties>
<bool value="false" key="includeFlags"/>
<bool value="false" key="includeComments"/>
</properties>
<metadata>
<title>Export of LMW_maasaa datafeed</title>
<institution>Deltares</institution>
</metadata>
</export>
</timeSeriesExportRun> |
Variant 2: Import, Transformation, Export.
The second workflow variant is similar to the first workflow variant. Here, the timestep of the original timeseries is also correct but additional transformations are required. A transformation module is added to the workflow to allow for location specific conversions (e.g. correction of water level) and/or the derivation of new timeseries. For the transformation module it remains necessary to explicitly configure TimeSeriesSet information. However it is not necessary to pre configure parameters and locations in the RegionConfig directory. As long as the required timeseries have been imported during a prior import run they will be available for the transformation.
In the Figure below a schematic illustration of workflow variant 2 is given:
Figure 12 Using Transformation functionality to change timeseries.
It is possible to filter the data that is to be exported by configuring TimeSeriesSet information. But in that case the locations and parameter information must be added to the export configuration (see export example in Variant 2). Here is an example of a PI-export that includes a filter:
Code Block | ||
---|---|---|
| ||
<?xml version="1.0" encoding="UTF-8"?>
<timeSeriesExportRun xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews
http://fews.wldelft.nl/schemas/version1.0/timeSeriesExportRun.xsd">
<export>
<general>
<exportType>PI</exportType>
<folder>$EXPORT_FOLDER_ROOT$/LMW_lixhe</folder>
<exportFileName>
<name>_LMWlixhe.txt</name>
<prefix>
<timeZeroFormattingString>yyyy-MM-dd'T'HHmmss</timeZeroFormattingString>
</prefix>
</exportFileName>
<unitConversionsId>ExportUnitConversions</unitConversionsId>
<exportMissingValueString>-9999.0</exportMissingValueString>
<exportTimeZone>
<timeZoneOffset>+01:00</timeZoneOffset>
</exportTimeZone>
</general>
<properties>
<bool value="false" key="includeFlags"/>
<bool value="false" key="includeComments"/>
</properties>
<metadata>
<title>Export of LMW_lixhe datafeed</title>
<institution>Deltares</institution>
</metadata>
<timeSeriesSet>
<moduleInstanceId>LMWlixhe_interpolation</moduleInstanceId>
<valueType>scalar</valueType>
<parameterId>C10P</parameterId>
<locationSetId>LMWlixhe</locationSetId>
<timeSeriesType>temporary</timeSeriesType>
<timeStep unit="minute" multiplier="10"/>
<relativeViewPeriod unit="hour" start="-3" end="0"/>
<readWriteMode>read complete forecast</readWriteMode>
</timeSeriesSet>
</export>
</timeSeriesExportRun> |
Variant 3: Import, (dis)Aggregation, Export.
The third workflow variant is used when the original import data does not have the appropriate timestep. In Delft-FEWS it is relatively simple to convert the original import data to the appropriate timestep by using a (dis-)aggregation function.
However, in order to perform a (dis-)aggregation it is necessary to have data for multiple timesteps. In an operational system the following aspects have to be taken into account:
- It is likely that external data is delivered per timestep. Hence, these files only contain data for one single timestep.
- One of the characteristics of the DCM is that it removes all information from its localDataStore when Delft-FEWS is closed. Consequently, import files from previous DCM runs are not available in the current DCM run.
This means that simply adding a (dis-)aggregation transformation will not work because there is only information for a single timestep. In order to prevent this problem several modules are added to the workflow which enable the ‘storage’ of information from previous timesteps.
In Figure 13 a schematic illustration of this third workflow variant is given. In this workflow an additional import from the back-up folder is added. The files from the back-up folder contain the import data from the previous DCM runs. In the second step of this workflow the files from both imports are merged. The merge transformation contains a hierarchy which states that the (new) import data is more important than the (old) back-up import data. The result of this merge transformation is a new timeseries which contains both new and old import data. Part of this new timeseries is exported to the back-up folder to serve as back-up timeseries in the next DCM run.
This approach guarantees the availability of a timeseries that contains data for multiple timesteps. It is now possible to carry out a (dis-)aggregation transformation to convert the import data to the appropriate timestep.
Figure 13 This construction is needed whenever functionalities are used that need data from previous DCM runs. The extra export/import routine ensures that the user has data from previous DCM runs.
Run the DCM
The DCM can be executed as a script from the command line or as a scheduled task (see Figure 2 for the location of these files).
- On a Windows machine the DCM is run using the BATCH file: DataConversion.bat
- On a Linux machine the DCM is run using the bash script file: DataConversion.sh
Both scripts require the same arguments:
- regionpath=<name or path> : Name of region directory containing all configuration files (DataConversion_SA). If the region directory is not located in the same directory as the script then a path is required.
- configfile=<name or path> : Name of the DCM configuration file (here we use the dataconversion.xml). If the DCM configuration file is not located in the root of the region directory then a path is required.
- workflowid=<identifier> : Identifier of workflow that is to be run. This argument is required when the 'configfile' is not provided.
- importpath=<path> : Path to folder containing import files. The value of this argument will overrule global property IMPORT_FOLDER_ROOT.
- exportpath=<path> : Path to where export files will be written. Thre value of this argument will overrule global property EXPORT_FOLDER_ROOT
- systemtime=<timestamp> : Timestamp value (yyyy-MM-dd HH:mm:ssZ) to be used as System Time when running workflows.
- binpath=<name or path> : Path to the directory containing the Delft-FEWS binaries (bin). This argument is configured in the script files.
Figure 14: Screenshot of the command prompt when running the DCM on a Windows machine.
...