| Table of Contents |
|---|
Introduction
The Data Conversion Module (DCM) is a generic application with which timeseries data can be converted. The DCM is a ‘stripped’ version of Delft FEWS and therefore supports the same import and export file formats. It is also possible to perform some transformations that are available in the transformation module of Delft-FEWS.
...
Figure 1: Illustration of interaction between Data Interface Module (DiM) and Data Conversion Module (DCM).
...
Configuration
File structure of DCM
Figure 2: Overview of directory structure after installation of the DCM.
...
Figure 3: Example of DataConversion_SA directory. Note the dataconversion.xml in the root configuration.
...
DCM configuration file
In order to run the DCM a configuration file is required containing all its run instructions. In the DCM test-package this is the XML file ‘dataconversion’ (see Figure 3). The dataconversion This configuration file is an XML file that is structured according to the schema file ‘dataconversion.xsd’ which can be found in the release package.
Figure 4: DataConversion schema
The DataConversion configuration file contains a list of activities which will be run sequentially. This list of activities contains references to the configured workflow files.
| Code Block | ||
|---|---|---|
| ||
<?xml version="1.0" encoding="UTF-8"?>
<dataConversion xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews file:/dataconversion.xsd">
<activities>
<!-- Run workflows from Config\WorkflowFiles -->
<workflowActivity>
<workflowId>Convert_maasaa</workflowId>
</workflowActivity>
<workflowActivity>
<workflowId>Convert_aberdeen</workflowId>
</workflowActivity>
<workflowActivity>
<workflowId>Convert_lixhe</workflowId>
</workflowActivity>
<copyActivity id="saveLog">
<srcFilePath>%REGION_HOME%/log.txt</srcFilePath>
<destFilePath>%REGION_HOME%/Logs</destFilePath>
</copyActivity>
<importStatusActivity id="ImportStatus">
<importStatusPath>%REGION_HOME%/ImportStatus</importStatusPath>
</importStatusActivity>
</activities>
</dataConversion>
|
Execute the DCM
There is no limit to the number and order of the activities. It is possible to choose from the following list of activities:
- PurgeActivity: With this activity it is possible to delete both files and directories. This activity supports the wildcards ‘?’ and ‘*’.
This activity contains the following fields:
id: required idendifier.
description: optional information
purgeFilePath: path to files that are to be deleted. Wildcards accepted.
Figure 5 DataConversion purge activity
- CopyActivity: With this activity it is possible to copy files from one location to another. This activity supports the wildcards ‘?’ and ‘*’.
This activity contains the following fields:
id: required idendifier.
description: optional information
srcFilePath: path to files that are to be copied. Wildcards accepted.
destFilePath: path to destination directory. No wildcards allowed.
prefix: optional text that can be added before the name of the destination file.
suffix: optional text that can be added after the name of the destination file.
setCurrentFileNameProperty: option that sets the name of the file currently being copied as a global property. This property can then be used in the configuration of other activities. This option only works when this activity is nested in a ‘runInLoopActivityRunner’ activity.
| Code Block | ||
|---|---|---|
| ||
<runInLoopActivityRunner id="runInLoop">
<activities>
<!— This CopyActivity will copy all XML files form the Import/aquo directory. However because
The activity is being run in a loop each file will be copied individually. By setting the property 'setCurrentFileNameProperty' the name of the file being copied will be mapped to the key CURRENT_FILENAME. This key can then be used by the other activities as shown below. -->
<copyActivity id="loopedCopy">
<srcFilePath>%REGION_HOME%/TestData/Import/aquo/*.xml</srcFilePath>
<destFilePath>$IMPORT_FOLDER_ROOT$/aquo/</destFilePath>
<setCurrentFileNameProperty>CURRENT_FILENAME</setCurrentFileNameProperty>
</copyActivity>
<moveActivity id="loopedMove">
<srcFilePath>$EXPORT_FOLDER_ROOT $/aquo/*</srcFilePath>
<destFilePath>$BACKUP_FOLDER_ROOT$/aquo/$CURRENT_FILENAME$.xml</destFilePath>
</moveActivity>
...
|
Figuur 6 DataConversion copy activity
- WorkflowActivity: With this activity FEWS workflows can be run.
This activity contains the following fields:
description: optional information
workflowId: id of workflow to be run
Figuur 7 DataConversion workflow activity
- MoveActivity: With this activity it is possible to move files from one location to another. This activity supports the wildcards ‘?’ and ‘*’.
This activity contains the following fields:
id: required idendifier.
description: optional information
srcFilePath: path to files that are to be copied. Wildcards accepted.
destFilePath: path to destination directory. No wildcards allowed.
setCurrentFileNameProperty: option that sets the name of the file currently being moved as a global property. This property can then be accessed by other activities. This option only works when this activity is nested in a ‘runInLoopActivityRunner’ activity.
Figuur 8 DataConversion move activity
- ImportStatusActivity: With this activity the content of the ImportStatus table can be exported to CSV file. The ImportStatus is information produced by the Delft FEWS import modules, describing basic information about the last import run.
This activity contains the following fields:
id: required idendifier.
description: optional information
importStatusPath: path to the directory in which status file is to be generated.
dataFeedId: option to filter the ‘dataFeedId’ values that are to be exported to file. If omitted all available datafeed information will be exported.
Figuur 9 DataConversion import status
- RunInLoopActivity: With this activity one or more of the above can be run repeatedly. This activity will keep looping over the sub activities until one of the sub activities indicates that it has completed all its loops or until the ‘timeOut’ has been exceeded.
This activity contains the following fields:
id: required idendifier
description: optional information
activities: list of activities over which to loop
timeOutSeconds: option timeout in seconds after which looping is terminated.
Figuur 10 DataConversion run in loop activity
Configuration of DCM: conversion task variants
Variant 1: Import, Export.
Variant 2: Import, Transformation, Export.
Variant 3: Import, (dis)Aggregation, Export.
Run the DCM
The DCM can be executed as a script from the command line or as a scheduled task (see Figure 2 for the location of these files).
- On a Windows machine the DCM is run using the BATCH file: DataConversion.bat
- On a Linux machine the DCM is run using the bash script file: DataConversion.sh
Both scripts require the same arguments:
- 1st argument argument: Name of region directory containing all configuration files (DataConversion_SA). If the region directory is not located in the same directory as the script then a path is required.
- 2nd argument argument: Name of the DCM configuration file (here we use the dataconversion.xml). If the DCM configuration file is not located in the root of the region directory then a path is required.
- 3rd argument argument: Path to the directory containing the Delft-FEWS binaries (bin). Can be omitted if the script is executed from within the ‘bin’ directory.
Figure 4 x: Screenshot of the command prompt when running the DCM on a Windows machine.
Configuration of DCM: conversion task variants
...
.
...
Variant 2: Import, Transformation, Export.
Variant 3: Import, (dis)Aggregation, Export.
...