Introduction

The Data Conversion Module (DCM) is a generic application with which timeseries data can be converted. The DCM is a ‘stripped’ version of Delft FEWS and therefore supports the same import and export file formats. It is also possible to perform some transformations that are available in the transformation module of Delft-FEWS.

Basically the DCM ensures that (1) Delft-FEWS starts up, (2) that the relevant workflows are run and (3) that Delft-FEWS is shut off. The workflows contain tasks that ensure the import of the available import files, the conversion of these imported files and finally the export of the converted files to an export folder. One of the basic DCM-properties is the deletion of the Delft-FEWS database (or localDataStore) after each DCM run. This means that information from previous DCM runs is not available.

The DCM can be used in combination with the Data Interface Module (DiM). The interaction between the DiM and the DCM is illustrated in Figure 1.1. The DiM collects data from an external source and puts these files into a predefined import folder. This import folder contains a sub-folder for each datafeed. The DCM imports all files from this import folder and carries out the required conversions on the data. The results of these conversions are being exported in the desired file format to a predefined export folder. The DiM collects the data from this location and distributes it to the data destination.

Figure 1: Illustration of interaction between Data Interface Module (DiM) and Data Conversion Module (DCM).

Running the DCM

File structure of DCM

Figure 2: Overview of directory structure after installation of the DCM.

After installation of the DCM your system will contain the following files (see Figure 2): 

• bin                                 Directory containing a subset of the Delft-FEWS jars.
• DataConversion_SA     FEWS Region Home directory for the DCM (contains the XML-configuration files)
• jre7                                Java Runtime Environment
• DataConversion.bat      Script file needed to run DCM on a Windows machine
• DataConversion.sh       Script file needed to run DCM on a Linux machine
• lcp.bat                            Helper file for DataConversion.bat 

This is the complete set of files needed to run the DCM. In the directory DataConversion_SA one can find all configuration files of the DCM (see Figure 3). The DCM configuration is based upon the Delft-FEWS configuration; this implies that its configuration is distributed over several configuration files. All configuration files are written in XML format.

Figure 3: Example of DataConversion_SA directory. Note the dataconversion.xml in the root configuration. 

The Dataconversion.xml 

In order to run the DCM a configuration file is required containing all its run instructions. In the DCM test-package this is the XML file ‘dataconversion’ (see Figure 3). The dataconversion configuration file contains a list of activities which will be run sequentially. This list of activities contains references to the configured workflow files. 

<?xml version="1.0" encoding="UTF-8"?>
<dataConversion xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews file:/dataconversion.xsd">
	<activities>
		<!-- Run workflows from Config\WorkflowFiles -->
		<workflowActivity>
			<workflowId>Convert_maasaa</workflowId>
		</workflowActivity>
		<workflowActivity>
			<workflowId>Convert_aberdeen</workflowId>
		</workflowActivity>
		<workflowActivity>
			<workflowId>Convert_lixhe</workflowId>
		</workflowActivity>
		<copyActivity id="saveLog">
			<srcFilePath>%REGION_HOME%/log.txt</srcFilePath>
			<destFilePath>%REGION_HOME%/Logs</destFilePath>
		</copyActivity>
		<importStatusActivity id="ImportStatus">
			<importStatusPath>%REGION_HOME%/ImportStatus</importStatusPath>
		</importStatusActivity>
	</activities>
</dataConversion>

Execute the DCM

The DCM can be executed as a script from the command line or as a scheduled task (see Figure 2 for the location of these files).

• On a Windows machine the DCM is run using the BATCH file: DataConversion.bat
• On a Linux machine the DCM is run using the bash script file: DataConversion.sh

Both scripts require the same arguments:

• 1^st argument:   Name of region directory containing all configuration files (DataConversion_SA). If the region directory is not located in the same directory as the script then a path is required.
• 2^nd argument:  Name of the DCM configuration file (here we use the dataconversion.xml). If the DCM configuration file is not located in the root of the region directory then a path is required.
• 3^rd argument:   Path to the directory containing the Delft-FEWS binaries (bin). Can be omitted if the script is executed from within the ‘bin’ directory.

Figure 4: Screenshot of the command prompt when running the DCM on a Windows machine.

Configuration of DCM: conversion task variants

Variant 1: Import, Export.

Variant 2: Import, Transformation, Export.

Variant 3: Import, (dis)Aggregation, Export.