Introduction

The Data Conversion Module (DCM) is a generic application with which timeseries data can be converted. The DCM is a ‘stripped’ version of Delft FEWS and therefore supports the same import and export file formats. It is also possible to perform some transformations that are available in the transformation module of Delft-FEWS.

...

Figure 1: Illustration of interaction between Data Interface Module (DiM) and Data Conversion Module (DCM).

Configuration

File structure of DCM after installation

Figure 2: Overview of directory structure after installation of the DCM.

...

Figure 3: Example of DataConversion_SA directory. Note the dataconversion.xml in the root configuration. 

DCM configuration file

In Figure 3 the dataconversion file can be distinguished, this file is the DataConversion configuration file. The DCM configuration file contains all required instructions in order to run the DCM. This configuration file is an XML file that is structured according to the schema file ‘dataconversion.xsd’ which can be found in the release package.

...

The DataConversion configuration file contains a list of activities which will be run sequentially. There is no limit to the number and order of the activities. It is possible to choose from the following list of activities:

PurgeActivity:

With this activity it is possible to delete both files and directories. This activity supports the wildcards ‘?’ and ‘*’.

...

Figure 5 DataConversion purge activity

CopyActivity:

With this activity it is possible to copy files from one location to another. This activity supports the wildcards ‘?’ and ‘*’.

...

Figure 6 DataConversion copy activity

WorkflowActivity:

With this activity FEWS workflows can be run.

...

Figure 7 DataConversion workflow activity

MoveActivity: 

 With this activity it is possible to move files from one location to another. This activity supports the wildcards ‘?’ and ‘*’.

...

Figure 8 DataConversion move activity

ImportStatusActivity:

With this activity the content of the ImportStatus table can be exported to CSV file. The ImportStatus is information produced by the Delft FEWS import modules, describing basic information about the last import run.

...

Figure 9 DataConversion import status

RunInLoopActivity: 

With this activity one or more of the above can be run repeatedly. This activity will keep looping over the sub activities until one of the sub activities indicates that it has completed all its loops or until the ‘timeOut’ has been exceeded.

...

Figure 10 DataConversion run in loop activity

Configuration of DCM: conversion task variants

The DCM makes use of the file based Delft-FEWS configuration. Hence, all modules that are available in Delft-FEWS can also be used for the DCM. The configuration is located in the Config directory of the 'region_home' directory (see Figure 2). In this section some general remarks about the Delft-FEWs configuration in relation to the DCM can be found. We do this by defining three 'conversion tasks' with increasing complexity:

• Variant 1: workflow with import and export module.
• Variant 2: workflow with import, transformation (only on new data) and export module.
• Variant 3: workflow with import, transformation (on new data and on data of previous DCM runs) and export module . 

Variant 1: Import, Export.

This is the simplest workflow variant. The workflow only consists out of an import and an export module. This variant can be used when the original import files already have the correct timestep and when location specific transformations are not needed. In this case the DCM is mainly used to convert the file format of the import files.  In the Figure below a schematic illustration of this workflow variant is given. 

...

<timeSeriesExportRun xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews
http://fews.wldelft.nl/schemas/version1.0/timeSeriesExportRun.xsd">
	<export>
		<general>
			<exportType>NETCDF-CF_TIMESERIES</exportType>
			<folder>$EXPORT_FOLDER_ROOT$/LMW_maasaa</folder>
			<exportFileName>
				<name>_LMWmaasaa.nc</name>
				<prefix>
					<timeZeroFormattingString>yyyy-MM-dd'T'HHmmss</timeZeroFormattingString>
				</prefix>
			</exportFileName>
			<unitConversionsId>ExportUnitConversions</unitConversionsId>
			<exportMissingValueString>-9999.0</exportMissingValueString>
			<exportTimeZone>
				<timeZoneOffset>+01:00</timeZoneOffset>
			</exportTimeZone>
		</general>
		<properties>
			<bool value="false" key="includeFlags"/>
			<bool value="false" key="includeComments"/>
		</properties>
		<metadata>
			<title>Export of LMW_maasaa datafeed</title>
			<institution>Deltares</institution>
		</metadata>
	</export>
</timeSeriesExportRun>

Variant 2: Import, Transformation, Export.

The second workflow variant is similar to the first workflow variant. Here, the timestep of the original timeseries is also correct but additional transformations are required. A transformation module is added to the workflow to allow for location specific conversions (e.g. correction of water level) and/or the derivation of new timeseries. For the transformation module it remains necessary to explicitly configure TimeSeriesSet information. However it is not necessary to pre configure parameters and locations in the RegionConfig directory. As long as the required timeseries have been imported during a prior import run they will be available for the transformation. 

...

<?xml version="1.0" encoding="UTF-8"?>
<timeSeriesExportRun xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews
http://fews.wldelft.nl/schemas/version1.0/timeSeriesExportRun.xsd">
	<export>
		<general>
			<exportType>PI</exportType>
			<folder>$EXPORT_FOLDER_ROOT$/LMW_lixhe</folder>
			<exportFileName>
				<name>_LMWlixhe.txt</name>
				<prefix>
					<timeZeroFormattingString>yyyy-MM-dd'T'HHmmss</timeZeroFormattingString>
				</prefix>
			</exportFileName>
			<unitConversionsId>ExportUnitConversions</unitConversionsId>
			<exportMissingValueString>-9999.0</exportMissingValueString>
			<exportTimeZone>
				<timeZoneOffset>+01:00</timeZoneOffset>
			</exportTimeZone>
		</general>
		<properties>
			<bool value="false" key="includeFlags"/>
			<bool value="false" key="includeComments"/>
		</properties>
		<metadata>
			<title>Export of LMW_lixhe datafeed</title>
			<institution>Deltares</institution>
		</metadata>
		<timeSeriesSet>
			<moduleInstanceId>LMWlixhe_interpolation</moduleInstanceId>
			<valueType>scalar</valueType>
			<parameterId>C10P</parameterId>
			<locationSetId>LMWlixhe</locationSetId>
			<timeSeriesType>temporary</timeSeriesType>
			<timeStep unit="minute" multiplier="10"/>
			<relativeViewPeriod unit="hour" start="-3" end="0"/>
			<readWriteMode>read complete forecast</readWriteMode>
		</timeSeriesSet>
	</export>
</timeSeriesExportRun>

Variant 3: Import, (dis)Aggregation, Export.

The third workflow variant is used when the original import data does not have the appropriate timestep. In Delft-FEWS it is relatively simple to convert the original import data to the appropriate timestep by using a (dis-)aggregation function.

...

Figure 13 This construction is needed whenever functionalities are used that need data from previous DCM runs. The extra export/import routine ensures that the user has data from previous DCM runs.  

Run the DCM 

The DCM can be executed as a script from the command line or as a scheduled task (see Figure 2 for the location of these files).

...