Note: This document does NOT describe all the possible options of the Import module; this information can be found on this wiki page 03 Import Module.

{panel title=Draft Documentation}

Please note that this is a draft. This document may contain erorrs and is incomplete. Once finished this document will become part of the ormal Delft-Fews documentation.
{panel}

h1. Summary

Importing data in DELFT-FEWS is different from running a wizard in e.g. a much used office software package. To be able to import data a number of xml configuration files need to be adjusted or created. In short, this entails the following steps:
# The file-type you want to import should be readable by [one of the imports available in Delft-Fews|Which file types can Delft-Fews Import?]
# You must configure an instance of this import module
# You must add the moduleInstance to an workflow (usually the ImportExternal Workflow)
# In case the locationId's and parameterId's in your Delft-Fews configuration are not identical to those in the files you need to import a IdMap file should be set up and referred to in the import module
# You will probably need to adjust one or more locationsets, the filters.xml file and the DisplayGroups.xml file if you need to display or process the newly imported data types.
{info}Please note that Delft-Fews _DELETES_ all files after importing them. As such, you should _ALWAYS_ copy files from another directory to the location from which Delft-Fews imports files. If you do not do this you might loose you files.
{info}

h1. Setting up the import module

Steps:
# make the Configuration file for the import module and store in the ModuleConfigFiles directory. In this example it should be named *Import 1.00 default.xml*
# register this new module in the *ModuleInstanceDescriptors 1.00 default.xml* file in the RegionConfigFiles directory
# Add the new module at a workflow in order to run it.

h2. Step 1: Configure the import module (create the [Import 1.00 default.xml|^Import 1.00 Default.xml] configuration File)

The first step is to configure an instance of the import module for the type of data you need to import. In this example we will import data in [PI|PI - Delft-Fews Published Interface Format] XML format for 4 locations. The [file we want to import|^MyNewImportData.xml] is shown below:
{noformat}
<?xml version="1.0" encoding="UTF-8"?>
<TimeSeries xmlns="http://www.wldelft.nl/fews/PI"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.wldelft.nl/fews/PI
http://fews.wldelft.nl/schemas/version1.0/pi-schemas/pi_timeseries.xsd"
version="1.2">
	<series>
		<header>
			<type>instantaneous</type>
			<locationId>locA<<locationId>LocA</locationId>
			<parameterId>WaterLevel</parameterId>
			<timeStep unit="second" multiplier="3600"/>
			<startDate date="2006-08-23" time="17:00:00"/>
			<endDate date="2006-08-24" time="00:00:00"/>
			<missVal>-8888.0</missVal>
			<longName>locA<<longName>LocA</longName>
			<units>m</units>
		</header>
		<event date="2006-08-23" time="17:00:00" value="8.66"/>
		<event date="2006-08-23" time="18:00:00" value="9.66"/>
		<event date="2006-08-23" time="19:00:00" value="8.66"/>
		<event date="2006-08-23" time="20:00:00" value="-8888.0"/>
		<event date="2006-08-23" time="21:00:00" value="3"/>
		<event date="2006-08-23" time="22:00:00" value="3"/>
		<event date="2006-08-23" time="23:00:00" value="3"/>
		<event date="2006-08-24" time="00:00:00" value="-8888.0"/>
	</series>
	<series>
		<header>
			<type>instantaneous</type>
			<locationId>locB<<locationId>LocB</locationId>
			<parameterId>WaterLevel</parameterId>
			<timeStep unit="second" multiplier="3600"/>
			<startDate date="2006-08-23" time="13:00:00"/>
			<endDate date="2006-08-24" time="00:00:00"/>
			<missVal>-999.0</missVal>
			<longName>locB<<longName>LocB</longName>
			<units>m</units>
		</header>
		<event date="2006-08-23" time="13:00:00" value="0.63"/>
		<event date="2006-08-23" time="14:00:00" value="0.61"/>
		<event date="2006-08-23" time="15:00:00" value="0.59"/>
		<event date="2006-08-23" time="16:00:00" value="0.59"/>
		<event date="2006-08-23" time="17:00:00" value="0.59"/>
		<event date="2006-08-23" time="18:00:00" value="0.59"/>
		<event date="2006-08-23" time="19:00:00" value="0.60"/>
		<event date="2006-08-23" time="20:00:00" value="0.57"/>
		<event date="2006-08-23" time="21:00:00" value="0.59"/>
		<event date="2006-08-23" time="22:00:00" value="0.59"/>
		<event date="2006-08-23" time="23:00:00" value="-999.0"/>
		<event date="2006-08-24" time="00:00:00" value="-999.0"/>
	</series>
	<series>
		<header>
			<type>instantaneous</type>
			<locationId>LocC</locationId>
			<parameterId>WaterLevel</parameterId>
			<timeStep unit="second" multiplier="3600"/>
			<startDate date="2006-08-23" time="13:00:00"/>
			<endDate date="2006-08-24" time="00:00:00"/>
			<missVal>-999.0</missVal>
			<longName>LocC</longName>
			<units>m</units>
		</header>
		<event date="2006-08-23" time="13:00:00" value="-1.42"/>
		<event date="2006-08-23" time="14:00:00" value="-1.42"/>
		<event date="2006-08-23" time="15:00:00" value="-1.44"/>
		<event date="2006-08-23" time="16:00:00" value="-1.46"/>
		<event date="2006-08-23" time="17:00:00" value="-1.46"/>
		<event date="2006-08-23" time="18:00:00" value="-1.46"/>
		<event date="2006-08-23" time="19:00:00" value="-1.48"/>
		<event date="2006-08-23" time="20:00:00" value="-1.48"/>
		<event date="2006-08-23" time="21:00:00" value="-1.48"/>
		<event date="2006-08-23" time="22:00:00" value="-1.49"/>
		<event date="2006-08-23" time="23:00:00" value="-999.0"/>
		<event date="2006-08-24" time="00:00:00" value="-999.0"/>
	</series>
	<series>
		<header>
			<type>instantaneous</type>
			<locationId>LocD</locationId>
			<parameterId>WaterLevel</parameterId>
			<timeStep unit="second" multiplier="3600"/>
			<startDate date="2006-08-23" time="13:00:00"/>
			<endDate date="2006-08-24" time="00:00:00"/>
			<missVal>-999.0</missVal>
			<longName>LocD</longName>
			<units>m</units>
		</header>
		<event date="2006-08-23" time="13:00:00" value="-1.42"/>
		<event date="2006-08-23" time="14:00:00" value="-1.42"/>
		<event date="2006-08-23" time="15:00:00" value="-1.44"/>
		<event date="2006-08-23" time="16:00:00" value="-1.46"/>
		<event date="2006-08-23" time="17:00:00" value="-1.46"/>
		<event date="2006-08-23" time="18:00:00" value="-1.46"/>
		<event date="2006-08-23" time="19:00:00" value="-1.48"/>
		<event date="2006-08-23" time="20:00:00" value="-1.48"/>
		<event date="2006-08-23" time="21:00:00" value="-1.48"/>
		<event date="2006-08-23" time="22:00:00" value="-1.49"/>
		<event date="2006-08-23" time="23:00:00" value="-999.0"/>
		<event date="2006-08-24" time="00:00:00" value="-999.0"/>
	</series>
</TimeSeries>
{noformat}

Please see the

Delft-Fews

configuration

guide

for

detailed

information

on

how

to

configure

the

Import

modules.

In

addition,

the

.xsd schema may be consulted.

 schema file will provide documentation.
{info}
{noformat}
<?xml version="1.0" encoding="UTF-8"?>
<timeSeriesImportRun xmlns="http://www.wldelft.nl/fews"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.wldelft.nl/fews
http://fews.wldelft.nl/schemas/version1.0/timeSeriesImportRun.xsd">
	<import>
		<general>
			<importType>PI</importType> <!-- Indicated we will read PI Files -->
			<folder>c:/Import/PI</folder> <!-- read the files from here -->
                                                      <!-- Files WILL BE DELETED after import -->
			<failedFolder>c:/Backup/PI</failedFolder>
                                <!-- If specified the imported files will be copied to this location -->
			<idMapId>IdImport</idMapId> <!-- The IdMap for this import -->
			<importTimeZone>
				<timeZoneOffset>+00:00</timeZoneOffset>
			</importTimeZone>
		</general>
		<timeSeriesSet>
			<moduleInstanceId>Import</moduleInstanceId> <!-- The name of this moduleinstance -->
			<valueType>scalar</valueType>
			<parameterId>H.obs</parameterId> <!-- The parameter as it will be stored in Delft-Fews,
                                                          different from the parameter in the XML. Therefore,
                                                          IdMapping has to be set up-->
			<locationSetId>LevelGauges</locationSetId>
			<timeSeriesType>external historical</timeSeriesType>
			<timeStep unit="hour" multiplier="1"/>
			<readWriteMode>add originals</readWriteMode>
			<synchLevel>1</synchLevel>
		</timeSeriesSet>
	</import>
</timeSeriesImportRun>
{noformat}

In this example only one moduleInstance is present. Usually you will need to add your module to an existing list.

 file we are importing the id mapping will need to map them.
- timeStep - the timestep defined here should map the timestep in the XML files to be imported.

h2. Step 2: Register the new module - Add to the [ModuleInstanceDescriptors 1.00 default.xml|^ModuleInstanceDescriptors 1.00 default.xml] file

_All_ moduleInstances must be registered in Delft-Fews. This is done by adding en entry in the moduleInstancseDescriptors file in the RegionConfigDiles section of the configuration. To register this module we will need to a the following to the [moduleInstancedescriptors|^ModuleInstanceDescriptors 1.00 default.xml]:
{info}
In this example only one moduleInstance is present. Usually you will need to ad your module to an existing list.
{info}
{noformat}
<?xml version="1.0" encoding="UTF-8"?>
<moduleInstanceDescriptors xmlns="http://www.wldelft.nl/fews"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews
http://fews.wldelft.nl/schemas/version1.0/moduleInstanceDescriptors.xsd"
version="1.0">
	<!--Import Module Instances-->
	<moduleInstanceDescriptor id="Import">
		<description>Import Our new data</description>
		<moduleId>TimeSeriesImportRun</moduleId>
	</moduleInstanceDescriptor>
</moduleInstanceDescriptors>
{noformat}

h2. Step 3: Add the module to a workflow ([ImportExternal 

|^ImportExternal 1.00 Default.xml])

In order to use the module (to be able to run it from the manual forecast dialog) you will need to list the module as an activity in a workflow. Assuming this workflow is already registered (see the Delft-Fews configuration manual on how to register workflow) the only thing to do is to add in to the workflow. In the following example the new module is the only activity in the workflow:
{noformat}
<?xml version="1.0" encoding="UTF-8"?>
<workflow xmlns="http://www.wldelft.nl/fews"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.wldelft.nl/fews
[http://fews.wldelft.nl/schemas/version1.0/workflow.xsd]" version="1.1">
	<activity>
		<runIndependent>true</runIndependent>
		<moduleInstanceId>Import</moduleInstanceId>
	</activity>
</workflow>
{noformat}

h1. Set up Id Mapping


h2. Step 4: Make the idmap file

&nbsp;The format of the IdMap files is described in the DELFT-FEWS configuration guid and also in the .xsd scheme. All idMaps are stores in the IdMapFiles directory.&nbsp; The Idmap needed for this example ([IdImport 1.00 Default.xml|^IdImport 1.00 default.xml]) should match the parameter WaterLevel to H.obs and should also map the four locations. The files is shown below:

{noformat}
<?xml version="1.0" encoding="UTF-8"?>
<idMap xmlns

<?xml version="1.0" encoding="UTF-8"?>
<locationSets version="1.1" xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews"
xmlns:xsi="[http://www.w3.org/2001/XMLSchema-instance" 
xsi:schemaLocation="http://www.wldelft.nl/fews 
http://fews.fews.wldelft.nl/schemas/version1.0/idMaplocationSets.xsd" version="1.1"]">
	<parameter<locationSet internalid="H.obsLevelGauges" externalname="WaterLevelLevelGauges"/>
		<locationId>one</locationId>
		<locationId>two</locationId>
		<locationId>three</locationId>
		<locationId>four</locationId>
	</locationSet>
</locationSets>

<location internal="one" external="LocA"/>
	<location internal="two" external="LocB"/>
	<location internal="three" external="LocC"/>
	<location internal="four" external="LocD"/>
</idMap>
{noformat}

h2. Step 5: register the idmap file

Similar to ModuleInstances IdMap should also be registered. In this case we need to register the IdImport file in the file [IdMapDescriptors|^IdMapDescriptors 1.0 default.xml] in the RegionConfigFiles directory:
{noformat}
<?xml version="1.0" encoding="UTF-8"?>
<idMapDescriptors<idMap xmlns="http://www.wldelft.nl/fews"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://www.wldelft.nl/fews 
http://fews.wldelft.nl/schemas/version1.0/idMapDescriptorsidMap.xsd" version="1.01">
	<idMapDescriptor id<parameter internal="H.obs" external="IdImportWaterLevel"/>
</idMapDescriptors>
{noformat}


h1. Setting up a (temporary) display to view the imported data

Delft-Fews only shows data is you specifically tel it to show data. The file [DisplayGroups.xml|^DisplayGroups 1.00 default.xml] (located in the SystemConfigFiles directory) defines pre-defined graphs. In the example below five graphs 	<location internal="one" external="LocA"/>
	<location internal="two" external="LocB"/>
	<location internal="three" external="LocC"/>
	<location internal="four" external="LocD"/>
</idMap>

have been defined to view the newly imported data:

{noformat}
<?xml version="1.0" encoding="UTF-8"?>
<displayGroups version="1.0" 
xmlns="http://www.wldelft.nl/fews" 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
xsi:schemaLocation="http://www.wldelft.nl/fews 
http://fews.wldelft.nl/schemas/version1.0/displayGroups.xsd">
	<displayGroup name="Test">
		<display name="Gauge One">
			<relativeViewPeriod unit="hour" start="-12" end="36"/>
			<subplot>
				<timeSeriesSet>
					<moduleInstanceId>Import</moduleInstanceId>
					<valueType>scalar</valueType>
					<parameterId>H.obs</parameterId>
					<locationId>one</locationId>
					<timeSeriesType>external historical</timeSeriesType>
					<timeStep unit="hour" multiplier="1"/>
					<relativeViewPeriod unit="hour" end="48" start="-48"/>
					<readWriteMode>add originals</readWriteMode>
				</timeSeriesSet>
			</subplot>
		</display>
		<display name="Gauge Two">
			<relativeViewPeriod unit="hour" start="-12" end="36"/>
			<subplot>
				<timeSeriesSet>
					<moduleInstanceId>Import</moduleInstanceId>
					<valueType>scalar</valueType>
					<parameterId>H.obs</parameterId>
					<locationId>two</locationId>
					<timeSeriesType>external historical</timeSeriesType>
					<timeStep unit="hour" multiplier="1"/>
					<relativeViewPeriod unit="hour" end="48" start="-48"/>
					<readWriteMode>add originals</readWriteMode>
				</timeSeriesSet>
			</subplot>
		</display>
		<display name="Gauge Three">
			<relativeViewPeriod unit="hour" start="-12" end="36"/>
			<subplot>
				<timeSeriesSet>
					<moduleInstanceId>Import</moduleInstanceId>
					<valueType>scalar</valueType>
					<parameterId>H.obs</parameterId>
					<locationId>three</locationId>
					<timeSeriesType>external historical</timeSeriesType>
					<timeStep unit="hour" multiplier="1"/>
					<relativeViewPeriod unit="hour" end="48" start="-48"/>
					<readWriteMode>add originals</readWriteMode>
				</timeSeriesSet>
			</subplot>
		</display>
		<display name="Gauge Four">
			<relativeViewPeriod unit="hour" start="-12" end="36"/>
			<subplot>
				<timeSeriesSet>
					<moduleInstanceId>Import</moduleInstanceId>
					<valueType>scalar</valueType>
					<parameterId>H.obs</parameterId>
					<locationId>four</locationId>
					<timeSeriesType>external historical</timeSeriesType>
					<timeStep unit="hour" multiplier="1"/>
					<relativeViewPeriod unit="hour" end="48" start="-48"/>
					<readWriteMode>add originals</readWriteMode>
				</timeSeriesSet>
			</subplot>
		</display>
		<display name="All together">
			<relativeViewPeriod unit="hour" start="-12" end="36"/>
			<subplot>
				<timeSeriesSet>
					<moduleInstanceId>Import</moduleInstanceId>
					<valueType>scalar</valueType>
					<parameterId>H.obs</parameterId>
					<locationSetId>LevelGauges</locationSetId>
					<timeSeriesType>external historical</timeSeriesType>
					<timeStep unit="hour" multiplier="1"/>
					<relativeViewPeriod unit="hour" end="48" start="-48"/>
					<readWriteMode>add originals</readWriteMode>
				</timeSeriesSet>
			</subplot>
		</display>
	</displayGroup>
</displayGroups>
{noformat}


h1. Test the import

To test the import run the ImportExternal Workflow. Check to log panel for error messages. If you want to view the data please make sure to first set the system time to a time/data you have data for. In this example August 23 2006.


h1. Add to the appropriate filters(s)


To be able to use the main map display and the filter to access the data you should set up en entry in the Filters.xml file in the RegionConfigFiles directory. See the example below:

{noformat}
{noformat}

<?xml version="1.0" encoding="UTF-8"?>
<filters version="1.1"
xmlns="http://www.wldelft.nl/fews"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.wldelft.nl/fews
http://fews.wldelft.nl/schemas/version1.0/filters.xsd">
	<filter name="Gauges" id="Gauges">
		<timeSeriesSet>
			<moduleInstanceId>Import</moduleInstanceId>
			<valueType>scalar</valueType>
			<parameterId>H.obs</parameterId>
			<locationSetId>LevelGauges</locationSetId>
			<timeSeriesType>external historical</timeSeriesType>
			<timeStep unit="hour" multiplier="1"/>
			<relativeViewPeriod unit="hour" end="48" start="-48"/>
			<readWriteMode>add originals</readWriteMode>
		</timeSeriesSet>
	</filter>
</filters>