Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Migration of unmigrated content due to installation of a new plugin

Excerpt
hiddentrue

Imports csv type time series data from the Woodleigh System Control Centre in Singapore

Overview

Note

This import is available in DELFT-FEWS versions after 06-12-2007

Imports time series data in csv format from the Woodleigh System Control Centre in Singpore. These file ar in a csv formatSingapore. The first line is a header for each column indicating the location. The filename encodes the parameter. egEg., in the file Aname_RF.txt is parameter is RF (rainfall). If the column after a data columns has a header named "Qf" it is interpreted as a column holding quality flags for the data column. The flags are converted to DELFT-FEWS data flags using the flagconversions mapping.

Configuring the Import

The reader is named WSCCCsv which should be configured in the general section of the import. An example import configuration is shown below:

No Format
<?xml version="1.0" encoding="UTF-8"?>
<timeSeriesImportRun xmlns="http://www.wldelft.nl/fews" 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
xsi:schemaLocation="http://www.wldelft.nl/fews
http://fews.wldelft.nl/schemas/version1.0/timeSeriesImportRun.xsd">
	<import>
		<general>
			<importType>WSCCCsv</importType>
			<folder>$IMPORT_FOLDER$/WSCC</folder>
			<failedFolder>$IMPORT_FAILED_FOLDER$</failedFolder>
			<idMapId>IdImportWSCC</idMapId>
			<flagConversionsId>ImportFlagConversions</flagConversionsId>
			<importTimeZone>
				<timeZoneOffset>+08:00</timeZoneOffset>
			</importTimeZone>
		</general>
		<timeSeriesSet>
			<moduleInstanceId>ImportWSCC</moduleInstanceId>
			<valueType>scalar</valueType>
			<parameterId>H.obs</parameterId>
			<locationSetId>WSCC_Level</locationSetId>
			<timeSeriesType>external historical</timeSeriesType>
			<timeStep unit="minute" multiplier="10"/>
			<readWriteMode>add originals</readWriteMode>
			<synchLevel>1</synchLevel>
			<expiryTime unit="day" multiplier="3650"/>			
		</timeSeriesSet>
	</import>
</timeSeriesImportRun>

...

No Format
<?xml version="1.0" encoding="UTF-8"?>
<idMap version="1.1" xmlns="http://www.wldelft.nl/fews" 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
xsi:schemaLocation="http://www.wldelft.nl/fews 
http://fews.wldelft.nl/schemas/version1.0/idMap.xsd">
	<parameter internal="H.obs" external="LEV"/>
	<parameter internal="Q.obs" external="FLW"/>
	<parameter internal="P.obs" external="RF"/>
	<parameter internal="Q.rated" external="VOL"/>
	<!-- vlume -->
	<location internal="LowerPierce" external="S23-USR-LEV-RES-1"/>
</idMap>

An example flag conversions file for the WSCC data is shown below:

No Format

<?xml version="1.0" encoding="UTF-8"?>
<!--WSCC flag conversion file for Import-->
<flagConversions xmlns="http://www.wldelft.nl/fews" 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
xsi:schemaLocation="http://www.wldelft.nl/fews 
http://fews.wldelft.nl/schemas/version1.0/flagConversions.xsd">
<!-- F -  Telemetry Failed
 N -  Telemetry Normal
 M -   ManuallySet ( value set by the opeartor)
 B -   Blocked ( alarm disabled by the operator )
 C -   Calculation Failure
-->
	<flagConversion>
		<inputFlag>
			<name>Telemetry Failed</name>
			<value>F</value>
		</inputFlag>
		<outputFlag>
			<name>ORIGINAL_MISSING</name>
			<value>9</value>
			<description>Missing value in originally observed series.
Note this is a special form of both ORIGINAL/UNRELIABLE
and ORIGINAL/RELIABLE.</description>
		</outputFlag>
	</flagConversion>
	<flagConversion>
		<inputFlag>
			<name>Telemetry Normal</name>
			<value>N</value>
		</inputFlag>
		<outputFlag>
			<name>ORIGINAL_RELIABLE</name>
			<value>0</value>
			<description>Observed value retrieved from external data source.
Value is valid, marked as original reliable as validation is yet to be done</description>
		</outputFlag>
	</flagConversion>
	<flagConversion>
		<inputFlag>
			<name>ManuallySet ( value set by the opeartor)</name>
			<value>M</value>
		</inputFlag>
		<outputFlag>
			<name>CORRECTED_RELIABLE</name>
			<value>1</value>
			<description>The original value was removed and corrected.
Correction may be through byteerpolation or manual editing</description>
		</outputFlag>
	</flagConversion>
	<flagConversion>
		<inputFlag>
			<name>Blocked ( alarm disabled by the operator )</name>
			<value>B</value>
		</inputFlag>
		<outputFlag>
			<name>ORIGINAL_UNRELIABLE</name>
			<value>6</value>
			<description>Observed value retrieved from external data source.
Value is invalid due to validation limits set. Value is removed.</description>
		</outputFlag>
	</flagConversion>
	<flagConversion>
		<inputFlag>
			<name>Calculation Failure</name>
			<value>C</value>
		</inputFlag>
		<outputFlag>
			<name>ORIGINAL_DOUBTFUL</name>
			<value>3</value>
			<description>Observed value retrieved from external data source.
Value is valid, but marked as suspect due to soft validation limits being exceeded</description>
		</outputFlag>
	</flagConversion>
	<defaultOuputFlag>
		<name>ORIGINAL_RELIABLE</name>
		<value>0</value>
		<description>The data value is the original value retrieved from an
external source and it successfully passes all validation criteria set.</description>
	</defaultOuputFlag>
	<missingValueFlag>
		<name>ORIGINAL_MISSING</name>
		<value>9</value>
	</missingValueFlag>
</flagConversions>

The file format

No Format
Date	Time	TN-1	QF-1	TN-2	QF-2	TN-3	QF-3	TN-N	QF-N
DataType	date	time	float	Char(1)	float	Char(1)	float	Char(1)	float	Char(1)
DataFormat	ddmmyyyy	hh:mm	xxxx.xxxxxx xxxx.xxxxxx	xxxx.xxxxxx		xxxx.xxxxxx

...

  1. Definition of the Data quality flags(QF) as listed below;

    F

    Telemetry Failed

    N

    Telemetry Normal

    M

    ManuallySet ( value set by the opeartor)

    B

    Blocked ( alarm disabled by the operator )

    C

    Calculation Failure

  2. Name of the text file will be 'yyyymmddhhmm_PARAMETER'
Note

The current import ignores the quality flags.

  1. Eg. 200703121500_RF
  2. TN stands for TagName (parameter name) which defined in the SCADA System. Also user can customise the TN which may different from the SCADA System

...

No Format
Date,Time,S23-USR-LEV-RES-1,Qf,S48(51)-UPP-LEV-RES-1,Qf,S46(50)-LPP-LEV-RES-1,Qf,S46(24)-MCP-LEV-RES-1,Qf
06052007,02:10,2.232906,N,8.376068,N,0.271062,F,0.039377,N
06052007,02:20,2.232906,B,8.376068,N,0.271062,F,0.039377,N
06052007,02:30,2.232906,F,8.376068,N,0.271062,F,0.039377,N
06052007,02:40,2.232906,M,8.376068,N,0.271062,F,0.039377,N
06052007,02:50,2.232906,C,8.376068,N,0.271062,F,0.039377,N
06052007,03:00,2.232906,N,8.376068,N,0.271062,F,0.039377,N