...

This is an import that makes use of the custom import mechanism described in Custom import formats. The WQ CSV is an extension of the UmAquo CSV import and makes use of the same text parser, UmAquoCsvTimeSeriesParser.java, and a bin directory containing all dependant library files (UmAquo.zip ).

For more information on the UmAquoCsvTimeSeriesParser.java please see the section UmAquoCsvImport.

...

The WQ CSV import format differs slightly from the UmAquo CSV import format is as described in the document SpecsImporterenCSV.doc. This format is not so strictly defined and it is also lacking the header information required by the UmAquoCsvTimeSeriesParser class. Here follows an example CSV import file:

Code Block

2003925AM_55;OEEUW600;Eeuwselscheloop Grens;201.32;305.22;25-09-2003;13:15:00;Ca;Ca;;5.48;nf;mg/l;OW;Caopg
60181;OBOSS100;Uit de Bosschen Pottevenweg;192.83;389.12;23-08-2004;11:55:00;Calcium opgelost;;;0.5229;;mmol/l;OW;Caopg
2003925AM_55;OEEUW600;Eeuwselscheloop Grens;201.32;305.22;25-09-2003;13:15:00;Aldrin;;;0.24;;mg/l;OW;Aldrin
2003925AM_55;OEEUW600;Eeuwselscheloop Grens;201.32;305.22;25-09-2003;13:15:00;Aldrin;;;1.35;;mg/kg;BS;Aldrin

...

Date and time formats
Format Begindatum: yyyyMMdd
Format Begintijd: HH:mm:ss

Syntax
Commentline prefix: '#'
Column separator: ','
Decimal separator: '.'

Validation

Validation of the values in the UmAquo import files is done against the Aquo schemas . Instead of validating over the web, validation is done locally by validating against the schemas present in the resource file UmAquo_schemas.jar. This archive contains a subset of schemas from the Aquo schemas site. If the Aquo schemas are updated then a new version of the schemas jar must be distributed.

Fews configuration

Because the WQ CSV import format is lacking header information, it is necessary to add this information to the timeseries import configuration file as shown below.In order to activate the UmAquo CSV import as a FEWS import, it is required to setup a TimeSeriesImportRun module configuration file and an accompanying IdMap file and FlagConversion file. Also the bin directory containing all UmAquo resources must be placed in a location that can be accessed by the FEWS system.

Here is an example import module configuration file:

Code Block
<?xml version="1.0" encoding="UTF-8"?>

...


<timeSeriesImportRun xmlns="http://www.wldelft.nl/fews"

...


 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

...


 xsi:schemaLocation="http://www.wldelft.nl/fews [http://fews.wldelft.nl/schemas/version1.0/timeSeriesImportRun.xsd

...

]">

...


<!-- This is an example import configuration file for importing

...

 WQ CSV data from an

...

 import directory&nbsp; -->
    <import>
        <general>
        <!-- Class name of WQ CSV parser is same as UmAquo CSV parser -->

...


            <parserClassName>nl.wldelft.webservice.umaquo.timeseriesparsers.UmAquoCsvTimeSeriesParser</parserClassName>

...



            <!-- Path to directory containing UmAquo libraries and schemas. If omitted then the content of the umaquo-bin

...


can be placed in the FEWS-bin dir \-->

...


            <binDir>%REGION_HOME%/Modules/umaquo-bin</binDir>

...



            <!-- Directory from which CSV files are to be imported -->

...


            <folder>$IMPORT_FOLDER$/

...

WQ/CSV</folder>

...


            <failedFolder>$IMPORT_FAILED_FOLDER$/

...

WQ/CSV</failedFolder>

...


            <backupFolder>$IMPORT_BACKUP_FOLDER$/

...

WQ/CSV</backupFolder>

...


            <idMapId>IdImportWqCsv</idMapId>
            <importTimeZone>
                <timeZoneOffset>+00:00</timeZoneOffset>

...


            </importTimeZone>
        </general>
        <properties>
            <!-- Optional:

...

 decimal character. Default

...

 = '.'. By configuring a value '*' it becomes possible to import both
            ',' and '.' separated values in the same file -->
            <string key="DECIMAL_SEPARATOR" value="*"/>
            <!-- Optional: column separator character. Default = , -->
            <string key="COLUMN_SEPARATOR" value=";"/>

...


            <!-- Optional:

...

 Regular expression for date value. Default = yyyy-MM-dd -->
            <string key="CSV_DATEPATTERN" value="dd-MM-yyyy"/>
            <!-- Optional: select UmAquo schema version 2009 or 2011. Default = 2009 -->
            <int key="SCHEMA_VERSION" value="2011"/>
            <!-- Optional: allow lenient validation. Default = false -->
            <bool key="LENIENT" value="true"/>
            <!-- Required for WQCsv: Provide a header line describing the columns. It is important that the column separator
           matches the value for the COLUMN_SEPARATOR field -->
            <string key="HEADER_LINE" value=";MeetPunt.identificatie;;;Waardebepalingsmethode.code;Begindatum;Begintijd;Typering.code;;Limietsymbool;Numeriekewaarde;Hoedanigheid.code;Eenheid.code;Compartiment.code;"/>
            <!-- Optional: allow import to skip the first number of lines. This can be required when the import file contains a head that does not match the above header -->
            <int key="FIRST_LINE" value="1"/>
        </properties>
        <timeSeriesSet>
            <moduleInstanceId>ImportUmAquo</moduleInstanceId>
            <valueType>scalar</valueType>
            <parameterId>MyPar</parameterId>
            <locationSetId>MyLocSet</locationSetId>
            <timeSeriesType>external historical</timeSeriesType>
            <timeStep unit="nonequidistant"/>
            <readWriteMode>add originals</readWriteMode>
            <synchLevel>1</synchLevel>
        </timeSeriesSet>
    </import>
</timeSeriesImportRun>

Expected CSV header names

The header names that can be provided to describe the content of the WQ-CSV must be selected from the list below.

Monster.identificatie: mapped to sampling id (optional). If MeetPunt.identificatie not specified then the monster identifier is mapped to location id.

MeetPunt.identificatie: mapped to location id (required)

Typering.code: Umaquo timeseries require either Typering.code or Grootheid.code. If defined it will be used as parameter id (either Typering.code or Grootheid.code)
Grootheid.code: Umaquo timeseries require either Typering.code or Grootheid.code. If defined it will be used as parameter id (either Typering.code or Grootheid.code). Can used in combination with Parameter.code or Parameter.omschrijving

Parameter.code: mapped to parameter.id in combination with Grootheid.code. Always used in combination with Grootheid.code. (optional)

Parameter.omschrijving: mapped to parameter.id in combination with Grootheid.code. Always used in combination with Grootheid.code. (optional)
Eenheid.code: mapped to timeseries unit and to qualifier 0 (required)
Hoedanigheid.code: mapped to qualifier 1 (required)
Compartiment.code: mapped to qualifier 2 (required)
Waardebepalingsmethode.code: mapped to qualifier 3 (optional)
Begindatum: mapped to timestamp together with BeginTijd (required)
Begintijd: mapped to timestamp. If not present only Begindatum is used. (optional)
Limietsymbool: mapped to OutOfDetectionRangeFlag value (optional)
Numeriekewaarde: contains numeric value. If empty then Alfanumeriekewaarde field is parsed to numeric value.
Alfanumeriekewaarde: only used if Numeriekewaarde field is empty
Kwaliteitsoordeel.code: mapped to flag value

For more information on how to configure IdMapping and FlagConversions check section UmAquoCsvImport

Internal timeseries location and parameter combinations are mapped to external location, parameter and qualifier sets as follows:

externalLocation values are read from the Meetpunt column
externalParameter values are read from the Parameter column
1st externalQualifier values are read from the Eenheid column
2nd externalQualifier values are read from the Hoedanigheid column
3rd externalQualifier values are read from the Compartiement column

Here is an example flag conversion file:<?xml version="1.0" encoding="UTF-8"?>
<flagConversions xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/flagConversions.xsd^{Image Removed}">
<flagConversion>
    <inputFlag>     <value>0</value></inputFlag>
    <outputFlag>    <value>0</value></outputFlag>
</flagConversion>
    <flagConversion>
      <inputFlag>     <value>3</value></inputFlag>
      <outputFlag>    <value>3</value></outputFlag>
    </flagConversion>
    <flagConversion>
      <inputFlag>     <value>4</value></inputFlag>
      <outputFlag>    <value>0</value></outputFlag>
    </flagConversion>
    <flagConversion>
      <inputFlag>     <value>5</value></inputFlag>
      <outputFlag>    <value>0</value></outputFlag>
    </flagConversion>
<flagConversion>
    <inputFlag>     <value>6</value></inputFlag>
    <outputFlag>    <value>0</value></outputFlag>
</flagConversion>
<flagConversion>
    <inputFlag>     <value>7</value></inputFlag>
    <outputFlag>    <value>0</value></outputFlag>
</flagConversion>
<flagConversion>
    <inputFlag>     <value>10</value></inputFlag>
    <outputFlag>    <value>2</value></outputFlag>
</flagConversion>
<flagConversion>
    <inputFlag>     <value>20</value></inputFlag>
    <outputFlag>    <value>2</value></outputFlag>
</flagConversion>
<flagConversion>
    <inputFlag>     <value>25</value></inputFlag>
    <outputFlag>    <value>2</value></outputFlag>
</flagConversion>
<flagConversion>
    <inputFlag>     <value>30</value></inputFlag>
    <outputFlag>    <value>1</value></outputFlag>
</flagConversion>
<flagConversion>
    <inputFlag>     <value>50</value></inputFlag>
    <outputFlag>    <value>6</value></outputFlag>
</flagConversion>
    <flagConversion>
      <inputFlag>     <value>98</value></inputFlag>
      <outputFlag>    <value>6</value></outputFlag>
    </flagConversion>
    <flagConversion>
      <inputFlag>     <value>99</value></inputFlag>
      <outputFlag>    <value>9</value></outputFlag>
    </flagConversion>
    <defaultOuputFlag><value>0</value></defaultOuputFlag>
   <missingValueFlag><value>9</value></missingValueFlag>
</flagConversions>

Page tree

Versions Compared

Old Version 2

New Version Current

Key

Validation

Fews configuration

Fews configuration

Page tree

Page History

Versions Compared

Old Version 2

New Version Current

Key

Validation

Fews configuration

Fews configuration