Page History

Overview

Imports time series data from files in CSV format with one header line containing a column heades of the time series:

...

• parameterId: determines which parameter this value belongs to, only used when there is no parameterColumn
• locationId: determines which location this value belongs to, only used when there is no locationColumn
• ignoreNumericalParameters: when when set to true, this column will not be read for numerical parameters. Useful when the value is provided in 2 columns, 1 for numerical and 1 for enumeration (i.e. alphanumeric). It prevents values beings overwritten with an undesired value from the wrong column.
• ignoreEnumerationParameters: when set to true, this column will not be read for enumeration parameters. Useful when the value is provided in 2 columns, 1 for numerical and 1 for enumeration. It prevents values beings overwritten with an undesired value from the wrong column.
• requireNumericalParameters: when set to true, the import will fail when this column is filled for non-numerical parameters (enumerations). Useful when the value is provided in 2 columns, 1 for numerical and 1 for enumeration and when the wrong column is filled the import file will be rejected so it can be corrected before the data will be imported.
• requireEnumerationParameters: when set to true, the import will fail when this column is filled for non-enumeration parameters (numerical). Useful when the value is provided in 2 columns, 1 for numerical and 1 for enumeration and when the wrong column is filled the import file will be rejected so it can be corrected before the data will be imported.

Flag column

A flag column <flagColumn name="Y"/> can be configured to store the quality flag with the value.

Importing multiple flag columns (since 2016.02)

Since 2016.02 it is possible to import multiple flag columns. This works only in combination with multiple value columns. For each flag column a parameter id and/or location id can be specified, this should be done in the same way as the value column so the flag will match the correct value. 

...

NoClimato;Date_Heure;TempMin_degC;Code_TMin;TempMax_degC;Code_TMax;TempMoy_degC;Code_TMoy;PrecipTotale_mm;Code_PTotale;PrecipLiquide_mm;Code_PLiquide;PrecipSolide_mm;Code_PSolide
H-2001;2017-02-03 06:00:00;-27.09;0;-26.66;3;-26.91;1;0;1;0;1;0;1
H-2001;2017-02-03 07:00:00;-14.01;6;-13.27;6;-13.69;1;-999;-999;-999;-999;-999;-999
H-2001;2017-02-03 08:00:00;-15.89;3;-15.64;0;-15.76;1;0;1;0;1;0;1
H-2001;2017-02-03 09:00:00;-17.69;0;-16.14;0;-16.94;1;0;1;0;1;0;1

 

Comment Column

A column can be configured to import a comment for a value: <commentColumn name="columnX"/>.

Importing comments only (since 2014.02)

For all imports a value column is required with the exception of importing comments. If the importer is configured without a valueColumn and a commentColumn was specified, the values will be set to missing values and the comments will be imported.

...

A way to work around this issue is to only manually import series for a period with less than 254 unique comments at once. In this way, for each period all the comments are read, and stored in the datastore (as can be checked through the database viewer). When viewing the entire series in the TimeSeriesDisplay, the table will still show "too many comments" for any comment past the 254 unique comments, even if the comment is stored correctly in the datastore. To show the comment as stored in the datastore for a specific value use the shortcut Ctrl + H or right click --> show history on a value in the table. This will still show the stored comment in the datastore, even if the table display shows "too many comments". 

...

Date time formatting

For the date time formatting please follow the Java conventions.

For example to parse a date 31-12-2023 01:10:23 you would use the following pattern:

dd-MM-yyyy HH:mm:ss

In case the datetime string contains a T or Z or any part that shouldn't be interpreted use single quotes (') to mark this.

https://docs.oracle.com/javase/8/docs/api/java/text/SimpleDateFormat.html says: Date and time formats are specified by date and time pattern strings. Within date and time pattern strings, unquoted letters from 'A' to 'Z' and from 'a' to 'z' are interpreted as pattern letters representing the components of a date or time string. Text can be quoted using single quotes (') to avoid interpretation. "''" represents a single quote. All other characters are not interpreted; they're simply copied into the output string during formatting or matched against the input string during parsing.

Anchor
flagSource flagSource

Importing flagSourceColumns (since 2015.02)

Below is given an example on how to configure a generalCSV import with flagSource columns.

Below is given an example on how to configure a generalCSV import with flagSource columns.

...

DATE,LOC,VALUE,UNIT,PARAM,FS_A,FS_B,FS_C,FS_D,FS_E01-01-81 00:00,H-2001,2.3,m,P.m,OK,OK,OK,OK,OK3110040601;Spiegelplas 4;133054;475032;MEA;WP_CAS_ZUY;OW;Van Veenhapper;SPV7030;WP_CAS_ZUY;SBP130;06-04-10 00:00;Tubificidae;2;n;MACEV2010;AANTL_MEA;gemeten

<?xml version="1.0" encoding="UTF-8"?>
<timeSeriesImportRun xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/timeSeriesImportRun.xsd">
   <import>
      <general>
         <importType>generalCSV</importType>
         <folder>../junit_test_output/nl/wldelft/fews/system/plugin/dataImport/TimeSeriesImportTestData/import/generalcsvyearmonthdayhourminutesecond</folder>
         <table>
            <dayColumn name="Day"/>
            <monthColumn name="Month"/>
            <yearColumn name="Year"/>
            <hourColumn name="Hour"/>
            <minuteColumn name="Minute"/>
            <secondColumn name="Second"/>
            <locationColumn name="Location"/>
            <parameterColumn name="Parameter"/>
            <valueColumn name="Value"/>
         </table>
         <missingValue>-999</missingValue>
         <importTimeZone>
       <parameterColumn name="Parameter"/>
    <timeZoneName>CET</timeZoneName>
        <valueColumn name="Value"/></importTimeZone>
         <columnSeparator> </table>columnSeparator>
         <missingValue>-999</missingValue><decimalSeparator>.</decimalSeparator>
      </general>
   <importTimeZone>
            <timeZoneName>CET</timeZoneName>
         </importTimeZone>
         <columnSeparator> </columnSeparator>
         <decimalSeparator>.</decimalSeparator>
      </general>
   </import>
</timeSeriesImportRun>

</import>
</timeSeriesImportRun>

Year Month Day Second Hour Minute Parameter Value Location
1980 01 1 02 4 59 H.m 1.23 H-2001
1980 02 04 33 2 32 H.m 4.56 H-2001
1981 12 31 22 18 0 H.m 7.89 H-2001

<dateColumn name="Year" pattern="yyyy"/>
<timeColumn name="Day" pattern="DDD"/> 

Year Month Day Second Hour Minute Parameter Value Location
1980 01 1 02 4 59 H.m 1.23 H-2001
1980 02 04 33 2 32 H.m 4.56 H-2001
1981 12 31 22 18 0 H.m 7.89 H-2001

Year Day
1980 360 
1980 361 
1981 362

<general>
   <importType>generalCSV</importType>
   <folder>importfolder</folder>
   <table>
      <dateTimeColumn name="DATE" pattern="dd-MM-yy HH:mm"/>
      <locationColumn name="LOC"/>
      <unitColumn name="UNIT"/>
      <parameterColumn name="PARAM"/>
      <valueColumn name="VALUE"/>
   </table>
   <missingValue>-999</missingValue>
   <importTimeZone>
      <timeZoneOffset>+01:00</timeZoneOffset>
   </importTimeZone>
   <columnSeparator>|</columnSeparator>
   <decimalSeparator>.</decimalSeparator>
</general>

DATE|LOC|VALUE|UNIT|PARAM
01-01-81 00:00|H-2001|2.3|m|P.m

Samples Import

For the import of ecological data based on samples, the general csv import is used extensively because there is a lot of information per timeseries.

...

When re-importing samples make sure the location and time are exactly the same, since it is used as an internal id. Different samples with the same location time combination are handled internally by applying a behind the scene time shift in the storage, when samples are read this time shift is automatically corrected so the data will not suffer from this shift.

rejectCompleteSampleOnUnmappableId

When true, if part of an imported sample cannot be mapped to a time serie in FEWS the whole sample will be rejected, when false part of a sample can be imported.

rejectCompleteSampleOnDuplicateValues

When true, if for the same time step within a time series multiple values are imported the sample will be rejected and a warning logged

Details of the import format

If the first line contains a comma, the decimal separator is taken to be a period (.), otherwise it is supposed to be a semicolon (;) and the decimal separator is taken to be a comma. This way locale-specific CSV files are supported. 

userColumn

Since 2016.02 there is a column that allows for specifying a user with each data point. 

...