...
Below is given an example on how to configure a generalCSV import with flagSource columns.
<? xml version = "1.0" encoding = "UTF-8" ?> < timeSeriesImportRun xmlns = "http://www.wldelft.nl/fews" xmlns:xsi = "http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation = "http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/timeSeriesImportRun.xsd" > < import > < general > < importType >generalCSV</ importType > < folder >../junit_test_output/nl/wldelft/fews/system/plugin/dataImport/TimeSeriesImportTestData/import/generalcsv</ folder > < table > < dateTimeColumn name = "DATE" pattern = "dd-MM-yy HH:mm" /> < locationColumn name = "LOC" /> < unitColumn name = "UNIT" /> < parameterColumn name = "PARAM" /> < flagSourceColumn id = "A" name = "FS_A" /> <!-- column with name fs a will be mapped to the flag source column with identifier A --> < flagSourceColumn id = "B" name = "FS_B" /> < flagSourceColumn id = "C" name = "FS_C" /> < flagSourceColumn id = "D" name = "FS_D" /> < flagSourceColumn id = "E" name = "FS_E" /> < valueColumn name = "VALUE" /> </ table > < logWarningsForUnmappableLocations >true</ logWarningsForUnmappableLocations > < logWarningsForUnmappableParameters >true</ logWarningsForUnmappableParameters > < logWarningsForUnmappableQualifiers >true</ logWarningsForUnmappableQualifiers > < maxLogWarnings >1000</ maxLogWarnings > < missingValue >-999</ missingValue > < importTimeZone > < timeZoneOffset >+01:00</ timeZoneOffset > </ importTimeZone > < dataFeedId >generalCSV</ dataFeedId > </ general > </ import > </ timeSeriesImportRun > |
The CSV file for this example could look like this:
Code Block | ||||
---|---|---|---|---|
| ||||
DATE,LOC,VALUE,UNIT,PARAM,FS_A,FS_B,FS_C,FS_D,FS_E01-01-81 00:00,H-2001,2.3,m,P.m,OK,OK,OK,OK,OK3110040601;Spiegelplas 4;133054;475032;MEA;WP_CAS_ZUY;OW;Van Veenhapper;SPV7030;WP_CAS_ZUY;SBP130;06-04-10 00:00;Tubificidae;2;n;MACEV2010;AANTL_MEA;gemeten |
Note that during import, the mapping goes from 'name' to 'id'. When importing the data, the flagSourceColumnId's should be configured in the flagSourceColumns configuration file.
Importing dates and times with separate year, month, day, hour, minute and second columns (since 2017.01)
There are several options for reading the date and time at which a value should be stored:
- A single dateTimeColumn (with a pattern) can be used for csv files containing a single column with both date and time.
- A separate dateColumn and timeColumn (both with patterns) can be used for csv files containing two columns, one containing the date and the other containing the time.
- A yearColumn, monthColumn and dayColumn can be used (instead of a dateColumn) when a csv file contains separate columns with the year, month and day respectively. Note that when used, all three columns must be specified and present in the csv file, together the columns must result in a valid date.
- An hourColumn, minuteColumn and/or secondColumn can be used (instead of a timeColumn) when a csv file contains separate columns with the hour, minutes and seconds respectively. Note that the hour value must be between 0 and 24 (where 24 is interpreted as 0 the next day), and the minute and second values must be between 0 and 59. The minute and second columns are optional, when an hour column is used without a minute and second column, the minutes and seconds are always set to 0. When an hour column and minute column are used without a second column, the seconds are always set to 0. It is not possible to use an hour and second column withour also specifying a minute column.
Note that the option to specify a time through separate hour, minute and/or second columns, can be combined with a dateColumn, and the option to specify the date through separate year, month and day columns can be combined with a timeColumn. Ultimately, each of the values needed to obtain the full date and time (year, month, day, hour, minute, second) must only be present in one of the configured columns, i.e., you can not use both a dateTimeColumn and a dateColumn (there would be two values for year, month and day), you can not use both an hourColumn and a timeColumn, etc.
A config example using the separate year, month, day, hour, minute and second columns (new in 2017.01):
Code Block | ||
---|---|---|
| ||
<?xml version="1.0" encoding="UTF-8"?> <timeSeriesImportRun xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/timeSeriesImportRun.xsd"> <import> <general> <importType>generalCSV</importType> <folder>../junit_test_output/nl/wldelft/fews/system/plugin/dataImport/TimeSeriesImportTestData/import/generalcsvyearmonthdayhourminutesecond</folder> <table> <dayColumn name="Day"/> <monthColumn name="Month"/> <yearColumn name="Year"/> <hourColumn name="Hour"/> <minuteColumn name="Minute"/> <secondColumn name="Second"/> <locationColumn name="Location"/> <parameterColumn name="Parameter"/> <valueColumn name="Value"/> </table> <missingValue>-999</missingValue> <importTimeZone> <timeZoneName>CET</timeZoneName> </importTimeZone> <columnSeparator> </columnSeparator> <decimalSeparator>.</decimalSeparator> </general> </import> </timeSeriesImportRun> |
The CSV file for this example would look like this:
Code Block |
---|
Year Month Day Second Hour Minute Parameter Value Location 1980 01 1 02 4 59 H.m 1.23 H-2001 1980 02 04 33 2 32 H.m 4.56 H-2001 1981 12 31 22 18 0 H.m 7.89 H-2001 |
Importing with Julian Days
Where Julian Days (1-366) are used to defined the day of year, then following configuration can be used:
Code Block | ||
---|---|---|
| ||
<dateColumn name="Year" pattern="yyyy"/>
<timeColumn name="Day" pattern="DDD"/> |
The year and date columns of the CSV file for this example would look like this:
Code Block |
---|
Year Day
1980 360
1980 361
1981 362 |
Defining column separator and decimal separator
Since 2016.01 it is possible to choose from multiple column separators: comma "," or semi-colon ";" or pipe "|" or tab "	" or space " "
When specifying a column separator it is compulsory to also specify the decimal separator as comma "," or point "."
Code Block | ||||
---|---|---|---|---|
| ||||
<general> <importType>generalCSV</importType> <folder>importfolder</folder> <table> <dateTimeColumn name="DATE" pattern="dd-MM-yy HH:mm"/> <locationColumn name="LOC"/> <unitColumn name="UNIT"/> <parameterColumn name="PARAM"/> <valueColumn name="VALUE"/> </table> <missingValue>-999</missingValue> <importTimeZone> <timeZoneOffset>+01:00</timeZoneOffset> </importTimeZone> <columnSeparator>|</columnSeparator> <decimalSeparator>.</decimalSeparator> </general> |
This enables importing for example the format below:
Code Block | ||||
---|---|---|---|---|
| ||||
DATE|LOC|VALUE|UNIT|PARAM 01-01-81 00:00|H-2001|2.3|m|P.m |
Samples Import
For the import of ecological data based on samples, the general csv import is used extensively because there is a lot of information per timeseries.
...
If the first line contains a comma, the decimal separator is taken to be a period (.), otherwise it is supposed to be a semicolon (;) and the decimal separator is taken to be a comma. This way locale-specific CSV files are supported.
DATE | LOC | VALUE | UNIT | PARAM | FS_A | FS_B | FS_C | FS_D | FS_E |
---|---|---|---|---|---|---|---|---|---|
01-01-81 00:00 | H-2001 | 2.3 | m | P.m | OK | OK | OK | OK | OK |
userColumn
Since 2016.02 there is a column that allows for specifying a user with each data point.
...