Overview
Imports time series data from files in CSV format with one header line containing a column heades of the time series:
- The first line contains the column names (fields) in the csv file, the line is used to determine the field separator and to determine the names of the data columns
- All other lines contain the date-time as field and the values for each time series.
- Values between -1000.0 and -999.0 (inclusive) are regarded as missing values.
The CSV files can be supplied in a ZIP file.
Import type
The import type is generalCSV. There is no particular file extension required.
Example
Here is a simple example:
Time,Waterstand,Pomp-1 Born 04-05-2011 03:24,0.000000,-0.450000 04-05-2011 03:44,0.000000,-0.450000 04-05-2011 03:54,0.000000,-0.440000 .....
for configuration of the table layout see
Table Layout
<?xml version="1.0" encoding="UTF-8"?> <timeSeriesImportRun xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/timeSeriesImportRun.xsd"> <import> <general> <importType>generalCSV</importType> <folder>$IMPORT_FOLDER$/OBS</folder> <failedFolder>$IMPORT_FAILED_FOLDER$</failedFolder> <backupFolder>$IMPORT_BACKUP_FOLDER$/OBS</backupFolder> <table> <dateTimeColumn name="Time" pattern="dd-MM-yyyy HH:mm"/> <valueColumn unit="m" locationId="Bosscheveld" parameterId="H.meting" name="Waterstand"/> <valueColumn unit="min" locationId="Bosscheveld" parameterId="DT.meting" name="Pomp-1 Born"/> </table> <idMapId>IdImportOBS</idMapId> <unitConversionsId>ImportUnitConversions</unitConversionsId> <importTimeZone> <timeZoneOffset>+00:00</timeZoneOffset> </importTimeZone> </general> </import> </timeSeriesImportRun>
Another example, skipFirstLinesCount
Example to read from CSV files where unfortunately the first contains a key instead of the column headers.
The CSV files look like:
[DATA] Tagname,TimeStamp,Value,DataQuality WDD.MEM_BER0001_01_LT01_MW,2010-04-01 12:21:00,-0.000,GOOD WDD.MEM_BER0001_01_LT01_MW,2011-01-12 10:34:05,-0.001,GOOD WDD.MEM_BER0001_01_LT01_MW,2011-01-12 10:35:00,-0.011,BAD WDD.MEM_BER0001_01_LT01_MW,2011-01-12 10:36:00,-0.003,GOOD WDD.MEM_BER0001_01_LT01_MW,2011-01-12 10:37:00,-0.000,GOOD WDD.MEM_BER0001_01_LT01_MW,2011-01-12 10:38:00,-0.000,GOOD WDD.MEM_BER0001_01_LT01_MW,2011-01-12 10:39:00,-0.000,GOOD
In this example the first line should be skipped, so skipFirstLinesCount = 1
<general> <importType>generalCSV</importType> <folder>$IMPORT_FOLDER_KETEN$</folder> <fileNamePatternFilter>*.csv</fileNamePatternFilter> <failedFolder>$IMPORT_FAILED_FOLDER_KETEN$</failedFolder> <backupFolder>$IMPORT_BACKUP_FOLDER_KETEN$</backupFolder> <table> <dateTimeColumn name="TimeStamp" pattern="yyyy-MM-dd HH:mm:ss"/> <locationColumn name="Tagname"/> <flagColumn name="DataQuality"/> <valueColumn name="Value" unit="SI"/> </table> <idMapId>IdKETEN</idMapId> <flagConversionsId>ImportKETENFlagConversions</flagConversionsId> <importTimeZone> <timeZoneOffset>+01:00</timeZoneOffset> </importTimeZone> <dataFeedId>CSV files</dataFeedId> <skipFirstLinesCount>1</skipFirstLinesCount> </general>
Another example, no header at all (since 2012.02)
The CSV files look like:
WDD.MEM_BER0001_01_LT01_MW,A,2010-04-01 12:21:00,-0.000,GOOD WDD.MEM_BER0001_01_LT01_MW,A,2011-01-12 10:34:05,-0.001,GOOD WDD.MEM_BER0001_01_LT01_MW,A,2011-01-12 10:35:00,-0.011,BAD WDD.MEM_BER0001_01_LT01_MW,A,2011-01-12 10:36:00,-0.003,GOOD WDD.MEM_BER0001_01_LT01_MW,A,2011-01-12 10:37:00,-0.000,GOOD WDD.MEM_BER0001_01_LT01_MW,A,2011-01-12 10:38:00,-0.000,GOOD WDD.MEM_BER0001_01_LT01_MW,A,2011-01-12 10:39:00,-0.000,GOOD
<general> <importType>generalCSV</importType> <folder>$IMPORT_FOLDER_KETEN$</folder> <fileNamePatternFilter>*.csv</fileNamePatternFilter> <failedFolder>$IMPORT_FAILED_FOLDER_KETEN$</failedFolder> <backupFolder>$IMPORT_BACKUP_FOLDER_KETEN$</backupFolder> <table> <locationColumn/> <skippedColumn/> <dateTimeColumn pattern="yyyy-MM-dd HH:mm:ss"/> <valueColumn unit="SI"/> <flagColumn/> </table> <idMapId>IdKETEN</idMapId> <flagConversionsId>ImportKETENFlagConversions</flagConversionsId> <importTimeZone> <timeZoneOffset>+01:00</timeZoneOffset> </importTimeZone> <dataFeedId>CSV files</dataFeedId> </general>
Details of the import format
If the first line contains a comma, the decimal separator is taken to be a period (.), otherwise it is supposed to be a semicolon (;) and the decimal separator is taken to be a comma. This way locale-specific CSV files are supported.
The field separator is either a comma or a semicolon. Tabs are not supported.