Custom time series

...

import formats using java

SINCE FEWS 2009.01

Fews allows you to write your own time series import format with javain Java.

In the import module you one can specify the fully qualified class name and the bin directory that contains a jar file with you compiled java code and optimally other third party libaries. A jar file is just a zip file that contains you compiled java files.
e.g. classname = com.yourcompany.yourparsers.YourParser
and binDir = $REGION_HOME$/parsers/bin

...

The import module will also open and close the files for you.
The import files can retain on the file system, in a zip file, tar file or gz file, ftp server, sftp server. The programmer will not notice the difference

Time Series Content handler

The interface is described in: TimeSeriesContentHandler.java

Types of parsers

Text parsers

Most import files are text based.

Binary parser

Grid coverage files are often binary

File parsers

This kind of parsers will use a third party library that not accepts streams

Database parsers

This parsers will read from a database.
Msaccess (mdb) of firebird (fdb) files are automatically recognized
It also possible to explicit configure a database connection with an external database in the import module

Server parsers

This kind of parsers will read data from a (internet) url
A url, username and password is provided to the parser

Additional consumer interfaces

PeriodConsumer

Database and server parsers often needs a period in their query to the database of server.
When this interface implemented the import module will provide an absolute period.

TimeSeriesHeadersConsumer

Database and server parsers often needs the location and parameter ids in their queries
When this interface implemented the import module will convert the FEWS headers with specified id map and provide them to the parser. The mapping is used in the opposite direction compared to normal mapping. This can result in different mapping when the id map is not one internal to one external and visa versa.

VirtualDirConsumer

With a virtual dir the parser can open meta files with additional information required for to parse file that retain in the same directory as the imported file. For example some grid coverage formats need an additional file for the geo referencing.

PropertiesConsumer

SINCE FEWS 2011.01
Ideally a parser does not need any additional information and is dedicated to parse one strict specified file format.
In some cases the parser needs additional configuration. Don't use a separate additional configuration file in this case because this file can not be managed by FEWS.
You can configure additional properties in the time series import module. By implementing the PropertiesConsumer interface the properties are injected in your parser.

<import>
    <general>
      <importType>PI</importType>
      <folder>$IMPORT_FOLDER$/meteo/tabel/tk</folder>
      <failedFolder>$IMPORT_FAILED_FOLDER$/meteo/tabel/tk</failedFolder>
      <backupFolder>$IMPORT_BACKUP_FOLDER$/meteo/tabel/tk</backupFolder>
      <idMapId>IdImportMeteo_1d</idMapId>
      <unitConversionsId>ImportUnitConversions</unitConversionsId>
      <dataFeedId>Neerslag_24uur</dataFeedId>
    </general>
    <properties>
      <string key="decimalSeparator" value="."/>
    /properties>
    <timeSeriesSet>

public class CsvTimeSeriesParser implements TextParser<TimeSeriesContentHandler>, PropertiesConsumer {
    @Override
    public void setProperties(Properties properties) {
        decimalSeparator = properties.getString("decimalSeparator", ".");
    }

Examples

TextParsers

LocationIdsHeaderCsvParser.java

HcsTimeSeriesParser.java

WQCSVTimeSeriesParser.java

...

WiskiTimeSeriesParser.java

XML parsers

PiTimeSeriesParser.java
PiMapStackParser.java

Binary parsers

GrayscaleImageTimeSeriesParser.java (with header is separate file)

MosaicRadarTimeSeriesParser.java (Little endian IEEE floats and integers)

NimrodGridTimeSeriesParser.java

Server parsers

RmoTimeSeriesServerParser.java

Example project with source code

FEWS_Example_Custom_Import_Export.119649.zip