Custom time series import formats using java

SINCE FEWS 2009.01

Fews allows you to write your own time series import format in Java.

In the import module one can specify the fully qualified class name and the bin directory that contains a jar file with compiled java code and optimally other third party libaries. A jar file is just a zip file that contains you compiled java files.
e.g. classname = com.yourcompany.yourparsers.YourParser
and binDir = $REGION_HOME$/parsers/bin

The source code required for a simple format is only a few lines of code
A parser tells the content handler every thing it finds a file in the order it is available in the file. The content handler will map everything to the right time series.
The content handler will do the id mapping, unit conversion, datum conversion, translating text to decimal values, translating text to date/times with the specified time zone, convert missing values, trace values, validate the time series.

The content handler is also optimized for speed
It can handle 4 million lines per minute (about 400MB of text files)
It uses an optimized replacement for the java SimpleDateFormat and DecimalFormat to achieve this performance.

The import module will also open and close the files for you.
The import files can retain on the file system, in a zip file, tar file or gz file, ftp server, sftp server. The programmer will not notice the difference

Time Series Content handler

The interface is described in:

Types of parsers

Text parsers

Most import files are text based.

Binary parser

Grid coverage files are often binary

File parsers

This kind of parsers will use a third party library that not accepts streams

Database parsers

This parsers will read from a database.
Msaccess (mdb) of firebird (fdb) files are automatically recognized
It also possible to explicit configure a database connection with an external database in the import module

Server parsers

This kind of parsers will read data from a (internet) url
A url, username and password is provided to the parser

Additional consumer interfaces


Database and server parsers often needs a period in their query to the database of server.
When this interface implemented the import module will provide an absolute period.


Database and server parsers often needs the location and parameter ids in their queries
When this interface implemented the import module will convert the FEWS headers with specified id map and provide them to the parser. The mapping is used in the opposite direction compared to normal mapping. This can result in different mapping when the id map is not one internal to one external and visa versa.


With a virtual dir the parser can open meta files with additional information required for to parse file that retain in the same directory as the imported file. For example some grid coverage formats need an additional file for the geo referencing.


SINCE FEWS 2011.01
Ideally a parser does not need any additional information and is dedicated to parse one strict specified file format.
In some cases the parser needs additional configuration. Don't use a separate additional configuration file in this case because this file can not be managed by FEWS.
You can configure additional properties in the time series import module. By implementing the PropertiesConsumer interface the properties are injected in your parser.

      <string key="decimalSeparator" value="."/>
public class CsvTimeSeriesParser implements TextParser<TimeSeriesContentHandler>, PropertiesConsumer {
    public void setProperties(Properties properties) {
        decimalSeparator = properties.getString("decimalSeparator", ".");



XML parsers

Binary parsers (with header is separate file) (Little endian IEEE floats and integers)

Server parsers

Example project with source code

  • No labels