Custom time series import formats using java
SINCE FEWS 2009.01
Fews allows you to write your own time series import format in Java.
In the import module one can specify the fully qualified class name and the bin directory that contains a jar file with compiled java code and optimally other third party libaries. A jar file is just a zip file that contains you compiled java files.
e.g. classname = com.yourcompany.yourparsers.YourParser
and binDir = $REGION_HOME$/parsers/bin
The source code required for a simple format is only a few lines of code
A parser tells the content handler every thing it finds a file in the order it is available in the file. The content handler will map everything to the right time series.
The content handler will do the id mapping, unit conversion, datum conversion, translating text to decimal values, translating text to date/times with the specified time zone, convert missing values, trace values, validate the time series.
The content handler is also optimized for speed
It can handle 4 million lines per minute (about 400MB of text files)
It uses an optimized replacement for the java SimpleDateFormat and DecimalFormat to achieve this performance.
The import module will also open and close the files for you.
The import files can retain on the file system, in a zip file, tar file or gz file, ftp server, sftp server. The programmer will not notice the difference
Time Series Content handler
The interface is described in: TimeSeriesContentHandler.java
Types of parsers
Text parsers
Most import files are text based.
Binary parser
Grid coverage files are often binary
File parsers
This kind of parsers will use a third party library that not accepts streams
Database parsers
This parsers will read from a database.
Msaccess (mdb) of firebird (fdb) files are automatically recognized
It also possible to explicit configure a database connection with an external database in the import module
Server parsers
This kind of parsers will read data from a (internet) url
A url, username and password is provided to the parser
Additional consumer interfaces
PeriodConsumer
Database and server parsers often needs a period in their query to the database of server.
When this interface implemented the import module will provide an absolute period.
TimeSeriesHeadersConsumer
Database and server parsers often needs the location and parameter ids in their queries
When this interface implemented the import module will convert the FEWS headers with specified id map and provide them to the parser. The mapping is used in the opposite direction compared to normal mapping. This can result in different mapping when the id map is not one internal to one external and visa versa.
VirtualDirConsumer
With a virtual dir the parser can open meta files with additional information required for to parse file that retain in the same directory as the imported file. For example some grid coverage formats need an additional file for the geo referencing.
PropertiesConsumer
SINCE FEWS 2011.01
Ideally a parser does not need any additional information and is dedicated to parse one strict specified file format.
In some cases the parser needs additional configuration. Don't use a separate additional configuration file in this case because this file can not be managed by FEWS.
You can configure additional properties in the time series import module. By implementing the PropertiesConsumer interface the properties are injected in your parser.
<import>
<general>
<importType>PI</importType>
<folder>$IMPORT_FOLDER$/meteo/tabel/tk</folder>
<failedFolder>$IMPORT_FAILED_FOLDER$/meteo/tabel/tk</failedFolder>
<backupFolder>$IMPORT_BACKUP_FOLDER$/meteo/tabel/tk</backupFolder>
<idMapId>IdImportMeteo_1d</idMapId>
<unitConversionsId>ImportUnitConversions</unitConversionsId>
<dataFeedId>Neerslag_24uur</dataFeedId>
</general>
<properties>
<string key="decimalSeparator" value="."/>
/properties>
<timeSeriesSet>
public class CsvTimeSeriesParser implements TextParser<TimeSeriesContentHandler>, PropertiesConsumer {
@Override
public void setProperties(Properties properties) {
decimalSeparator = properties.getString("decimalSeparator", ".");
}
Examples
TextParsers
LocationIdsHeaderCsvParser.java
KnmiSynopsTimeSeriesParser.java
NtuQuarterTimeSeriesParser.java
NtuQuarterTimeSeriesParser.java
XML parsers
PiTimeSeriesParser.java
PiMapStackParser.java
Binary parsers
GrayscaleImageTimeSeriesParser.java (with header is separate file)
MosaicRadarTimeSeriesParser.java (Little endian IEEE floats and integers)
NimrodGridTimeSeriesParser.java
Server parsers
RmoTimeSeriesServerParser.java
