Overview

Imports time series data with radar rainfall information from the KNMI that are delivered to the Dutch waterboards. The files are in HDF5 file format. See KNMI site for information on the files.

Notice that the rainfall forecast is supplied as an accumulative rainfall sum in 0.01 mm over the last 3 hours where the time is in GMT. The files are supplied once per hour.

KNMI supplies several radar images, like:

  • 5 minute radar intensity (uncalibrated)
  • accumulated calibrated precipitation of last 3 hours, supplied every hour
  • accumulated calibrated daily precipitation, supplied every day

The accumulated precipitation files contain the rainfall depth (in millimeters). The 5-minute radar intensity files contain a 8-bit value (0-255) that represents the rainfall depth. To convert from this bit value to normal rainfall depth an additional conversion should be applied. The conversion table is listed at KNMI site. The conversion can be done by a Transformation that uses a log-function that fits the conversion table.

This description continues using the accumulated precipitation as example configuration.

This import uses a general C++ DLL for reading the HDF5 files. On some Windows systems the correct runtime components of Visual C++ Libraries are not installed by default. A Microsoft Visual C++ 2008 SP1 Redistributable Package must be installed on the computers to solve the problem. Problems have been found on Windows 2003 and Windows 2008 server computers.

On Linux importing HDF5 files will fail if the operating system is too old. From available evidence,
the kernel must be at GLIBC 2.6.18 (see the output of the "uname -a" command).

Configuration (Example)

A complete import module configuration consists of an ID Mapping file and a Import Module Instance file. To convert the rainfall in a proper unit (from 0.01 mm per 3 hour to mm/hr for example) it is also required to configure a Unit Conversion file.

ModuleConfigFiles

The following example of an Import Module Instance will import the time series as equidistant series for timezone GMT with a time step of 3 hours.

ImportKNMI.xml
<timeSeriesImportRun ......">
  <import>
    <!--Radar-->
    <general>
      <importType>KNMI-HDF5</importType>
      <folder>$IMPORT_FOLDER_KNMI_RADAR$</folder>
      <failedFolder>$IMPORT_FAILED_FOLDER_KNMI_RADAR$</failedFolder>
      <backupFolder>$IMPORT_BACKUP_FOLDER_KNMI_RADAR$</backupFolder>
      <idMapId>IdImportRADAR</idMapId>
      <unitConversionsId>ImportKNMIUnits</unitConversionsId>
      <!--radar is in GMT-->
      <importTimeZone>
        <timeZoneOffset>+00:00</timeZoneOffset>
      </importTimeZone>
      <dataFeedId>KNMI-RADAR</dataFeedId>
    </general>
    <timeSeriesSet>
      <moduleInstanceId>ImportKNMI</moduleInstanceId>
      <valueType>grid</valueType>
      <parameterId>P.radar.cal</parameterId>
      <locationId>KNMI-RADAR</locationId>
      <timeSeriesType>external historical</timeSeriesType>
      <timeStep unit="hour" multiplier="3" timeZone="GMT+1"/>
      <readWriteMode>add originals</readWriteMode>
      <synchLevel>6</synchLevel>
    </timeSeriesSet>
    <!--to let the import module know that the KNMI rainfall is in 0.01 mm/3hr that should be converted to for example mm/hr-->
    <externUnit parameterId="P.radar.cal" unit="0.01 mm/3hr"/>
  </import>
</timeSeriesImportRun>

IdMapFiles

Defines mappings between KNMI and FEWS parameters and locations.

sample of IdImportRADAR.xml
<idMap version="1.1" ......>
  <map internalParameter="P.radar.cal" internalLocation="KNMI-RADAR" externalParameter="image_data" externalLocation="KNMI-RADAR"/>
</idMap>

UnitConversionFile

Defines the conversion of the units that should be applied.

sample of ImportKNMIUnits.xml
<?xml version="1.0" encoding="UTF-8"?>
<unitConversions ...................>
  <unitConversion>
    <inputUnitType>0.01 mm/3hr</inputUnitType>
    <outputUnitType>mm/hr</outputUnitType>
    <multiplier>0.00333333</multiplier>
    <incrementer>0</incrementer>
  </unitConversion>
  ........
  ........
</unitConversions>

Grid definition

Defines the definition of the radar grid. This definition is not read from the file as in GRIB files like HIRLAM is done. Therefore a grid definition is required for the KNMI radar grid.

sample of Grids.xml
<?xml version="1.0" encoding="UTF-8"?>
<grids ...........>
  <regular locationId="KNMI-RADAR">
    <rows>765</rows>
    <columns>700</columns>
    <polarStereographic>
      <originLatitude>90</originLatitude>
      <originLongitude>0</originLongitude>
      <trueScalingLatitude>60</trueScalingLatitude>
      <equatorRadius>6378137</equatorRadius>
      <poleRadius>6356752</poleRadius>
    </polarStereographic>
    <firstCellCenter>
      <x>500</x>        <!-- = projectie_shift + 1/2 cellsize = 0 + 500 -->
      <y>-3650500</y> <!-- = projectie_shift + 1/2 cellsize = 3650000 + 500 -->
      <z>0</z>
    </firstCellCenter>
    <xCellSize>1000</xCellSize>
    <yCellSize>1000</yCellSize>
  </regular>
  <!-- the old KNMI 2,5 km grid-->
  <regular locationId="KNMI-RADAR2.5km">
    <rows>256</rows>
    <columns>256</columns>
    <polarStereographic>
      <originLatitude>90</originLatitude>
      <originLongitude>0</originLongitude>
      <trueScalingLatitude>60</trueScalingLatitude>
      <equatorRadius>6378388</equatorRadius>
      <poleRadius>6356912</poleRadius>
    </polarStereographic>
    <firstCellCenter>
      <x>1250</x>     <!-- = projectie_shift + 1/2 cellsize = 0 + 1250 -->
      <y>-3728515</y> <!-- = projectie_shift + 1/2 cellsize = 3727265 + 1250 -->
      <z>0</z>
    </firstCellCenter>
    <xCellSize>2500</xCellSize>
    <yCellSize>2500</yCellSize>
  </regular>
</grids>

Java source code

KnmiHdf5Parser.java

  • No labels

Disclaimer