WIWB

Overview

The WIWB Import connects to the Hydronet WIWB API and has support for downloading observations and forecasts for both scalar and grid data. WIWB has a concept of DataSources which have to be configured as a property of the import. Using the meta data api of  the WIWB API the type of data is determined and processed accordingly. The import feature is available since December 2018, from Delft-FEWS version 2016.02 onwards.

A detailed presentation highlighting various WIWB configuration options in Delft-FEWS can be found in the attached document.

Note that the various improved configuration options related to the WIWB import in Delft FEWS (as described in the detailed presentation) are made available for Delft-FEWS versions 2016.01 and higher, with a patch from later then December 2018.

A template configuration for WIWB imports are also available in the attached ZIP file. 

Import Configuration

To configure the WIWB import the dataSource property has to be specified. Se the list of supported dataSource values in the next section.

For ensembles the ensemble id can be configured using the ensembleId property. In case no property is configured, the default value "EPS" will be used.

DataSource values

The following data sources are currently provided through WIWB. New data sources might become available in future.

More information on the data sources can be obtained form the links provided in the table. All times are in UTC. The parameter code - between [brackets] - is the parameterID which must be used in the FEWS mapping configuration as 'parameter external'. 

An overview of the grid definitions of the complete grid extent are defined in the attached WIWB grid definitions file.

An extended table, offering more information about the WIWB parameters can be found the attached overview Excel file (version November 2018). 

Aggregated products

All data sources as in the table above have their original interval. If you want to import an aggregated product, the WIWB API supports that. You can aggregate every product with a multiplier of its original interval. The <timestep> configured in the Delft-FEWS import moduleInstance of that datasource will be interpreted whether you want to import the original or an aggregated product. E.g. the KNMI Radar products have an original interval of 5 minutes, so aggregation options are 10 minutes (2 x 5 minutes), 15 minutes (3 x 5 minutes) etc. As mentioned, the <timestep> definition in Delft-FEWS will be leading. Disaggregating is not supported.

The WIWB API has the default option that aggregation to larger timesteps is valid when at least one timestep is available. For example: a 5 minute timestep is aggregated to hourlies when at least one 5 minute timestep is available. For Delft-FEWS applications this is usually not the best solution. Therefore a so-called "minimumAggregationAvailability" option has been added. This value (of type double) has a valid range from 0 to 1 and indicates to the WIWB API the fraction of how much original timesteps should be available in the new timestep. Usually the value will be set to 1, as also can be seen from the below config example "aggregated radar products".

Authentication

By default the WIWB api is protected by a user name and password, that can be specified in the import configuration.

Since 2019.02 it is possible to specify a Bearer token. This is typically used when using OpenID Connect as authentication method. To enable a bearer token as authentication mechanism, the following property should be set:

<string key="authentication" value="BearerToken"  />

The actual bearer token should be set in the password field of the import configuratio

n. The username is not used when using a bearer token for authentication.

The hydronet (WIWB) API connection for the BHP project works on our standalones, but within the BHP network it is required to connect to the Hydronet API through the proxy API from BHP. As a consequence, the API call requires two additional parameters (X-IBM-Client-Id, X-IBM-Client-Secret). To set these headers use the properties below:

<properties>
			<string  key="xIBMClientId" value="ID"/>
			<string key="xIBMClientSecret" value="Secret"/>			
</properties>

Import types

The original FEWS import from WIWB data is done through a direct soap connection. Because this method is not intended to be used for bulky data downloads, another second so-called asynchronous download option has been developed. This async option is to be used for non-operational and larger datasets. The normal importType is WIWB, while the asynchronous option is WIWB_async. Note that the async option is only used in case of gridded. Scalar data is always imported through the norma, direct way. 

The WIWB_async import type works as follows: 

• It first sends a request to the WIWB API for a later download, this request contains all info about what data is requested.
• The API then returns a unique id which can be used to track the progress of the download, which will return one of the following status:

• Idle : this means it is on the waiting list for being processed, process will always be 0%
• • Processing : the download is being prepared this involves getting the data, processing the data and writing to netcdf files and zipping them. This will always have a progress larger than 0% and lower than 100%.
  • Error : something went wrong on the API server side, currently there will be no explanation provided on why it went wrong, therefor FEWS has now ability to provide an explanation for it since it is out of FEWS control.
  • Finished : The download is ready.
• When the download is ready a separate request is done for this specific download. First the whole zip is downloaded, then unzipped in a temporary directory and finally those files are imported one by one.

Missing values

If the WIWB specifies the missing value for a datasource, this will be used by FEWS. The missing value can also be specified in the import configuration as well. The default value for missing value is -9999 for the WIWB API.

Grid definitions

The WIWB API can provide the grid definitions of the models. The WIWB Import will log the Grid Definition if debug is enabled during the import.

It is possible to download a subextent of a grid by defining properties in the Import file. The methodology is described in detail in this presentation.

Note that defining a subgrid for the KNMI radar product is a bit more complicated due to the projection of the frid. The methodology to get the correct configuration values in the import file and the grids.xml can be found in this document: Nieuwe instructies voor import - KNMI Radar Subgrid.pdf.

View period

Where appropriate a viewPeriod can be configured. The view period can be no longer than one year, since the WIWP api doesn't support periods longer than that.

For model forecasts only the latest forecast will be downloaded.

Examples

This example is configured to import scalar observations from the Knmi.AwsTenMinutes dataSource, using the direct import:

<import>
	<general>
		<importType>WIWB</importType>
		<serverUrl>https://wiwb.hydronet.com/api</serverUrl>
		<user>$WIWB_USER$</user>
		<password>$WIWB_PW$</password>
		<relativeViewPeriod unit="hour" start="-120" end="0" startOverrulable="true"/>
		<idMapId>IdImportWIWB_obs</idMapId>
		<dataFeedId>WIWB (Knmi.AwsTenMinutes)</dataFeedId>
	</general>
	<properties>
		<string key="dataSource" value="Knmi.AwsTenMinutes"/>
	</properties>
	<timeSeriesSet>
		<moduleInstanceId>ImportKNMI</moduleInstanceId>
		<valueType>scalar</valueType>
		<parameterId>P.meting</parameterId>
		<locationSetId>KNMI-EVP</locationSetId>
		<timeSeriesType>external historical</timeSeriesType>
		<timeStep unit="minute" multiplier="10"/>
		<readWriteMode>add originals</readWriteMode>
	</timeSeriesSet>
</import>

The following example is configured for importing grids from the Knmi.Radar.Uncorrected dataSource, using the direct import:

<import>
	<general>
		<importType>WIWB</importType>
		<serverUrl>https://wiwb.hydronet.com/api</serverUrl>
		<user>$WIWB_USER$</user>
		<password>$WIWB_PW$</password>
		<relativeViewPeriod unit="hour" start="-12" end="-10" startOverrulable="false"/>
		<idMapId>IdImportWIWB_obs</idMapId>
		<dataFeedId>WIWB (Knmi.Radar.Uncorrected)</dataFeedId>
	</general>
	<properties>
		<string key="dataSource" value="Knmi.Radar.Uncorrected"/>
	</properties>
	<timeSeriesSet>
		<moduleInstanceId>ImportKNMI</moduleInstanceId>
		<valueType>grid</valueType>
		<parameterId>P.radar</parameterId>
		<qualifierId>uncorrected</qualifierId>
		<locationId>Hydronet_radar</locationId>
		<timeSeriesType>external historical</timeSeriesType>
		<timeStep unit="minute" multiplier="5"/>
		<readWriteMode>add originals</readWriteMode>
		<synchLevel>6</synchLevel>
	</timeSeriesSet>
</import>

Example of idMapping to some of the WIWB parameters in the IdImportWIWB_obs.xml file:

<idMap version="1.1" xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/idMap.xsd">
	<parameter external="Evaporation" internal="E.meting"/>
	<parameter external="P" internal="P.meting"/>
	<parameter external="P" internal="P.radar" internalQualifier="realtime"/>
	<parameter external="WindDirection" internal="Wind.meting.richting"/>
	<parameter external="WindSpeed" internal="Wind.meting.snelheid"/>
	<parameter external="TMP" internal="T.meting"/>
	<location external="Hydronet_radar" internal="Hydronet_radar"/>
	<locationIdPattern internalLocationSet="KNMI-EVP" internalLocationPattern="KNMI_*" externalLocationPattern="06*"/>
	<enableOneToOneMapping/>
</idMap>

Example of the grids.xml file for the KNMI Radar grid products

<?xml version="1.0" encoding="UTF-8"?>
<grids xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/grids.xsd">
	<regular locationId="knmi_radar_1km">
		<rows>765</rows>
		<columns>700</columns>
		<polarStereographic>
			<originLatitude>90</originLatitude>
			<originLongitude>0</originLongitude>
			<trueScalingLatitude>60</trueScalingLatitude>
			<equatorRadius>6378137</equatorRadius>
			<poleRadius>6356752</poleRadius>
		</polarStereographic>
		<firstCellCenter>
			<x>500</x>
			<y>-3650500</y>
			<z>0</z>
		</firstCellCenter>
		<xCellSize>1000</xCellSize>
		<yCellSize>1000</yCellSize>
	</regular>
</grids>

Example of a aggregated radar product using the asychronous import

<import>
	<general>
		<importType>WIWB_async</importType>
		<serverUrl>https://wiwb.hydronet.com/api</serverUrl>
		<connectionTimeOutMillis>900000</connectionTimeOutMillis>
		<user>$WIWB_USER$</user>
		<password>$WIWB_PW$</password>
		<relativeViewPeriod unit="hour" start="-24" end="0" startOverrulable="true"/>
		<idMapId>IdImportWIWB_radar</idMapId>
		<dataFeedId>WIWB (Knmi.Radar.CorrectedC2, hour)</dataFeedId>
	</general>
	<properties>
		<string key="dataSource" value="Knmi.Radar.CorrectedC2"/>
		<double key="minimumAggregationAvailability" value="1.00"/>
    </properties>
	<timeSeriesSet>
		<moduleInstanceId>ImportRadar</moduleInstanceId>
		<valueType>grid</valueType>
		<parameterId>P.radar</parameterId>
		<qualifierId>CorrectedC2</qualifierId>
		<locationId>Hydronet_radar</locationId>
		<timeSeriesType>external historical</timeSeriesType>
		<timeStep unit="hour" multiplier="1"/>
		<readWriteMode>add originals</readWriteMode>
		<synchLevel>6</synchLevel>
	</timeSeriesSet>
</import>

It is possible to request a selection extent only for any grid.