Overview
The WIWB Import connects to the Hydronet WIWB API and has support for downloading observations and forecasts for both scalar and grid data. WIWB has a concept of DataSources which have to be configured as a property of the import. Using the meta data api of the WIWB API the type of data is determined and processed accordingly. The import feature is available since December 2018, from Delft-FEWS version 2016.02 onwards.
More about WIWB
WIWB is the Dutch national database for storage and dissemination of weather related data, specifically for the Dutch water sector. It has been developed in 2017 by HydroLogic, HKV and HydroConsult for Het Waterschapshuis, a national body that represents all Dutch water boards. The WIWB database is based on the HydroNET platform (www.hydronet.com) and offers a REST API for retrieving data and metadata. WIWB contains weather observations and forecasts, based on weather stations, meteorological radar data and numerical weather models.
WIWB is intended for professional use only by Dutch water boards. It is not intended for public or commercial use. Only officials from Dutch water boards can request access to the WIWB API. More information on specifications and use of the WIWB API can be requested from the WWIB helpdesk (helpdesk@hydrologic.com) or in the Technische Instructies WIWB API.pdf.
Access to WIWB API
Access to the WIWIB API is limited to the regional water authorities in the Netherlands and can be requested from the Waterschapshuis Water (HWH) by sending an e-mail to Wouter Bakker. Please list the following items in your e-mail
- Organisation name
- Contact person and contact details (e-mail and telephone number).
- List with IP addresses that require access to the API.
- Short explanation of intended use:
- Systems to be connected to the API
- Required data sources
- Expected data request per time interval.
HWH will forward the required information to the WIWB back-office operated by HydroLogic. Your account will be configured within 5 days after the request was received by the WIWB back office.
More info about WIWB at HWH can be found here.
Technical instructions about the WIWB API can be found in the Technische Instructies WIWB API.pdf.
HTTPS and client.truststore
Access to the WIWIB API is done using a secure HTTPS connection. If a FEWS configuration is used with a client.truststore it may be required (for FEWS versions older than 2018.02) to import the latest public certificate of the WIWB service into the existing client.truststore. For more information on configuring secure HTTPS connection with a client.truststore, see How to configure secure https connection to Matroos
A detailed presentation highlighting various WIWB configuration options in Delft-FEWS can be found in the attached document.
Note that the various improved configuration options related to the WIWB import in Delft FEWS (as described in the detailed presentation) are made available for Delft-FEWS versions 2016.01 and higher, with a patch from later then December 2018.
A template configuration for WIWB imports are also available in the attached ZIP file.
Import Configuration
To configure the WIWB import the dataSource property has to be specified. Se the list of supported dataSource values in the next section.
For ensembles the ensemble id can be configured using the ensembleId property. In case no property is configured, the default value "EPS" will be used.
DataSource values
The following data sources are currently provided through WIWB. New data sources might become available in future.
More information on the data sources can be obtained form the links provided in the table. All times are in UTC. The parameter code - between [brackets] - is the parameterID which must be used in the FEWS mapping configuration as 'parameter external'.
An overview of the grid definitions of the complete grid extent are defined in the attached WIWB grid definitions file.
An extended table, offering more information about the WIWB parameters can be found the attached overview Excel file (version November 2018).
Datasource name | Datasource code (ID) | Type | Original Interval | Available Parameter(s) [parameter code] | Remarks |
---|---|---|---|---|---|
Harmonie 4.3 | Knmi.Harmonie.43 | Model Grid, forecast | 1 hour | Air pressure [AirPressure] | Grid definition is automatically read from WIWB, so no need to define in grids.xml |
Harmonie 4.0 | Knmi.Harmonie | Model Grid, forecast | 1 hour | Air pressure [AirPressure] | Grid definition is automatically read from WIWB, so no need to define in grids.xml |
Harmonie 36 | Knmi.Harmonie.36 | Model Grid, forecast | 1 hour | Air Pressure [AirPressure] | Grid definition is automatically read from WIWB, so no need to define in grids.xml |
HiRLAM | Knmi.Hirlam | Model Grid, forecast | 1 hour | Air Pressure [AirPressure] | Grid definition is automatically read from WIWB, so no need to define in grids.xml |
KNMI AWS 10 Minutes | Knmi.AwsTenMinutes | Scalar, observed | 10 minute | Air Pressure [AirPressure] | |
KNMI Evaporation (From Meteobase) | Knmi.FromMeteobase.Evaporation | Scalar, observed | 1 day | Evaporation [Evaporation] | |
KNMI Reference Evaporation | Knmi.Evaporation | Scalar, observed | 1 day | Evaporation [Evaporation] | |
KNMI Iris Stations | Knmi.IrisValidated | Scalar, observed | 1 day | Precipitation [P] | |
KNMI Naval Reports | Knmi.Naval.Forecasts | Report, observed | 1 day | Naval reports for warnings and forecasts [Naval.Reports] | |
Real-time international radar composite product | Knmi.International.Radar.Composite | Grid, radar | 5 minute | Precipitation [P] | Grid definition required in grids.xml. Product has no correction to data from meteo stations. |
Real-time international radar composite product | Knmi.International.Radar.Composite.Near.Realtime Knmi.International.Radar.Composite.Near.Realtime.Test | Grid, radar | 5 minute | Precipitation [P] | Grid definition required in grids.xml. Product has only realtime correction to data from meteo stations. The *.Test product is the beta product which uses a more advanced algorithm. |
Early reanalysis international radar composite | Knmi.International.Radar.Composite.Early.Reanalysis (Knmi.International.Radar.Composite.Early.Reanalysis.Test) | Grid, radar | 5 minute | Precipitation [P] | Grid definition required in grids.xml. Product available after 24-48 hours to enable early correction to data from meteo stations. The *.Test product is the beta product which uses a more advanced algorithm. |
Final reanalysis international radar composite | Knmi.International.Radar.Composite.Final.Reanalysis (Knmi.International.Radar.Composite.Final.Reanalysis.Test) | Grid, radar | 5 minute | Precipitation [P] | Grid definition required in grids.xml. Product available after 30 days to enable final correction to data from meteo stations. The *.Test product is the beta product which uses a more advanced algorithm. |
KNMI Radar Uncorrected (Real time) | Knmi.Radar.Uncorrected | Grid, radar | 5 minute | Precipitation [P] | Grid definition required in grids.xml. See below for the configuration example. |
KNMI Regional EPS | Knmi.RegionalEps | Ensemble Scalar, forecast | 6 hour | Dewpoint Temperature [DPT] | Note that these are the ensemble member: 1: Operational/deterministic run So it is advised to store the ensemble in a |
KNMI Synops | Knmi.Synops | Scalar, observed | 1 hour | Air Pressure [AirPressure] | |
KNMI Synops (From Meteobase) | Knmi.FromMeteobase.Synops | Scalar, observed | 1 hour | Precipitation [P] | |
KNMI Waqua TS | Knmi.WaquaTs | Model Scalar, forecast | 10 minute | Astronomical Tide [AstronomicalTide] | |
KNMI Weather Warnings | Knmi.Warnings | Warning, observed | 1 hour | Heat warning [Warning.Heat] | |
KNMI Water Setup EPS | Knmi.WaterSetupEps | Ensemble Scalar, forecast | 6 hour | Water Surge [WaterSurge] | |
Meteobase Makkink Evaporation | Meteobase.Evaporation.Makkink | Grid, observed | 1 day | Evaporation [Evaporation] | |
Meteobase Pennman-Monteith Evaporation | Meteobase.Evaporation.PennmanMonteith | Grid, observed | 1 day | Evaporation [Evaporation] | |
Meteobase Precipitation | Meteobase.Precipitation | Grid, observed | 1 hour | Precipitation [P] | |
SATDATA3.0 Evapotranspiration | Satdata.Evapotranspiration | Model grid, forecast | 1 day | Actual evapotranspiration [Evapotran-spirationActual] | Operational product with values for day -1, 0, 1 and 2 |
SATDATA3.0 Evapotranspiration Re-analysis | Satdata.Evapotranspiration.Reanalysis Satdata.Evapotranspiration.Reanalysis.V2 | Grid, observed | 1 day | Actual evapotranspiration [Evapotran-spirationActual] | Archive of reanalysis data: the regular product ranges from 2020-01-01 to 2022-06-01 and will not be available after 2023-04-01. The *.V2 products starts from 2012-07-24 and has a slightly improved algoritm. |
Interim Evapotranpiration dataproduct LHM [SATDATA] | Interim.Satdata.Evapotranspiration | Grid, observed | 1 day | Actual evapotranspiration [Evapotran-spirationActual] Evapotranspiration shortage [Evapo-transpirationShortage] | |
ECMWF Ensemble grid | Ecmwf.Ensemble.15day | Ensemble Grid, forecast | varies from 1-6 hour | Precipitation [P] | Note: this product is not yet supported |
Aggregated products
All data sources as in the table above have their original interval. If you want to import an aggregated product, the WIWB API supports that. You can aggregate every product with a multiplier of its original interval. The <timestep> configured in the Delft-FEWS import moduleInstance of that datasource will be interpreted whether you want to import the original or an aggregated product. E.g. the KNMI Radar products have an original interval of 5 minutes, so aggregation options are 10 minutes (2 x 5 minutes), 15 minutes (3 x 5 minutes) etc. As mentioned, the <timestep> definition in Delft-FEWS will be leading. Disaggregating is not supported.
The WIWB API has the default option that aggregation to larger timesteps is valid when at least one timestep is available. For example: a 5 minute timestep is aggregated to hourlies when at least one 5 minute timestep is available. For Delft-FEWS applications this is usually not the best solution. Therefore a so-called "minimumAggregationAvailability" option has been added. This value (of type double) has a valid range from 0 to 1 and indicates to the WIWB API the fraction of how much original timesteps should be available in the new timestep. Usually the value will be set to 1, as also can be seen from the below config example "aggregated radar products".
Authentication with OpenID Connect
By default the HydroNet WIWB API is protected by a user name and password, that can be specified in the import configuration.
Since 2019.02 it is possible to specify a Bearer token. This is typically used when using OpenID Connect as authentication method. To enable a bearer token as authentication mechanism, the following properties should be set:
- authentication, which is to be defined as "BearerToken"
- tokenUrl, which should direct to the URL that is used to get the BearerToken. This URL is to be set to https://login.hydronet.com/auth/realms/hydronet/protocol/openid-connect/token
<string key="authentication" value="BearerToken"/> <string key="tokenUrl" value="https://login.hydronet.com/auth/realms/hydronet/protocol/openid-connect/token"/>
Note that the username and password still has to be set in the general section to be able to login. The username is provided as the "clientId" and the password is provided as the "clientSecret".
<import> <general> <importType>WIWB</importType> <serverUrl>https://wiwb.hydronet.com/api</serverUrl> <connectionTimeOutMillis>900000</connectionTimeOutMillis> <user>$MY_WIWB_CLIENT_ID$</user> <password>$MY_WIWB_CLIENT_SECRET$</password> <relativeViewPeriod unit="hour" start="-2" end="0" startOverrulable="false"/> <idMapId>IdImportWIWB_radar</idMapId> <unitConversionsId>ImportKNMIUnits</unitConversionsId> <dataFeedId>WIWB (Knmi.International.Radar.Composite 5min)</dataFeedId> <actionLogEventTypeId>Imported.IRC</actionLogEventTypeId> </general> <properties> <string key="dataSource" value="Knmi.International.Radar.Composite"/> <string key="authentication" value="BearerToken"/> <string key="tokenUrl" value="https://login.hydronet.com/auth/realms/hydronet/protocol/openid-connect/token"/> </properties> <timeSeriesSet> <moduleInstanceId>ImportWIWB</moduleInstanceId> <valueType>grid</valueType> <parameterId>P.radar</parameterId> <locationId>KNMI-RADAR</locationId> <timeSeriesType>external historical</timeSeriesType> <timeStep unit="minute" multiplier="5"/> <readWriteMode>add originals</readWriteMode> </timeSeriesSet> </import>
Proxy Servers
When the HydroNet WIWB API connection is required to connect to the Hydronet API through a proxy like MuleSoft, additional authentication information might be required in the header. It is possible to add two additional parameters clientId and clientSecret. The default headers are X-IBM-Client-Id and X-IBM-Client-Secret, but these can be changed by configuring the respective headers.
<properties> <string key="dataSource" value="BHP.Composite.Adjusted"/> <string key="clientId" value="[to be provided by proxy admin]"/> <string key="clientIdHeader" value="[to be provided by proxy admin]"/> <string key="clientSecret" value="[to be provided by proxy admin]"/> <string key="clientSecretHeader" value="[to be provided by proxy admin]"/> <bool key="logClientIdHeader" value="true"/> </properties>
Starting with May 1 2022, Hydronet API uses a different endpoint to provide the users with an access token (BearerToken, see above). To retrieve the token a client id, client secret and a token URL are necessary. The client ID and secret can be configured as username and password respectively. The token URL should be configured as a property. If configured, FEWS will retrieve the token for the import. If it isn’t added to the config, FEWS will assume that the old authentication process is still in use.
To set these headers use the properties below:
<properties> <string key="dataSource" value="BHP.Composite.Adjusted"/> <string key="authentication" value="BearerToken"/> <string key="clientId" value="to be provided by proxy admin"/> <string key="clientIdHeader" value="to be provided by proxy admin"/> <string key="clientSecret" value="to be provided by proxy admin"/> <string key="clientSecretHeader" value="to be provided by proxy admin"/> <bool key="logClientIdHeader" value="true"/> <string key="tokenUrl" value="https://login.hydronet.com/auth/realms/hydronet/protocol/openid-connect/token"/> </properties>
Import types: WIWB and WIWB_async (grids only)
The original FEWS import from WIWB data is done through a direct soap connection. Because this method is not intended to be used for bulky data downloads, another second so-called asynchronous download option has been developed. This async option is to be used for non-operational and larger datasets. The normal importType is WIWB, while the asynchronous option is WIWB_async. Note that the async option is only used in case of gridded data. Scalar data is always imported through the normal, direct way.
The WIWB_async import type works as follows internally in the import module:
- It first sends a request to the WIWB API for a later download, this request contains all info about what data is requested.
- The API then returns a unique id which can be used to track the progress of the download, which will return one of the following status:
- Idle : this means it is on the waiting list for being processed, process will always be 0%
- Processing: the download is being prepared this involves getting the data, processing the data and writing to netcdf files and zipping them. This will always have a progress larger than 0% and lower than 100%.
- Error: something went wrong on the API server side. Currently there will be no explanation provided on why it went wrong, therefor FEWS has now ability to provide an explanation for it since it is out of FEWS control.
- Finished: The download is ready.
- Idle : this means it is on the waiting list for being processed, process will always be 0%
- When the download is ready a separate request is done for this specific download. First the whole zip is downloaded, then unzipped in a temporary directory and finally those files are imported one by one.
Note that all of this accounting and communication is handled automatically by the FEWS client and the WIWB servers and progress is displayed as INFO messages in the logs. When the processing is finished, Delft-FEWS will download and import the dataset.
WIWB_async: download disclaimer
The amount of data in one WIWB request (download option) also has its maximum. This maximum data request is limited to the currently installed WIWB-server (internal) memory capacity, which is 12 GB. This means that an individual data request can not exceed this size. This maximum of 12 GB is valid per data request, since all data requests are processed sequentially. If the data request is too large, the server runs out of memory and this data requests fails. Delft-FEWS reports that an error has occurred at the WIWB-server side. To download the requested data, the user needs to split up the data request. A common approach would be to reduce the number of timesteps. Note that the requested view period can never exceed a period of one year, as this is limited by WIWB.
Below, three examples of benchmark data requests are given. These examples stay within the current WIWB-server limits and will be successful. Please use these as inspiration for your own data requests.
Spatial extent | Period (RVP) | Timestep |
---|---|---|
Netherlands | 20 days | 5 min |
Netherlands | 8 months | 1 hour |
Waterboard | Complete KNMI IRC Final Reanalysis dataset | 5 min |
Missing values
If the WIWB specifies the missing value for a datasource, this will be used by FEWS. The missing value can also be specified in the import configuration as well. The default value for missing value is -9999 for the WIWB API.
Grid definitions
The WIWB API can provide the grid definitions of the models. The WIWB Import will log the Grid Definition if debug is enabled during the import.
Radar Grid definition
Note that the grid definition of the KNMI radar grids (which is polar stereographic) is not supported by the WIWB netcdf libraries. As a result of that, the returned netcdf file is not correct with respect to it's grid definition. That definition should not be used
It is possible to download a subextent of a grid by defining properties in the Import file. The methodology is described in detail in this presentation.
Note that defining a subgrid for the KNMI radar product is a bit more complicated due to the projection of the grid. The methodology to get the correct configuration values in the import file and the grids.xml can be found in this document: Nieuwe instructies voor import - KNMI Radar Subgrid.pdf.
View period
Where appropriate a viewPeriod can be configured. The view period can be no longer than one year, since the WIWP api doesn't support periods longer than that.
For model forecasts only the latest forecast will be downloaded.
Examples
This example is configured to import scalar observations from the Knmi.AwsTenMinutes dataSource, using the direct import:
<import> <general> <importType>WIWB</importType> <serverUrl>https://wiwb.hydronet.com/api</serverUrl> <user>$MY_WIWB_CLIENT_ID$</user> <password>$MY_WIWB_CLIENT_SECRET$</password> <relativeViewPeriod unit="hour" start="-120" end="0" startOverrulable="true"/> <idMapId>IdImportWIWB_obs</idMapId> <dataFeedId>WIWB (Knmi.AwsTenMinutes)</dataFeedId> </general> <properties> <string key="dataSource" value="Knmi.AwsTenMinutes"/> <string key="authentication" value="BearerToken"/> <string key="tokenUrl" value="https://login.hydronet.com/auth/realms/hydronet/protocol/openid-connect/token"/> </properties> <timeSeriesSet> <moduleInstanceId>ImportKNMI</moduleInstanceId> <valueType>scalar</valueType> <parameterId>P.meting</parameterId> <locationSetId>KNMI-EVP</locationSetId> <timeSeriesType>external historical</timeSeriesType> <timeStep unit="minute" multiplier="10"/> <readWriteMode>add originals</readWriteMode> </timeSeriesSet> </import>
The following example is configured for importing grids from the Knmi.Radar.Uncorrected dataSource, using the direct import:
<import> <general> <importType>WIWB</importType> <serverUrl>https://wiwb.hydronet.com/api</serverUrl> <user>$MY_WIWB_CLIENT_ID$</user> <password>$MY_WIWB_CLIENT_SECRET$</password> <relativeViewPeriod unit="hour" start="-12" end="-10" startOverrulable="false"/> <idMapId>IdImportWIWB_obs</idMapId> <dataFeedId>WIWB (Knmi.Radar.Uncorrected)</dataFeedId> </general> <properties> <string key="dataSource" value="Knmi.Radar.Uncorrected"/> <string key="authentication" value="BearerToken"/> <string key="tokenUrl" value="https://login.hydronet.com/auth/realms/hydronet/protocol/openid-connect/token"/> </properties> <timeSeriesSet> <moduleInstanceId>ImportKNMI</moduleInstanceId> <valueType>grid</valueType> <parameterId>P.radar</parameterId> <qualifierId>uncorrected</qualifierId> <locationId>Hydronet_radar</locationId> <timeSeriesType>external historical</timeSeriesType> <timeStep unit="minute" multiplier="5"/> <readWriteMode>add originals</readWriteMode> <synchLevel>6</synchLevel> </timeSeriesSet> </import>
Example of idMapping to some of the WIWB parameters in the IdImportWIWB_obs.xml file:
<idMap version="1.1" xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/idMap.xsd"> <parameter external="Evaporation" internal="E.meting"/> <parameter external="P" internal="P.meting"/> <parameter external="P" internal="P.radar" internalQualifier="realtime"/> <parameter external="WindDirection" internal="Wind.meting.richting"/> <parameter external="WindSpeed" internal="Wind.meting.snelheid"/> <parameter external="TMP" internal="T.meting"/> <location external="Hydronet_radar" internal="Hydronet_radar"/> <locationIdPattern internalLocationSet="KNMI-EVP" internalLocationPattern="KNMI_*" externalLocationPattern="06*"/> <enableOneToOneMapping/> </idMap>
Example of the grids.xml file for the KNMI Radar grid products
<?xml version="1.0" encoding="UTF-8"?> <grids xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/grids.xsd"> <regular locationId="knmi_radar_1km"> <rows>765</rows> <columns>700</columns> <polarStereographic> <originLatitude>90</originLatitude> <originLongitude>0</originLongitude> <trueScalingLatitude>60</trueScalingLatitude> <equatorRadius>6378137</equatorRadius> <poleRadius>6356752</poleRadius> </polarStereographic> <firstCellCenter> <x>500</x> <y>-3650500</y> <z>0</z> </firstCellCenter> <xCellSize>1000</xCellSize> <yCellSize>1000</yCellSize> </regular> </grids>
Example of a aggregated radar product using the asychronous import
<import> <general> <importType>WIWB_async</importType> <serverUrl>https://wiwb.hydronet.com/api</serverUrl> <connectionTimeOutMillis>900000</connectionTimeOutMillis> <user>$MY_WIWB_CLIENT_ID$</user> <password>$MY_WIWB_CLIENT_SECRET$</password> <relativeViewPeriod unit="hour" start="-24" end="0" startOverrulable="true"/> <idMapId>IdImportWIWB_radar</idMapId> <dataFeedId>WIWB (Knmi.Radar.CorrectedC2, hour)</dataFeedId> </general> <properties> <string key="dataSource" value="Knmi.Radar.CorrectedC2"/> <double key="minimumAggregationAvailability" value="1.00"/> <string key="authentication" value="BearerToken"/> <string key="tokenUrl" value="https://login.hydronet.com/auth/realms/hydronet/protocol/openid-connect/token"/> </properties> <timeSeriesSet> <moduleInstanceId>ImportRadar</moduleInstanceId> <valueType>grid</valueType> <parameterId>P.radar</parameterId> <qualifierId>CorrectedC2</qualifierId> <locationId>Hydronet_radar</locationId> <timeSeriesType>external historical</timeSeriesType> <timeStep unit="hour" multiplier="1"/> <readWriteMode>add originals</readWriteMode> <synchLevel>6</synchLevel> </timeSeriesSet> </import>
It is possible to request a selection extent only for any grid.
Multiple external forecasts
Example of requesting multiple (older) external forecasts using externalForecastTimesSearchRelativePeriod & externalForecastTimesCardinalTimeStep:
<import> <general> <importType>WIWB</importType> <serverUrl>https://wiwb.hydronet.com/api</serverUrl> <connectionTimeOutMillis>900000</connectionTimeOutMillis> <user>$MY_WIWB_CLIENT_ID$</user> <password>$MY_WIWB_CLIENT_SECRET$</password> <externalForecastTimesSearchRelativePeriod unit="day" start="-7" end="0"/> <externalForecastTimesCardinalTimeStep unit="hour" multiplier="12"> <idMapId>IdImportWIWB_fc</idMapId> <dataFeedId>WIWB (Knmi.RegionalEps)</dataFeedId> </general> <properties> <string key="dataSource" value="Knmi.RegionalEps"/> <string key="authentication" value="BearerToken"/> <string key="tokenUrl" value="https://login.hydronet.com/auth/realms/hydronet/protocol/openid-connect/token"/> </properties> <timeSeriesSet> <moduleInstanceId>ImportKNMI</moduleInstanceId> <valueType>scalar</valueType> <parameterId>P.voorsp.ens</parameterId> <locationSetId>KNMI-EPS</locationSetId> <timeSeriesType>external forecasting</timeSeriesType> <timeStep unit="hour" multiplier="6"/> <readWriteMode>add originals</readWriteMode> <ensembleId>EPS</ensembleId> </timeSeriesSet> </import>
Only requesting new data by default
When the WIWB import is connected to a <dataFeedId> FEWS keeps track of the latest time step imported and it will limit the request to a period that is only after that latest time step.
This has been done to be more efficient and prevent the import to keep requesting data that is already imported.
In certain cases this may not be desirable, for instance to request older data that has been missed (it may not have been available or the import didn't run for some time)
In that case the option <trimPeriodWhenLastImportedTimeStepAfterPeriod> can be set to false, then the full period will be requested.