Introduction
The Raven Modelling Framework is integrated with Delft-FEWS for a variety of inflow forecasting systems. This connection is currently done with a Python based model adaptor developed by Alan Barton of the National Research Council of Canada. See Figure 1.1 for a high level schematic of the current Raven-FEWS Connections.
The adaptor has been iteratively improved to allow advanced functionality between Delft-FEWS and Raven. However there are limitations identified with the current adaptor approach, namely:
- Dependencies of the model package, including use of Python 2.7 which is no longer actively supported.
- Limitation in flexibility for new applications, which vary from the initial applications for which it was developed.
- Additional processing time in the conversion of model files.
Figure 1.1: Current Raven-FEWS Connections - High Level Schematic
Recent implementations and iterations of the Raven have included the capability of netCDF reading and writing, removing the need for an intermediary adaptor for some functionality.
In collaboration with James Craig, the developer of the Raven model software, a design was made for seamless integration of Raven. In this “adaptor-less” approach the output of the FEWS general adaptor can be directly read by the Raven model. See Figure 1.2 for the desired approach.
Figure 1.2: Desired Raven-FEWS Connections - High Level Schematic
This design required changes to the Raven software as well as to Delft-FEWS (only for model settings, see par 5.6). The result of this development leads to several important benefits:
- Increased performance (faster workflows),
- Simplified configuration
- More robust integration due to id mapping
- Increase of the number of parameters/variables that are allowed
- Reduced adaptor logging information, allowing for cleared debugging.
Supported by the Delft-FEWS users employing Raven models in their forecasting systems, James Craig and Deltares have implemented the new approach. This report describes the configuration needed to integrate Raven via the new approach. Because of the versatility of Delft-FEWS, deviations from the described configuration approach are possible. Most of the described Delft-FEWS functionality is available in version NA 2019.02 (and probably also 2018.02): where NA 2020.02 is required this is mentioned. The Raven version containing all the relevant functionality is: 3.66.
Module folder structure
Although different set-ups are possible, the following folder structure is used in the following examples:
- Modules\Raven\bin\Raven.exe
- Modules\Raven\[Basin]
- Modules\Raven\[Basin]\work: contains model files
- Modules\Raven\[Basin]\input: contains input files
- Modules\Raven\[Basin]\output: contains output files
This is not prescribed, but a uniform approach is advised. With the above approach, it is possible to run individual Raven simulations in the %TEMP_DIR% folder, which allows FEWS to kickoff multiple simulations in parallel by placing them in unique folders in the FEWS temp directory. This greatly speeds-up ensemble forecasts.
Timezone
In case Raven is run with daily timesteps, input timeseries should have a timezone setting without daylight saving. If timesteps for a timezone with daylight saving are defined, Raven will crash when the simulation period contains the moment when daylight saving is started or ended.
Raven executable
The Raven executable is preferably included as a ModuleDataSet in the configuration. By including this moduleDataSet in the clientconfig used by OC and FSS, the Raven executable will be deployed automatically:
<autoExportModuleDataSet name="Raven" exportDir="Modules/Raven/bin"/>
General Settings
Raven
The following general settings are needed in the rvi file for integrating a model in Delft-FEWS (see also C.1 of the Raven user manual):
:DeltaresFEWSMode
:WriteNetCDFFormat
:OutputDirectory ../output/
:FEWSRunInfoFile ../run_info.nc
The DeltaresFEWSMode has several functions:
- The model will not crash in case the first timestep of a forcing timeserie has missing values which Raven would typically complain about
- Corrects for the missing data in the first time index when checking whether the input timespan overlaps the simulation duration
- Relaxes the constraint that memory chunks are complete days, i.e., start and end at midnight
- Raven will only use location IDs for reading input data and writing output data. Without this setting, location names are used for output data.
Model RunName
There are several ways the model can be provided with a RunName:
- -the -r runname flag from the command line, or
- -the RunName attribute in the properties variable of the RunInfo.nc file
- -the :RunName command in the .rvi file
The RunName is used as prefix for the output files: this is relevant in the configuration of the ImportActivity. If no RunName is given (or has zero length), no prefix will be used. If the Runname appears in more than one of these, the precedence takes over, with command line dominating the others.
GA - General section
The general section mentioned below follows the folder structure as mentioned above:
<general> <description>Raven Model for $MODEL$</description> <piVersion>1.5</piVersion> <rootDir>%REGION_HOME%/Modules/Raven/</rootDir> <workDir>%ROOT_DIR%/$MODEL$/work</workDir> <exportDir>%ROOT_DIR%/$MODEL$/input</exportDir> <exportDataSetDir>%ROOT_DIR%/$MODEL$/work</exportDataSetDir> <exportIdMap>IdExportRaven</exportIdMap> <exportUnitConversionsId>ExportRaven</exportUnitConversionsId> <importDir>%ROOT_DIR%/$MODEL$/output</importDir> <importIdMap>IdImportRaven</importIdMap> <dumpFileDir>$GA_DUMPFILEDIR$</dumpFileDir> <dumpDir>%ROOT_DIR%/$MODEL$</dumpDir> <diagnosticFile>%ROOT_DIR%/$MODEL$/output/diag.xml</diagnosticFile> <!--ignored since Raven logfile will be read--> <timeZone> <timeZoneName>EST</timeZoneName> <!--Needs to be non daylight-saving--> </timeZone> <startDateTimeFormat>yyyy-MM-dd hh:mm:ss</startDateTimeFormat> <modelTimeStep id="day_EST"/> <!—ensures end time is full timestep--> </general>
GA - startUpActivities
With the following settings, the relevant folders are cleared first.
<startUpActivities> <purgeActivity> <filter>%ROOT_DIR%/$MODEL$/*.*</filter> </purgeActivity> <purgeActivity> <filter>%ROOT_DIR%/$MODEL$/input/*.*</filter> </purgeActivity> <purgeActivity> <filter>%ROOT_DIR%/$MODEL$/output/*.*</filter> </purgeActivity> <purgeActivity> <filter>%ROOT_DIR%/$MODEL$/work/*.*</filter> </purgeActivity> </startUpActivities>
GA – exportStateActivity
In the exportStateActivity, the current (warm or cold) state is exported. The following example is in line with the above folder structure.
<exportStateActivity> <moduleInstanceId>Raven_$MODEL$_Historical</moduleInstanceId> <stateExportDir>%ROOT_DIR%/$MODEL$/work</stateExportDir> <loopTimeStep id="day_EST"/><!-- only when daily warm state should be generated--> <stateSelection> <warmState> <stateSearchPeriod start=”-10” end="-1" unit="day"/> </warmState> </stateSelection> </exportStateActivity>
The loopTimeStep can be used to have FEWS run the model with daily timesteps: after each daily simulation a warm state is stored in the database. So in case a simulation of 10 days is run, the user would be able to select a warm state for any of these daily timesteps as starting point for a new run. Note that this does not work well when the model is provided with state update timeseries.
The warm state selection period can be left out of the configuration if it is defined in the topology.xml or the workflowdescriptors.
GA - exportDataSetActivity
This activity exports the model itself. The model is included in a zip file in the ModuleDataSetFiles in the configuration. Generally it would include an rvi, rvp, rvt and rvh files. These files should not be in a folder in the zip file: upon extraction, the contents of the moduleDataSet will be extracted to the exportDataSetDir described in the general section.
<exportDataSetActivity> <moduleInstanceId>Raven_$MODEL$_Historical</moduleInstanceId> </exportDataSetActivity>
GA - exportNetCDFRunFileActivity
Raven uses a netCDF Run Info File to determine information about the simulation such as start and end time. Other commands can be passed to Raven via the Run Info file. Commands that should be adjustable by the user via the FEWS interface should be defined as locationAttributes. The locationAttributeModifier allows for changing these settings, which are then passed to FEWS instead. This functionality was added in Delft-FEWS release 2020.02 and is therefore not available in earlier releases.
The commands that can be passed to Raven are (see also C2.1. in the Raven user manual):
- blockRavenWarnings
- BlockRavenCustomOutput (not advised because it can lead to the workflow failing if the expected output is not present)
- NoisyMode
- SilentMode
- Mode
- AssimilateStreamFlow
<exportNetcdfRunFileActivity> <exportFile>%ROOT_DIR%/$MODEL$/run_info.nc</exportFile> <properties> <string key="RunName" value="$MODEL$"/> <locationAttribute key="BlockRavenWarnings" locationId="$MODEL$" attributeId="BlockRavenWarnings"/> </properties> </exportNetcdfRunFileActivity>
Required model inputs
Meteo input
Raven allows for different types of meteo input; per gauge, per spatial entity or gridded. FEWS exports timeseries in NetCDF format, being scalar or grid. The NetCDF files are exported to the exportDir as defined in the general section above. In Raven, the reference to this file is made in the rvt file. These different approaches are discussed below.
Gauges
Gauges are defined in the rvt and have this format:
:Gauge [gaugename]
:Latitude [latitude]
:Longitude [longitude]
Data for the forcing_types MAX_TEMPERATURE, MIN_TEMPERATURE and PRECIP need to be provided in NetCDF format (when integrated in FEWS). The definition in the rvt file is as follows (see also A.4.1 of the Raven User Manual):
:Data [forcing_type]
:ReadFromNetCDF
:FileNameNC ../input/meteo_inputs.nc :VarNameNC [forcing_type]
:DimNamesNC stations time
:PeriodEndingNC
:StationIdx 1
:EndReadFromNetCDF
:EndData
In the above example, the data is read from the first column of the matrix with data (stations x time) in the NetCDF file. An inspection of the NetCDF file is needed in order to determine the column number. A change in the FEWS configuration might lead to a reordering of columns, which might break this mapping. To have Raven look for the correct column, use the FROM_STATION_VAR command instead:
:StationIdx FROM_STATION_VAR
With this setting, Raven will select the station in the NetCDF file that corresponds with [gaugename]. This requires settings the correct name either in the idMap in FEWS, or by changing the [gaugename] to match the station Id in the NetCDF file. To map multiple locations at once, use the
:MapStationsTo SUBBASINS or
:MapStationsTo HRUS
A typical configuration of the general Adapter would look like;
<exportNetcdfActivity> <exportFile>meteo_inputs.nc</exportFile> <timeSeriesSets> <timeSeriesSet> <moduleInstanceId>$HIST_PRECIP_MODULE$</moduleInstanceId> <valueType>scalar</valueType> <parameterId>P.obs</parameterId> <locationSetId>precip_stations_$CATCHMENT$</locationSetId> <timeSeriesType>external historical</timeSeriesType> <timeStep id="day_MST"/> <relativeViewPeriod unit="day" end="$ENDTIME$"/> <readWriteMode>read only</readWriteMode> </timeSeriesSet> … </timeSeriesSets> </exportNetcdfActivity>
The element in the export IdMap would be:
<function internalLocationSet="precip_stations" internalParameter="P.obs" externalLocationFunction="@Raven_ID@" externalParameterFunction="PRECIP"/>
Grid
Some models are set-up to directly read gridded data. In the model the HRUs are mapped to grid cells. The rvt file looks like this (from A.4.7 of the Raven User Manual):
:GriddedForcing [forcing name]
:ForcingType [type]
:FileNameNC [path/filename of .nc file]
:VarNameNC [name of variable in .nc file]
:DimNamesNC x y time
:GridWeights
:NumberHRUs [total number of HRUs]
:NumberGridCells [total number of grid cells]
[HRU ID] [Cell ID] [weight]
...
:EndGridWeights
:EndGriddedForcing
A typical exportActivity in the general Adapter would be:
<exportNetcdfActivity> <exportFile>meteo_input.nc</exportFile> <timeSeriesSets> <timeSeriesSet> <moduleInstanceId>Modify_Grids_historic_$catchment$</moduleInstanceId> <valueType>grid</valueType> <parameterId>P</parameterId> <locationId>HRDPA</locationId> <timeSeriesType>external historical</timeSeriesType> <timeStep id="EST_day"/> <relativeViewPeriod unit="day" end="0"/> <readWriteMode>read only</readWriteMode> </timeSeriesSet> … </timeSeriesSets> </exportNetcdfActivity>
Polygons
In case the meteo input per HRU is calculated in FEWS, a NetCDF with multiple timeseries can be provided as input for Raven. In that case, the rvt file would have the following setting:
:StationForcing PRECIP
:ForcingType PRECIP
:FileNameNC ../input/precip_inputs.nc :VarNameNC PRECIP
:DimNamesNC stations time
:PeriodEndingNC
:RedirectToFile GriddedForcings.txt
:EndStationForcing
The redirected file (GriddedForcing.txt in above example) could look like this:
:GridWeights
:NumberHRUs 14
:NumberGridCells 1
# [HRU ID] [station ID] [weight]
22020000 0 1
22020100 1 1
….
:EndGridWeights
The above approach requires knowledge of the order of the stations in the NetCDF file. If a configuration change leads to a change in the order of the exported stations, the GridWeights settings would have to change accordingly. Therefore a direct IdMapping is possible. In the example above, the
:RedirectToFile GriddedForcings.txt
command (which points to the :GridWeights command from within a :GriddedForcing) within the :GriddedForcing block can be replaced with the following:
:MapStationsTo SUBBASINS
Or
:MapStationsTo HRUS
In the first case, Raven expects that the number of stations is equal to the number of subbbasins in the model, with the station attribute vector filled with SubBasin IDs. In the second, the number of stations must be equal to the number of HRUs in the model, with the station attribute filled with HRU IDs. This overrides the need for custom :GridWeights commands.
This method can be applied to the:StationForcing command as well.
