Introduction
Since 2021.01 auto calibration has been introduced for multiple module parameter modifiers.
By specifying observed and simulated time series FEWS can calibrate workflows via module parameter modifiers by letting the external software package OpenDA calculate the error between the observed and simulated time series and suggest new parameters for FEWS to rerun the workflow. This will continue until the error is minimized, a minimized error will be a sign of optimal parameters. The calibration will apply to the same workflow that is connected to the IFD node. For this to work, the OpenDA_bin.zip should be downloaded and unzipped in %REGION_HOME%/Modules/OpenDA, FEWS will automatically look for the OpenDA binaries in %REGION_HOME%/Modules/OpenDA/bin.
Configuration
Auto calibration will be activated via the ModifierTypes.xml
Via the elements <calibrationObservationTimeSeries> and <calibrationSimulationTimeSeries> time series will be filtered out of the display groups connnected to the selected IFD node.
By specifying these elements auto calibration buttons will become enabled.
<useAttributeModifiers>true</useAttributeModifiers> <calibrationObservationTimeSeries> <moduleInstanceId>ImportDataCard</moduleInstanceId> <parameterId>QME</parameterId> <timeStep unit="day"/> </calibrationObservationTimeSeries> <calibrationSimulationTimeSeries> <moduleInstanceSetId>STATQME_Calibration</moduleInstanceSetId> <parameterId>SQME</parameterId> <locationSetId>Basins_Kuskokwim</locationSetId> <timeStep unit="day"/> </calibrationSimulationTimeSeries> </multipleModuleParameterModifier>
Multiple module parameter modifiers panel
When auto calibration time series have been configured extra buttons will appear in the multiple module parameter modifiers panel:
Inside each modifiable decimal number column an "AC" button appears which wil open up a small window in which the parameter can be included in auto calibration and the upper and lower boundaries should be specified. The auto calibration will keep the auto calibrated values between those boundaries. When present the initial values will be taken from the user settings for this parameter. When they are not available, it will take the <minimumValue> and <maximumValue> from the <numberParameter id="XXX"> from the modifierTypes as Lower and Upper boundaries. When the user changes those values they will be stored in the user settings for next time.
Parameters included in auto calibration will have the "AC" in bold font on its button.
When all parameters are set according to plan, the "Auto calibrate" can be pressed to setup and start the auto calibration run.
The workflow start time can be chosen as well as the period for which the observed and simulated time series should be used on which the workflow should be optimized.
"Min improvement percentage", "Shuffling loops for min improvement" and "Number of complexes" and "Maximum number of evaluations" are specific settings for the SCE search algorithm used by OpenDA.
When the SCE algorithm does not manage to decrease the cost (error) by "Min improvement percentage" in a certain number of "Shuffling loops" the algorithm will stop and report back the best set of parameters.
The "Maximum number of evaluations" should at least be set higher than "numberOfComplexes * (2 * numberOfParams + 1)" in order for the algorithm to pass initialization and being able to report the best set of parameters back to FEWS.
OpenDA Debug option can be used by FEWS-OpenDA developers to connect OpenDA from their own source code, this helps to debug OpenDA. This option will not be visible for regular users.
At the start of an auto calibration run the OpenDA version will be logged so the user can provide in for debugging.
RMSE_iterations.csv
When the auto calibration is done there will be a RMSE_iterations.csv in the Modules\OpenDA\runDir\ which will contain all the different parameters and costs of all iterations.
Iteration RMSE SCF_KLCA2UPR
1 197.7554942918825 1.62058
2 202.47525431618783 0.9355220612224797
3 199.22431011695718 1.287716146636049
4 205.032788011499 0.09238746978143175
5 197.99980480800866 1.847140935891086
6 206.50347361209896 0.47016513526707504
7 198.3726319341696 1.9534438533639509
8 197.9243771706116 1.7870119266819753
9 198.3042976913001 1.4541480733180245
10 197.97311496998975 1.537364036659012
11 198.53417791856666 1.3940190641089139
12 198.10908582326743 1.5072995320544567
-1 197.7554942918825 1.62058
Example
In the picture above of the multiple module parameter modifiers panel, the "SCF" parameter is included for both "KLCALWR" and "KLCAUPR" locations, these will be treated as 2 separate auto calibration parameters.
When looking at the observed (black) and simulated (red) time series the peak between 05-21-2013 and 06-13-2013 does not match very well with the current "SCF" values of "2" for both locations.
After executing an auto calibration run for this specific period and giving upper and lower boundaries for the SCF of 3.0 and 0.0, the auto calibration came to the next results of 1.08 for SCF of KLCA2LWR and 2.215 for SCF of KLCA2UPR:
This resulted in the next graph where the (red) simulated time series are a lot closer to the (black) observed time series for the peak period which the auto calibration optimized on:
