Note: the content of this page is outdated. There is new functionality available within the Modifier Display and IFD to perform manual calibration in standalone Delft-FEWS applications, when parameters are available in pi-xml format.
Calibration Display
The calibration display forms a special part of the forecasting systems that allow automatic calibration of any model connected to FEWS through the General Adapter. The model to calibrate must have a correctly configured General Adapter module Instance and included in a workflow. The calibration display (or utility) will be available only in the stand alone system, as it is not considered a functional part of an on-line system.
The calibration display will provide simple automatic calibration methods using either a non-derivative based local optimisation method or a more global optimisation approach (SCE-UA). While it is accepted that many alternative calibration approaches and algorithms are available, the functional scope of the module within FEWS is to provide a general purpose utility only, not necessarily encompassing all different optimisation methods. The algorithms applied are subject to research and development and the functional availability of the calibration environment may be subject to future expansion.
It is noted that for forecasting module development, the calibration utility is not seen as the primary tool with which to calibrate modules. These should first be calibrated outside FEWS in the native environments. The module within FEWS can then assess how the module perform in the linked environment. Alternatively, for modules where no native calibration environment is available, this module can be used, but it is a generic tool and therefore contains no module specific optimisation strategies.
The calibration utility will make use of performance indicators provided by the performance indicator utility, allowing comparison of calculated and observed data series.
Because of its interactive nature, the module is provided solely though a user interface plug-in that allows:
- Selection of the workflow to run during calibration. This workflow should obviously contain modules that allow calibration, though the system will not explicitly check on this.
- Selection of parameter and/or parameter sets to be calibrated.
- Selection of how module performance is to be assessed. The performance indicator module will be used to establish the performance.
- Selection of various calibration criteria, including termination criteria.
The Calibration display includes 4 tabs:
- Model to Calibrate
- Model Parameters
- Run Options
- Results
Model to Calibrate
In the first tab the user must enter all information concerning the model to calibrate. Not all models can be selected, the requirements for a calibration model run are:
- The model must allow module parameters to be exchanged through the Published Interface Exchange XML format.
- For each model the Parameter Files must be added to the ModuleParFiles configuration directory.
Currently only the Model Adapters for the MCRM & DODO models fulfil these requirements.
The required steps in the Model to Calibrate tab are:
1. Select the workflow that includes the model to calibrate from the list of available workflows.
2. When the workflow is selected the General Adapter Module Instance of the external module should be selected from the list. This General Adapter Module Instance must include configured elements that export model parameter files, adapted by the Calibration utility.
3. Select the time period to calibrate for by entering a start date and an end date
4. Select the observed time series; this is the time series that will be used to compare the model results with. It is important to select the correct module instance id that generated the observed time series (most often an import module instance) and the correct location id and parameter id.
5. Select the simulated time series that should be compared with the observed time series. The module instance id of this time series will be the module instance of the module to calibrate.
6. Enter a period for the Grace period. The model results produced in the grace period will not be included in the calibration optimisation.
7. Select the cold state group that will be used in the workflow.
Once the selection has been entered in the Model Calibration tab it can be stored to a file by pressing the [Save] button. Next time the calibration utility is used the [Load] button can be used to load the selection.
Model Parameters
The Model Parameters tab includes a spreadsheet with all model parameters that can be adjusted by the calibration utility.
The parameters shown in the spreadsheet are the model parameters of the parameter file that is selected in the upper drop down list. Once the spreadsheet has been filled with parameter values the model parameters can be selected that need to be optimised through the calibration utility. A model parameter can be selected by checking the "Adjust" column in the spreadsheet. Make sure that the original, minimum and maximum values for the selected parameter have realistic values.
Run Options tab
In the Run Options tab the final set of definitions must be set that allows the method and constraints of calibration to be defined. These include:
1. The calibration method to be used- thus may either be the Local gradient method "SuperDUD" or the global "SCE-UA" method. At the moment only the SuperDUD method is implemented.
2. The performance indicator to be used in calibration. The enumeration of this is "BIAS", "MAE", "MSE", "NS", "PEAK_MSE" and "PERCENT_VOLUME".
3. A number of termination criteria. All are available and may be selected for application or not considered:
a) Maximum number of runs.
b) Threshold change in performance
c) Threshold change in parameter vector
Results
The results tab shows the calibration results in a graph. To start the calibration utility the [Run] button must be pressed. The workflow will be started and the model parameters adjusted by the calibration utility. The user can follow the calibration progress by analysing the graph and the current iteration and current performance values located in the progress info frame.
After the calibration run is completed you can select to see a report by pressing the [Report] button. The report of the calibration run contains information on the model parameters that have been optimised in each iteration step.
By pressing [Close] the Calibration display can be closed.
Working efficiently with calibration
When working with the calibration utility a complete workflow must be run in order to optimise one of the modules in that workflow. In the calibration process this workflow will be run for multiple iterations. When the module is included in a fluvial workflow for the complete region each iteration will require a long time to complete. It is therefore recommended to create specific "calibration" workflows that only include one or several modules; the number of modules not contributing to the calibration module must be minimised. This will bring down the calculation time considerably. Especially for rainfall-runoff modules it will not take much time to create such calibration workflows.
When running the calibration utility a database must be used with a complete set of historical data. Such a database can be created using the on-line or off-line systems or by importing data from HARP.
