The DIMR-software is used to run a SOBEK 3 model in Delft-FEWS. In the General Adapter of Delft-FEWS we refer to the different components of the dimr-tool. This wiki will describe what you need to do to convert a SOBEK3 model to a dimr-model and how to set-up the different components of your Delft-FEWS configuration. Read the steps in this guide to implement your SOBEK 3 model in Delft-FEWS using the dimr-software. When updating an existing model, some steps are not necessary.
How to set-up a DIMR Modules folder?
The Delft-FEWS DIMR-adapter requires a fixed folder structure:
|sub-folder with binaries for the DIMR-software.
|sub-folder with DIMR-files that describe the SOBEK3 model.
|sub-folder with binaries for the fews adapter that communicates with the DIMR.
|sub-folder with input files requried to run the model.
|sub-folder for the log-files that are created during the model run.
Follow the steps below to set-up a DIMR model folder in your Delft-FEWS system. Step 1 can be skipped when updating an existing model.
Step 1: create the dimr model folder structure in Delft-FEWS
The contents of the Modules folder can now be filled. We will start with the getting the dimr_bin files.
Step 2: getting the contents of the dimr_bin
There are two ways for getting the dimr_bin files:
- The preferred method is getting the dimr_bin files from the used SOBEK 3 installation in which the model is created. Typically, the files are found in c:\Program Files (x86)\Deltares\SOBEK (xxx)\plugins\DeltaShell.Dimr\kernels\x64, where xxx is the SOBEK version. Copy the x64 directory to dimr_bin folder.
- Another method is to getting the dimr_bin files from the Deltares Build server which can be found using the following link. This is not preferred as these are the latest development releases of DIMR Which version you will need depends on your used SOBEK 3 model version. Which DIMR version belongs to which SOBEK 3 version is therefore currently not known. If you are sure which version to use, lick the artifacts button , select either the Windows or Linux zip file to download and unpack in the dimr_bin directory.
It is good practice to add a README.txt file in the dimr_bin directory where you document to DIMR version and from where you got the DIMR files.
The actual model files must be created by converting your SOBEK 3 model. DeltaShell contains standard functionality to aid you in this step.
Step 3: generate the dimr_model files
- In DeltaShell open your SOBEK 3 model.
It is possible that a message will appear in DeltaShell stating that your project was made in an older version and needs to be migrated to the latest version of DeltaShell. Press "Yes" and the model will automatically migrate to this version. If this is not preferred, because of new model developments or stability, use the DeltaShell which belongs to the SOBEK 3 version you used to create the model.
- Before you continue it is recommended to run the model by pressing the button in the DeltaShell GUI. This is an easy way to check if all functionality is up-to-date. Check the log messages and fix all issues that are mentioned.
When your model can finish awithout displaying errors it is possible to convert the SOBEK 3 model to a dimr-model.
- In DeltaShell right-click on your Integrated Model (e.g. ) and select the option "Export...". In the menu that opens you now select the option "DIMR configuration" and you follow the instructions.
- Put the all model files that are generated in the folder dimr_model.
The contents of the fews-dimr-adapter-bin folder can be downloaded.
Step 4: download the contents of the fews-dimr-adapter-bin folder
The DIMR and Delft-FEWS are communicating through the fews adapter. This adapter is part of Delft-FEWS and must be downloaded from the dpcbuild.deltares.nl:
- Be sure the get the adapter from the environment corresponding to your FEWS version (click on to view other version)
- Select the latest stable version (or the Delft-FEWS development version): FEWS – Development > install adapters > artifacts > fews-dimr-adapter-bin.XXXXX.zip.
- Put all these unzipped files in the folder fews-dimr-adapter-bin
The folders Input and Logs will be filled when you run the model.
How to run the DIMR model?
Run DIMR model
When the contents of every directory is set-up, we can run the DIMR model. The model can be executed using a batch file with an argument pointing to the DIMR configuration xml file.
- To run the model, execute the following batch file: .\dimr_bin\x64\dimr\scripts\run_dimr.bat DIMR_CONFIG.xml (e.g. Rhine_LobRTK.xml or IJG.xml)
You could enter this command in a seperate batch file for easier starting the model run.
How to get the desired output from the DIMR model?
Run DIMR model
In SOBEK 3 you can specify for which parameters you want to write the output. This is the same in the DIMR model and can be specified in the *.md1d file located in .\dimr_model\dflow1d.
Default, all results options are set to None. Apply the same output options as used in the SOBEK 3 model (e.g. Current, Average, ...)
How to generate cache files to read the SOBEK 3 DIMR model faster?
SOBEK 3 DIMR cache files
DIMR has the option to make cache files of the SOBEK 3 model to read the model itself faster. This can be up to 100 times faster, because it does not have to processing large ASCII files. Therefore, it is good practice to generate the cache files.
To make the cache files, do the following:
- Go to FEWS_YourSystem / Modules / NewFolder / dimr_model / dflow1d and open the *.md1d file.
- In the file, enter under \[AdvancedOption] 'CacheMode = Write' and run the DIMR model
- In the dflow1d folder the .cache files are made.
- Add these .cache files to the dflow1d folder in your ModuleDataSetFiles.
- Lastly, set CacheMode = Write to CacheMode = Read to ensure the DIMR does not generate new cache files as this is the same as not using cache files.
How to generate ColdStateFiles?
Delft-FEWS will need a state file to be able to run the SOBEK model. In order to generate the first state files of the SOBEK model it is recommended to do a stand-alone run of the DIMR from the command line (see How to run the DIMR model? how to do that).
- Go to FEWS_YourSystem / Modules / NewFolder / dimr_model / dflow1d and open the *.md1d file.
- Set the correct restart options:
- Older version:
- Go to option WriteRestart and make sure to set this setting on true. The model will now generate its state files.
- Older version:
- Newer versions:
- Under \[Restart] there are 5 options available. These reflect the restart options in SOBEK 3.
- To make a restart file, set UseRestart = 0 and WriteRestart = 1. The start and stop time should be set very wide like in the example below to ensure a restart will always be written. Intermediate restartfiles can be written using the RestartTimeStep option. Note: on the end of the simulation run, a restart will always be written if WriteRestart is set to 1 independent of the RestartTimeStep.
- Newer versions:
- The SOBEK 3 model has default timeseries. We can thus do a stand-alone run of the model without providing the model with data.
- Run the DIMR model.
or in newer versions of the software
- After the model has run succesfully it is possible to create a ColdState-file for Delft-FEWS.
- In the Config\ColdStateFiles\ create a zip for the cold state files. In our example this is the Walrus_Hydr_Sobek3_Update Default.zip
- The zip file for the ColdState files contains the following information
- Walrus_Hydr_Sobek3_Update Default \ dflow1d \ sobek.rda
- Walrus_Hydr_Sobek3_Update Default \ dflow1d \ sobek.rdf
- Walrus_Hydr_Sobek3_Update Default \ rtc \ state_import.xml
- When you analyse the model results you will find a sobek.nda and sobek.ndf file in the dimr_model \ dflow1d folder and a state_export.xml in the dimr_model \ rtc folder. You can use these files. Make sure to rename the extension of these files.
- Set UseRestart to 1 in the *.md1d file to ensure the model will be using the restart.
for newer versions of the software
What information should I put in the ModuleDataSetFiles?
Delft-FEWS can distribute model files to FSS machines. It is recommended to zip the model files and put it in the Delft-FEWS configuration.
- In the Config\ModuleDataSetFiles create a zip file for the (update) model run. In our example the name for this zip file is Walrus_Hydr_Sobek3_Update.zip
- The zip file for the ModuleDataSetFile contains the following information:
- DIMR_CONFIG_NAME.xml (e.g. Rhine_LobRTK.xml or IJG.xml)
- Be sure to exclude .log, .rda, .rdf, .nda, .ndf, .\dlfow1d\ouput dir, state_import.xml and state_export.xml
- As you might have notice this is the content of the dimr_model file created previously.
How do I get hold of the import and export timeseries of the SOBEK 3 / DIMR model?
Not yet completed
How to create the General Adapter Module?
For more detailed information on the use of the General Adapter Module, see this link.
An example of the general section of the general adapter can be found below.
General section of Module
In the example configuration above it can be seen that the working directory of the DIMR should always be the folder ./dimr_model. The Delft-FEWS exports timeseries to the folder .\Input. After the modelrun is completed the output of the model is expected in the folder .\dflow1d/output.
The activities section is divided into four sub-sections: startUp-, export-, execute- and importActivities. The contents of all of these sections will be discussed in the info sections below. When you want to test your configuration during the building process then it is recommended to start with the startUpActivity, ExportActivities and ExecuteActivity sections. When no information (e.g. timeseries, states) are provided to the model DIMR will take its own default timeseries and states to perform a model run. Hence, this trick enables you to test if the model can be run from Delft-FEWS. The importDir can be checked to see if output has been generated.
Similarly, it is also possible to define a single input timeseries in the exportActivities. The DIMR will add default values to the missing timeseries in order to start the run. This trick allows you to check early on whether your ExportActivity and IdMapping is working properly.
The startUpActivities are typically used to purge the model directory in the Modules folder to ensure a clean slate. See example below
The exportActivities are used to export all the necessary data and the model itself. These typically consists of (in order): exportStateActivity, exportTimeSeriesActivities, exportDataSetActivity and exportRunFileActivity. See examples below.
This activity exports the state of the model. This can be a cold state (the one generated under How to generate ColdStateFiles?) or a warm state saved in the Delft-FEWS database.
This activity exports all the necessary data from FEWS to be used in the model. There can be multiple activities.
This activity exports the ModuleDataSet file to the working directory.
This activity makes a run xml file which the pre- and post adapter uses to distribute the timeseries to the correct locations for the model to pick up.
Improved DIMR adapter
The new DimrPreAdapter needs to be called with as only argument a reference to the run file.
Diagnostics can be ignored because it will write a plain text file dimr_pre_adapter.log in the workDir.
In the pi run file property with key="dimrConfigFile" must be present with a reference to the dimr configuration file. This reference must be relative to the parent directory of the pi run file.
If multiple pi time series files are exported, it must be specified which file is meant for which component, there are 3 different properties for the components:
The time series xml files must be numbered in the order they appear in the <inputTimeSeriesFile> elements
Important: the adapter doesn't support timezones. Therefore, the timezone in the General Adapter should be set at +00:00 and the timezones in the model too. This means for example for D-Flow FM that in the mdu file the Tzone property is set to 0.
Important: if the exported timeseries from the GA are exceeding the T0 (like in case of RR is often the case with daily evaporation timeseries), use the <ignoreRunPeriod> option in the export timeseries activity to ignore the timeseries that exceed the T0. This ensures that the modelstate is imported into FEWS on the proper time (in most cases T0) instead of the last timestamp. See also 05 General Adapter Module - DELFT-FEWS Documentation - Deltares Public Wiki
The dimr configuration file will be read to find the components that are being used so it knows which files need to be changed.
Also when the <time> element is present within <control> <parallel> <startGroup> its contents will be replaced with the appropriate values.
Will be used if library "FBCTools_BMI" is configured as component within the dimr config file
It will look for the hardcoded file name rtcRuntimeConfig.xml in working directory relative to the parent dir of dimr config file.
It will replace start date and end date in rtcRuntimeConfig.xml with the appropriate values.
It will look for the hardcoded file name timeseries_import.xml in the working dir and insert all timeseries from the time series file configured by "piFileForFbc" (or the first time series xml if there is only 1) by matching headers.
Will be used if library "dflowfm" is configured as component within the dimr config file
It will look for the .mdu file in working directory relative to the parent dir of dimr config file.
It will read "ExtForceFileNew" value and "ExtForceFile" specifically for wind forcing.
It will look for .ext file relative to .mdu file parent directory and extract all "forcingfile" properties from it.
Insert all timeseries from "piFileForFm" (or the first time series xml if there is only 1) into all .bc files (relative to .ext file parent dir) by matching headers.
Since 2020.02, it also parses the [lateral] section in the .mdu file, uses the value of the Type property to find the .bc file (in the above case the value of Type is discharge, and discharge points to the Maas_D4118_lat.bc file). The times series with locationIds which match the Id property are inserted in the identified .bc file.
Other parameters like wind, precipitation and atmospheric pressure can be supplied by FEWS only for 2D using a netcdf export directly to the file-location the model expects the nc-file to be. In the model you can use the old-style ext format file (in mdu keyword ExtForceFile) to use these netcdf-files. How to do this can be found in the D-Flow FM manual.
Since 2022.02 there is support for windxy tim files. Only 1 timeseries for windspeed and 1 for winddirection are supported in the pi xml file. The adapter looks at the parameterIds "Windspeed" and "Winddirection" and replaces the whole tim file with the values from these timeseries including new timesteps referenced to the refdate in the mdu file. It is advised to set METHOD=7 in the .ext file for wind timeseries.
In the .mdu file itself it will replace Tstart, Tstop and RstInterval with the appropriate values.
RstInterval can be overwritten by a property in the run info file called "restartIntervalForFm"
Will be used if library "rr_dll" is configured as component within the dimr config file:
It will look in the working dir for DELFT_3B.INI and will update the StartTime and EndTime variables of the [TimeSettings] section according to the start and end time from the run info file. Thus, the content would look like the following extracted example:
Precipitation, evapotranspiration and temperature can be supplied using netcdf-files exported from FEWS with the proper IDs related to the catchmentname/weatherstation in the model. In order to use the netcdf-file data in the model, you have to change both the DELFT_3B.ini and the sobek_3b.fnm files. See code snippets below.
Important: notice that the above snippet starts at number 125. You need the lines till 125 in the sobek_3b.fnm file for it to work. If you are missing the lines for numbers 121 till 124, see the snippet below to add them. Another very important step after adding the lines is to make an empty file in the rr model directory called CacheFile.txt. If this file doesn't exists, this could lead the model to use infinity precipitation values!
DFlow1D component (Sobek 3)
Will be used if library "cf_dll" is configured as component within the dimr config file
It will look for the .md1d file in working directory relative to the parent dir of dimr config file.
It will replace StartTime and StopTime with the appropriate values.
It will look for "boundCondFile" to find .bc file (relative to .md1d file parent dir) and insert all timeseries from "piFileForFlow1d" (or the first time series xml if there is only 1) by matching headers.
Will be used if library "flow1d2d" is configured
It will look for the .ini file in working directory relative to the parent dir of dimr config file.
It will extract .md1d file (with directory and model definition file relative to parent dir of the .ini file) by looking for model = Flow1D
It will extract .mdu file (with directory and model definition file relative to parent dir of the .ini file) by looking for model = FlowFM
It will process the .md1d and .mdu file the same way as described in DFlow1D and DFlowFM component
Will be used if library "wave" is configured
It will look for the .mdw file in working directory relative to the parent dir of dimr config file.
At this point the DIMR adapter only performs a conversion of the boundary conditions NetCDF file to ASCII format. The relative path of the netCDF file to convert must be provided in the PI runinfo file using the dWaveNetcdfBoundaryFileToConvert property as shown in the example runinfo file above. Two additional properties are provided, to summarize:
- dWaveNetcdfBoundaryFileToConvert = relative path of the boundary conditions netCdf file
- dWaveBoundaryFilePrefix = prefix for the filenames of the output ASCII files, the default prefix is "WW3_"
- dWaveBoundaryFileNumberOfCopies = the number of file copies to provide for each boundary, the default number is 3
Installing and running the Delft3D-FM model on Windows and Linux
A Delft3D-FM model can be run on Windows and LINUX. Installation of the model binaries on Windows and Linux can be straight forward when simple models are run on single VM's. A typical Delft-FEWS configuration solution can be the following.
- Create a ModuleDataSet "D3D_bin_Windows.zip" with the Windows binaries
- Create a ModuleDataSet "D3D_bin_Linux.zip" with the Linux binaries
- Create a ModuleDataSet "D3D_bin_adapter.zip" with the Delft3D Dimr Java Adapter
- In your fss_windows_clientConfig.xml file add a line: <autoExportModuleDataSet name="D3D_bin_Windows" exportDir="Modules"/>
- In your fss_linux_clientConfig.xml file add a line: <autoExportModuleDataSet name="D3D_bin_Linux" exportDir="Modules"/>
- In your Windows and Linux clientConfig.xml file add a line: <autoExportModuleDataSet name="D3D_bin_adapter" exportDir="Modules"/>
- In your Delft-FEWS General Adapter executeActivity element, start the Delft3D model with the following command <executable>%REGION_HOME%/Modules/Delft3DFM/bin_delft3dfm/$D3D_MODEL_EXE$</executable>
- In your fss_linux_global.properties add a line: D3D_MODEL_EXE=/bin/run_dimr.sh
- In your fss_windows_global.properties add a line: D3D_MODEL_EXE=dimr\\scripts\\run_dimr.bat
When a Delft-FEWS FSS is started the clientConfig.xml file automatically unzips the correct Delft3D-FM binaries to your \Modules folder ( \Delft3DFM). When the Delft3D-FM model is started in a workflow from the General Adapter the correct script is started to run the DIMR.