MGB adapter

Contents

This page documents the Delft-FEWS MGB-SA adapter required for running MGB-SA models from Delft-FEWS. The adapter supports MGB-SA model version 1.0.

MGB-SA is a continental-scale, hydrological hydrodynamic model for South America (hereafter named as MGB-SA). This model was primarily designed for the South American Flood Awareness System (SAFAS) project and has been continuously developed in the context of SAMEWater initiative (Paiva et al., 2017), which refers to a research agenda focused in a plethora of applications regarding continental water resources like understanding of hydrological 
processes, reservoir operation, water uses, climate change, hydrological monitoring and forecasting, sediment transport and data assimilation. 

The adapter consists of separate pre- and post adapters that can be used to import and export timeseries data to/from the MGB-SA model and control the simulation period. The FEWS general adapter is used to configure both the pre- and post adapter and execute the MGB-SA simulation using the MGB command line executable. This model adapter is written in java. The main functionality of the adapter is to read / write the binary files of MGB-SA in order to input precipitation data from FEWS into the MGB-SA model and to transfer output flow data from MGB-SA back to FEWS. The adapter currently does not provide state handling so the model can only run with a cold start.

Note: there is an earlier version adapter developed by the company LACTEC in 2013 for an earlier version of the MGB model, but there have been various changes to the model file structure of MGB-SA 1.0 so it was decided to create a new adapter which makes more efficient use of Delft-FEWS support libraries and supports the NetCdf file format for more efficient transfer of large datasets.

MGB pre-adapter

The MGB pre-adapter converts the NetCdf or pi-xml files from Delft-FEWS to model native input files.

Usage: MgbPreAdapter <PI XML run file pathname relative to current working directory>

Class name: nl.deltares.fews.mgb.MgbPreAdapter

Properties

Notes for users

• For all files that are written by this adapter, if the file to be written already exists, then it will be overwritten.
• This program can process the binary input / output files of MGB-SA as a whole, partial updates of only a subset of the data in these files is not possible.
• This program assumes that the MGB-SA model configuration uses the same time zone as given in the PI XML run file.
• The working directory (workDir) defined in the PI XML run file is the directory where the MGB-SA model will be run, the MGB input and output files will be stored in this directory.
• The default file names and location of the MGB input and output files can be altered using (optional) properties in the PI XML run file
• This program writes log messages to a PI-XML diagnostics file specified in the PI XML run file. This will be picked up by FEWS general adapter automatically.
• This program uses the information in the specified PI XML run file as input and uses this information to perform the following actions:
  
  1. Update the start time and simulation duration for the simulation in the infoMGB.sim file, which is in text format
     The default path and file name of the infoMGB.sim input file (*.inp) file can be specified in the (optional) property "infoMgb" in the PI XML run file.  The adapter will only modify the DAY, MONTH, YEAR, HOUR fields of the start date/time and  NT to indicate the number of timesteps. The length of the timestep of the model NT will not be modified and defaults to one day
  2. Convert input NetCDF (or PI XML timeseries files) to MGB binary input file format files, currently this is used only for precipitation, so only one InputTimeSeries field should be specified in the run info file.
     please note that the binary inputs of MGB contain only raw data without any location indexing. Therefore special care needs to be taken that ALL locations are in the FEWS InputTimeSeries and that these are in the same order, the adapter currently checks that the locations exported by FEWS all have a sequential location id numbered for 1..N, it will NOT reorder the data, as this does not seem to be necessary at this point.

PI XMl run file example

<?xml version="1.0" encoding="UTF-8"?>
<Run xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.wldelft.nl/fews/PI" xsi:schemaLocation="http://www.wldelft.nl/fews/PI http://fews.wldelft.nl/schemas/version1.0/pi-schemas/pi_run.xsd" version="1.5">
    <timeZone>0.0</timeZone>
    <startDateTime date="2022-07-24" time="00:00:00"/>
    <endDateTime date="2022-08-04" time="00:00:00"/>
    <time0 date="2022-07-24" time="21:00:00"/>
    <lastObservationDateTime date="2022-08-04" time="00:00:00"/>
    <workDir>.\work</workDir>
    <inputTimeSeriesFile>.\input\timeseries_input_mb.nc</inputTimeSeriesFile>
    <outputDiagnosticFile>.\diag.xml</outputDiagnosticFile>
    <outputTimeSeriesFile>.\output\timeseries_output.nc</outputTimeSeriesFile>
    <properties>
        <string key="infoMgb" value="input/infoMGB.sim" />
        <string key="P_input" value="input/Chuvabin.pbi" />
        <string key="Q_output" value="output/QTUDO.MGB" />
    </properties>
</Run>

note: the path of the InfoMGB.sim and Chuvabin.pbi model files in this case are not in the default "work" directory but in a subdirectory "work/input" as specified in the "infoMGB" and "P_input" properties. The paths in the run info file can be relative paths to the "work" folder as shown in this example.

MGB post-adapter

The MGB post-adapter converts the native model output (flows) to NetCdf (or pi-xml) files to be imported by Delft-FEWS.

Usage: MgbPostAdapter <PI XML run file pathname relative to current working directory>

Class name: nl.deltares.fews.mgb.MgbPostAdapter

Properties

Notes for users

• For all files that are written by this adapter, if the file to be written already exists, then it will be overwritten.
• This program assumes that the MGB-SA model configuration uses the same time zone as given in the PI XML run file.
• The working directory (workDir) defined in the PI XML run file is the directory where the MGB-SA model will be run, the MGB input and output files will be stored in this directory.
• The default file names and location of the MGB input and output files can be altered using (optional) properties in the PI XML run file
• This program writes log messages to a PI-XML diagnostics file specified in the PI XML run file. This will be picked up by FEWS general adapter automatically.
• This program uses the same PI XML run file as the pre adapter and uses this information to perform the following actions:
  
  1. Read the infoMGB.sim file, to determine the actual run period and timestep used by the model, and checks that this is compatible with the information in the run info file.
     The default path and file name of the infoMGB.sim input file (*.inp) file can be specified in the (optional) property "infoMgb" in the PI XML run file. 
  2. Convert the MGB binary output file to NetCDF (or PI XML) timeseries files, currently this is used only for flow data, so only one OutputTimeSeries field should be specified in the run info file.
     please note that the binary inputs of MGB contain only raw data without any location indexing. The resulting input for FEWS will contain location-id's numbered from 1 to N

System requirements

• This program needs Java version 11 or higher.
• The native libraries included will work on Windows but not on Linux
• This program needs the following Java/native libraries:
  • castor-0.9.5p.jar
  • Delft_NetCDF_Util.jar
  • Delft_PI.jar
  • Delft_PI_castor.jar
  • Delft_Util.jar
  • FastInfoset-1.2.6.jar
  • joda-time-2.10.jar
  • log4j-1.2-api-2.17.1.jar
  • log4j-api-2.17.1.jar
  • log4j-core-2.17.1.jar
  • log4j-slf4j-impl-2.17.1.jar
  • netcdf-4.6.3p.jar
  • slf4j-api-1.7.13.jar
  • xercesImpl.jar
  • crl_x64.dll
  • hhl_x64.dll
  • hx64.dll
  • ms120x64.dll
  • nc_x64.dll
  • z_x64.dll

GeneralAdapterRun Example Configuration

The following gives an example of how to set up the GeneralAdapterRun file for MGB-SA in FEWS using the MGB model pre and post-adapters.  The GeneralAdapterRun file follows the general structure as described here.

General

In this section general information regarding the module such as version number, file directories, missing values, and time zone information can be specified.

<general>
	<description>MGB Forecast Template</description>
	<rootDir>$REGION_HOME$/Modules/MGB</rootDir>
	<workDir>%ROOT_DIR%/work</workDir>
	<exportDir>%ROOT_DIR%/input</exportDir>
	<exportDataSetDir>%ROOT_DIR%</exportDataSetDir>
	<exportIdMap>IdExportMGB</exportIdMap>
	<importDir>%ROOT_DIR%/output</importDir>
	<importIdMap>IdImportMGB</importIdMap>
	<dumpFileDir>$DUMP_FOLDER$</dumpFileDir>
	<dumpDir>%ROOT_DIR%</dumpDir>
	<diagnosticFile>%ROOT_DIR%/diag.xml</diagnosticFile>
</general>

Start-up activities

It may be useful to clear the model working directory of any previous runs before starting a new run.

<startUpActivities>
	<purgeActivity>
		<filter>%ROOT_DIR%/diag.xml</filter>
	</purgeActivity>
	<purgeActivity>
		<filter>%ROOT_DIR%/input/*.*</filter>
	</purgeActivity>
	<purgeActivity>
		<filter>%ROOT_DIR%/output/*.*</filter>
	</purgeActivity>
</startUpActivities>

Export activities

In this section the data to be exported from FEWS as input to the module is specified. Data to export to MGB generally includes:

• Input timeseries data (currently only precipitation)
• Run file (i.e start and end time for simulation)

The run file contains information regarding the input file names, start and stop times, and time step. Additional properties can be passed using the run file as listed above under Properties.

<exportActivities>
	<exportNetcdfActivity>
		<exportFile>%ROOT_DIR%/input/timeseries_input_mb.nc</exportFile>
		<netcdfFormat>netcdf4</netcdfFormat>
		<timeSeriesSets>
			<timeSeriesSet>
				<moduleInstanceId>Preprocess_MGB_Forecast</moduleInstanceId>
				<valueType>scalar</valueType>
				<parameterId>P.pro</parameterId>
				<locationSetId>Basins_MGB</locationSetId>
				<timeSeriesType>simulated forecasting</timeSeriesType>
				<timeStep unit="day" multiplier="1"/>
				<relativeViewPeriod unit="day" start="0" startOverrulable="true" end="282" endOverrulable="true"/>
				<readWriteMode>add originals</readWriteMode>
			</timeSeriesSet>
		</timeSeriesSets>
	</exportNetcdfActivity>
	<exportRunFileActivity>
	<exportFile>runinfo.xml</exportFile>
	    <properties>
    	    <string key="infoMgb" value="input/infoMGB.sim" />
        	<string key="P_input" value="input/Chuvabin.pbi" />
	        <string key="Q_output" value="output/QTUDO.MGB" />
	    </properties>
	</exportRunFileActivity>
 </exportActivities>

Execute activities

This section calls the SWMM pre and post-adapters as well as the SWMM executable. Note: the run file must be passed as an argument to the SWMM pre and post-adapters.

<executeActivities>
	<executeActivity>
		<command>
			<className>nl.deltares.fews.mgb.MgbPreAdapter</className>
			<binDir>%ROOT_DIR%/bin_adapter</binDir>
		</command>
		<arguments>
			<argument>%ROOT_DIR%/input/runinfo.xml</argument>
		</arguments>
		<timeOut>90000</timeOut>
		<waitForOtherRun>true</waitForOtherRun>
	</executeActivity>
	<executeActivity>
	   <description>Run SWMM</description>
	   <command>
	      <executable>$MGB_BIN$/mgb.exe</executable>
	   </command>
	   <timeOut>3600000</timeOut>
	</executeActivity>
	<executeActivity>
		<command>
			<className>nl.deltares.fews.mgb.MgbPostAdapter</className>
			<binDir>%ROOT_DIR%/bin_adapter</binDir>
		</command>
		<arguments>
			<argument>%ROOT_DIR%/input/runinfo.xml</argument>
		</arguments>
		<timeOut>90000</timeOut>
		<waitForOtherRun>true</waitForOtherRun>
	</executeActivity>
</executeActivities>

Import activities

In this section the data to be imported into FEWS as output from the module is specified. Data to import from SWMM generally includes:

• Output flow data
  
  

<importActivities>
	<importNetcdfActivity>
		<exportPlaceholderFile>true</exportPlaceholderFile>
		<importFile>%ROOT_DIR%/output/timeseries_output.nc</importFile>
		<timeSeriesSets>
			<timeSeriesSet>
				<moduleInstanceId>$MODULE_INSTANCE_ID$</moduleInstanceId>
				<valueType>scalar</valueType>
				<parameterId>Q.pro</parameterId>
				<locationSetId>Basins_MGB</locationSetId>
				<timeSeriesType>simulated forecasting</timeSeriesType>
				<timeStep unit="day"/>
				<readWriteMode>read only</readWriteMode>
			</timeSeriesSet>
		</timeSeriesSets>
	</importNetcdfActivity>
</importActivities>

Please note that the <exportPlaceholderFile>true</exportPlaceholderFile> option is required by the post adapter !

PI XML Run file example

Typically, the general adapter needs to export only one PI XML run file for use by both adapters, as in the below example:

<?xml version="1.0" encoding="UTF-8"?>
<Run xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.wldelft.nl/fews/PI" xsi:schemaLocation="http://www.wldelft.nl/fews/PI http://fews.wldelft.nl/schemas/version1.0/pi-schemas/pi_run.xsd" version="1.5">
    <timeZone>0.0</timeZone>
    <startDateTime date="2022-07-24" time="00:00:00"/>
    <endDateTime date="2022-08-04" time="00:00:00"/>
    <time0 date="2022-07-24" time="21:00:00"/>
    <lastObservationDateTime date="2022-08-04" time="00:00:00"/>
    <workDir>.\work</workDir>
    <inputTimeSeriesFile>.\input\timeseries_input_mb.nc</inputTimeSeriesFile>
    <outputDiagnosticFile>.\diag.xml</outputDiagnosticFile>
    <outputTimeSeriesFile>.\output\timeseries_output.nc</outputTimeSeriesFile>
    <properties>
        <string key="infoMgb" value="input/infoMGB.sim" />
        <string key="P_input" value="input/Chuvabin.pbi" />
        <string key="Q_output" value="output/QTUDO.MGB" />
    </properties>
</Run>