Since 2014.01 |
This is an overview of some of the requirements for developing a Delft-FEWS compliant adapter. This list is by no means exhaustive so before starting to develop an adapter please contact fews.support@deltares.nl |
A model adapter is the interface between a so-called model and the Delft-FEWS system. It enables FEWS to run such a model, thus providing the essential forecasting functionality. For each particular FEWS application applying a model, however, some aspects of this system have to be configured by the user in order for the system to work correctly. To achieve this, it is useful that the user has at least some basic understanding of the relation between Delft-FEWS, the model adapter and the forecasting model. In this section, a brief overview of this relation is provided. For more information, the user is referred to the FEWS manual: General Adapter..
Also make sure to use the following new configuration options that have been added to the FEWS General Adapter recently. Use these options for handling log files and state files and writing placeholder files, so that model adapters do not need to do those things themselves:
Since FEWS 2014.01 it is possible to use only NetCDF files (no-xml) for the communication between FEWS and the model adapter. Good libraries for reading and writing NetCDF files exist for several languages. XML files are a lot more complex to read and to write.
NetCDF time series and attributes can be read with just a few lines of source code.
For more information about the NetCDF format in general, see NetCDF formats that can be imported in Delft-FEWS and the links on that page.
If you create a model adapter in one of these languages, you can use the model adapter within FEWS in the following way:
To inspect the NetCDF files there are several viewers available.
They are all free and all have their strengths
Panoply : http://www.giss.nasa.gov/tools/panoply/
ncBrowse: http://www.epic.noaa.gov/java/ncBrowse
intel array visualizer: http://software.intel.com/en-us/articles/intel-array-visualizer
For NetCDF there is also a text representation available.http://www.unidata.ucar.edu/software/netcdf/docs/netcdf/CDL-Syntax.html
This is very useful for debugging, documentation and as input/output for unit tests
Converting from NetCDF to CDL
ncdump: https://www.unidata.ucar.edu/software/netcdf/docs/netcdf/ncdump.html
Converting from CDL to NetCDF
ncgen: http://www.unidata.ucar.edu/software/netcdf/docs/netcdf/ncgen.html
Figure 1: schematic interaction between the FEWS and an external module
A pre-adapter is converting the NetCDF files exported by FEWS to input files for the model.
A pre- adapter is also adjusting the start and end times in the model configuration files so the model will run the right number of time steps .
The post-adapter is converting the output of the model to NetCDF files that are FEWS compliant and can be imported by FEWS.
Not required, but recommended to use a pre- and post-adapter. It is also possible to wrap the model. This wrapper is a pre- and post-adapter in one and it also starts the model and waits for the simulation to finish.
The NetCDF run file contains metadata about the run that has to be executed. The run file is every time re-generated by FEWS before the adapters and models are executed. The most important information is the start and end time of the simulation. The run file can contain extra configuration options as properties. The properties are key-value pairs that are configured in FEWS and added to the run file when the file is generated by FEWS.
Example of the PI run file by ncdump.
netcdf run_info {
dimensions:
path_length = 255 ;
log_level_length = 5;
input_netcdf_file_count = 2 ;
input_state_file_count = 1 ;
output_netcdf_file_count = 2 ;
variables:
char log_level(log_level_length) ;
double start_time ;
start_time:long_name = "start_time" ;
start_time:standard_name = "time" ;
start_time:units = "minutes since 2002-11-26 00:00:00.0 +0000" ;
double end_time ;
end_time:long_name = "end_time" ;
end_time:standard_name = "time" ;
end_time:units = "minutes since 2002-11-26 00:00:00.0 +0000" ;
char work_dir(path_length) ;
char input_netcdf_files(input_netcdf_file_count, path_length) ;
char input_state_files(input_state_file_count, path_length) ;
char output_netcdf_files(output_netcdf_file_count, path_length) ;
char properties ;
properties:string_property = "zs_0 with spaces" ;
properties:INT_property = 3 ;
properties:float_PROPERTY = 3.5f ;
properties:DOUBLE_PROPERTY_321 = 0.123456789 ;
properties:logical_property_1 = "true" ;
properties:logical_property_2 = "false" ;
// global attributes:
:title = "Run file" ;
:institution = "Deltares" ;
:source = "Run file from Delft-FEWS" ;
:history = "2014-03-18 12:22:55 GMT: exported from Delft-FEWS to D:\\fews\\workspace\\fews\\junit_test_output\\nl\\wldelft\\fews\\system\\plugin\\generaladapter\\exportDir\\run_info.nc" ;
:references = "http://www.delft-fews.com" ;
data:
logLevel = "debug";
start_time = 0 ;
end_time = 5880 ;
work_dir = "..\\work" ;
input_netcdf_files =
"..\\work\\boundary_data.nc",
"..\\work\\more_boundary_data.nc" ;
input_state_files =
"..\\input_state\\state.inp" ;
output_netcdf_files =
"..\\work\\model_output_data1.nc",
"..\\work\\model_output_data2.nc" ;
properties = " " ;
}
|
Adapters and models should write one or more log files. It is easiest to let them be plain text files. Log message should be recognizable as debug, info, warn or error by a pattern. In FEWS this pattern (for example “ERROR:*”) is configured. FEWS can read those lines and map them to a log level of choice: GeneralAdapterModule-logFile
When FEWS finds an error log line the simulation will be marked as failed.
FEWS can create empty output time series so called placeholder files. The place holder files are CF compliant and contains all the variables, dimension and locations FEWS expects but with zero time steps. The post adapter can easily add the time steps without bothering about CF compliance. As an optimization the pre-adapter can adjust the model configuration so the model only creates output for the locations and variables FEWS wants to import.
A model state makes it possible to continue a run later. The state files that are exported by FEWS and will be imported after the simulation are listed in the run. When the model state contains time information it is sometimes desired that the pre-adapter adjust this time to allow the states to be used for a different time to reduce the warming up period of a simulation. The pre-adapter can adjust the model configuration to switch on/off the generation of a state file at the end of a run. The pre-adapter knows if states are imported after the simulation by querying the “output_netcdf_files” variable in the run file.
The pre-adapter and post adapter don’t have to bother about translating the FEWS location ids and parameter ids to model location and variable ids. This mapping can be configured in FEWS. When the CF standard names are configured for the FEWS parameters the standard names will also be written to the exported NetCDF files and recognized while importing. FEWS is also exporting the location coordinates to the NetCDF input and place holder files. Some adapters choose to use the coordinates and/or the standard names for the mapping.