Unable to render {include} The included page could not be found.

Warning

This page contains only ideas (brainstorming material). Some of the ideas were already implemented in DelftTools / DelftShell.

Introduction

The main goal of this activity is to design and develop a framework which will enable to construct a user-friendly applications to work with different models.

TODOs

Define dictionary of all model-/modelling system-related terms.
Define an architecture, provide components/packages diagram.
Define concept of the Project, Task, Model, Data objects used to store Project and Models data.
Define concept of data management classes like Quantity, Unit, TimeSeries, Grid or ElementSet...
Design generic persistency/data access layer.
Define concept of the models integration.
Define concept of the GUI (Model-View-Controller, event-driven, plugin-based).

Design

Typical structure of the modelling system would need at least some framework class library which will define all classes to work with different data. Such framework will also provide an IModelinterface. It will be required to implement that interface by any model which will need to be integrated into the system. Such an interface has to be designed to be as simple as possible and should not contain anything else than mapping of the model data from/to the model engine and providing control functions to run the model.

Core

Project

A main working document of the user.
Is a main repository mainly containing Data and Tasks (Models).
Contains a set of data (measurements data, etc.) which may be used across the project.
Contains a set of spatial data / maps.

A typical project may look like:

Project1/
  Data/                 - any project can have any set of data which are used across the project
     DataFolder1/       - the data can be orginized freely using folders
        time series1    - but at the same time they represent actual data and not the files
        elevation2      - e.g.: rating_curve1, measured_concentration_of_pecticides_Station1,...
        rating curve1
        ...
     DataFolder2/
     ...
  Geometry/             - geometry-related data (maps) which can be used across the project
     Rhine/             - folders are used to orginize layers in the same way as in case of Data
        water line (polyline)
        bridges (point)
     North Sea/
        water body (polygon)
        ...
     ...
  Folder1/
     Folder2/           
        Task1/            - a manager of the models which can be linked with each other
          Model1/
            input data1   - any data types provided by the model
            output data1  
            ...
          Model2/
            input data1   - any data types provided by the model
            output data1  
            ... 
          ...
        Task2/
          Model1/
            input data1
            output data2
            output data1
            ...
         ...
     ...

Notes:
1. Probably all groups Data, Maps and Models can be put into a separate view/tabs (like classes, files, favorites in any IDE, IntelliJ, Visual Studio).
2. Probably when system will run as embedded into ArcGIS - Maps tab will not be used at all, because it will be also provided by ArcGIS

A typical use of the system (use case) may look like:

Start system GUI
Open Project to work with or create a new one.
Import some data into the project from external files or create/edit a new one.
Create Task to simulate some situation
- Add different models to task, each model can be added many times, for example to simulate different domains, schematisations.
- Modify different model input data using available editors such as:
  - Schematisation editor based on map.
  - Time series editors
  - Model parameter / attribute editors
  - Other model data such as initial, boundary conditions.
- Link different models together using automatic or manual linkage based on:
  - Model meta information (physical quantities, types of data)
  - Model schematization (spatial)
- Run task(s) of model(s) to obtain results of the simulation.
- Observe results from the model which they run or wait until it will finishes.

Task

Contains a set of Models which can be linked and run together
Can be in one of the states: Initial, Running, Paused, Failed, Completed.

Model

Wraps any computitional engine such as SOBEK 1D Flow, Delft3D Flow, Waq, etc.
Provides access to the model input/output data at design and run-time.

Data Objects

The next issues are important when designing model data classes.

Declaration - meta information about data like data value type, data role, common properties.
Values - define entities to store different value types: scalar, field of values, time series, ....
- Scalar Parameters / Parameter Sets.
- Time- and space-related data.
- Definition of arbitrary functions.
Concept of time.
Concept of space.

Ideas about classes to define fields of values on different grids are wanted (0D, 1D, 2D, 3D). Review existing ideas used in VANDA, FEWS, Netter...

DataItem

Alternative names: ExchangeItem (OpenMI).

Any atomic data item in the system (concrete parameter, function, a set of parameters, time series, etc.).

The DataItem can be described as an entity which holds some value(s) used to define input or output of specific module.

Notes: for more detailed definition of Function and it's properties see Function.

Any of DataItem has a set of properties which describe what kind of data it stores. Minumum set of properties would include:

Name - real name of the data: "rating curve", "bathymetry", "velocity field"...
Quantity - physical quantity defining data: "flow", "elevation", "velocity"...

Quantity

Defines real physical quantity of the data.

Properties:

Name - real name: "mass", "concentration"
Unit - unit of measure.

Type of data

Currently the next types will be supported:
- Parameter - any data defined by only one value, f = v.
- Function - any function defined as a discrete or analytical function.
  - TimeSeries - Function with at least one argument / dimension defined as Time, f = f(t).
  - Field - Function with at least one argument / dimension defined as Location / Geometry.

All data types listed here will be described later in more details.

Parameter

Any scalar parameter defined by only one single value of any type (int, float, string, ...).

Function

An arbitrary function defined in a discrete form (probably also defined in analytical form).
We need to know:
- What each column in the value table means (physical quantity, units of measure).
- Which columns are arguments and which are function components.
- For more specific functions (time- or space- dependend) we need more specific classes, e.g. TimeSeries, Field)
- Accuracy of the data.
- ...

Example (define rating curve function):

Declaration:

Q = Q(H) - rating curve

Q[m^3/s] - water discharge, function
H[m] - water depth, argument

Value:

H |  Q
----------
0.0  0.0
1.0  0.1
5.0  0.2
...

Example (define properties of the cross-section):

Declaration:

F = (Area, Width, WettedPerimeter)(Depth), where

Area, Width, WettedPerimeter - function components
Depth - function argument

Values:

Depth | Area | Width | WettedPerimeter
----------------------------------------
0       0      0       0
1       10     10      12
2       20     10      14
...

Notes

the funtion type of data item is defined as a vector function it means that in general case function can consist from many function components, see also definition of vector function by MathRefference)

_Alternative names: Series.

In HDF5 library alternative to FunctionArgument and FunctionComponent can be Dimension and Variable.

TimeSeries

Time axes type: continious, equidistant/non-equidistant.

Put some examples for different types of time series.

Field

Alternative names: Coverage, GridCoverage.

A discrete function which defines a set values on on specific spatial domain (any point of the domain). In general any model grid can be presented as a set of cells / grid points / etc. which are defined by the same geometry (e.g. rectangular regular / irregular grids, triangulated irregular grids - TIN). Since we're not implementing GIS we will need only subset of GridCoverate features and maybe HDF5 features to be able to work nicely with the spatio-temporal models data.

Example

Results from Delft3D model "velocity field" - v(X, t), X = {cells of the Delft3D grid}.
Bathymetry of the waterbody - h(X), X = {cells of the Delft3D grid}.
Roughness value for all locations on the river network - n(X), X = {cells of the river grid}.

X should be defined in terms of specific model grid which will limits its values to some
limited set defined by some topological rule(s) (how they are connected).

Geometry Types (gis-enabled model schematization and data connected geometry)

In this section we provide all minimum set of Geometry / Location classes which are required to store data of the models.

In general they should include minimum subset of the OpenGIS specifications which will be able to describe:

Schematisation of the model: river, pipe network, water body, measurement station network, etc.
Model numerical grids: graph grids, rectangular regular / irregular grids, triangulated irregular grids and also a set of boundaries on these grids. See also Field in Data Objects section.

Grid

Alternative names: ElementSet, OpenGIS Complex Feature.

Base class for any grids (0D/1D/2D/3D).
All cells are assigned to a index which defines their position, so we can consider grid as array of cells with indexes.
Defines spatial domain of the model as a set of grid cells.

GridCell

GridRange

A set of cell numbers defined as a sequence of as a range of cell indices.

Model-specific plugins

The idea of system is that it should not be dependent as specific to some model (SOBEK, Delft3D, etc.). Instead it will provide a plugin-based approach to integrate any kind of model engine which will implement its own classes to work with:

Data - model-specific data structures.
Model Grid Schematization - numerical grid or mesh, topology, structures, etc.
Views / Forms - model-specific graphical user interface.

But still most of these classes will be implemented as extended version of default system classes.

The system will try to use the same if possible instead of defining each new term for each model.

0D Models

... define terms specific to 0D models.

1D Models (SOBEK)

SOBEK model uses the following schematization to define river newtork:

From the figure above it is possible to conclude that:

Reach is defined as an egde on the model network grid.
Any Reach is associated with exactly two Nodes (from/to topology).
Any Node is associated with zero or more incoming or outcoming Reaches.
Each Reach consists of one or more cells.
Each Cell is associated with exactly two CellBoundaries.
Any CellBoundary is associated with one or two Cells - depending if it is last cell or internal cell or the Reach.

As result the following UML Static Class diagram was designed:

All classes defined on the diagram should be part of any one dimensional model which uses network to define its numerical grid (or mesh).

_Most probably all cells and cell boundaries should be contained in the NetworkGrid class instead of Reach. In this case it will be easier to access them like:

foreach (Cell in networkGrid.Cells)
{
  ... do something
}

2D Models (SOBEK / Delft3D)

... define terms specific to 2D models.

3D Models (Delft3D)

... define terms specific to 3D models.

Appendix A: Possible definition of the simple sluice gate hydraulic structure

Real world definition

A simple sluice gate can be defined for example like:

sluice1/ - some set of data containing all properties of the sluice
    Width = 10 m
    Bottom Level = 100 cm
    Flow = Flow(h)

Flow(h) function here is defined for example as a table:

Flow[m3/s]| h[m]
---------------
100       | 0
101       | 5
105       | 15

And also we know that it is defined as an object at certain Location / Geometry.

In terms of the DataObjects listed above the gate can be defined as some class derived from Dataset which holds all data required to define all gate properties.

Code: define structure of our data (meta info)

// 1. create new data object for sluice
SluiceGate s = new SluiceGate();
s.Name = "sluice N1, near Delft";
s.Location = <set some location here as a Geometry / Feature object>;

// 2. set structure of the sluice
// (in fact this step will be defined by the Sluice() constructor or XML :))
s.Data.Add(new Parameter<float>("Width", new Quantity("width", "m")))
s.Data.Add(new Parameter<float>("Bottom Level", new Quantity("elevation", "m")))

// create function to describe Flow(Depth) curve for sluice.
Function f1 = new Function("Flow");
f1.Components.Add("Q", new Quantity("flow", "m3/s")); // add function component
f1.Arguments.Add("h", new Quantity("depth", "m")); // add function argument
s.Data.Add(f1); // add sluice flow function to the sluice object

Code: define values of the structure properties

// 3. set values
// select sluice object by name or index
SluiceGate s = model.Data["Structures"][""sluice N1, near Delft""];

// get width parameter and set a new value
Parameter w = s.Data["Width"];
w.Value = 10;

// get bottom level parameter and set a new value
Parameter yb = s.Data["Bottom Level"];
yb.Value = 100;

// get flow function, clear it and set some new values
Function qh = s.Data["Flow"];
qh.Clear();
qh.Values[0] = 0;
qh.Values[100] = 5;
qh.Values[101] = 15;

Code: define structure using ModelML (XML)

<dataset name="sluice N1, near Delft">
   <location id="location1">

    <parameter name="Width" valueType="float" quantity="width" unit="m">10</parameter>
    <parameter name="Bottom Level" valueType="float" quantity="elevation" unit="cm">100</parameter>

    <function name="Flow">
        <component name="Q" valueType="float" quantity="flow" unit="m3/s" />
        <arguments name="h" valueType="float" quantity="depth" unit="m" />
        <value>
            0 0
            5 100
           15 101
        </value>
    </function>
 </dataset>

ModelML and HydraulicsML listed here are part of some international standard. They can be a separate product from WL providing universal format for import / export of different data. Of course it is not good to invent a new file format bu it maybe usefull since it is self-descriptive, easy to read and fill. Another issue is that we can easilly mare any kind of reporting or backward-compatibility using XSL transformations.

Code: another way of defining structure using ModelML extension, HydraulicML (XML)

<sluice name="sluice N1, near Delft">
   <location id="location1">

    <width unit="m">10</width>
    <bottomLevel unit="cm">100</width>

    <flow type="Q(h)">
        0 0
        5 100
       15 101
    </flow>
<sluice>

Appendix B: How simple river flow model input may look like

Using the class library of data objects defined above it should be possible to define any model input / output data as a complex Dataset object. For example for simple river flow model it may look like:

Structure of the data tree (Schema) can be defined strictly by the concrete ModelWrapper or by the configuration file (XML). At run-time user should be able to set any concrete values of the model input data.

A model data defined in such a way will be:

Self descriptive
Data storage independed - we can store the whole Dataset in memory, XML file or in database depending on implementation.
Easy import/export of the data.
File format independed editors and viewers of the data (Model-View-Controller pattern).

Implementation

Currently the next parts of the system are being implemented:

ModellingFramework - a set of class libraries containing data, geometry and system classes allowing to work with various model data.
DataAnalysisTool - desktop tool which will be replacement of existing CAT, ODSVIEW. Main functionality will include import of data from existing formats and analysis / visualization. Some brainstorming materials related to implementation can be found here

The source code ban be found here: Trac

Project Directory Layout

Defines current organization of files in the project folder.

apps/
  ModellingFramework/ ............. components of the modelling framework (only class libraries!)
     lib/ ......................... common libraries
        log4net.dll
     src/ ......................... source code
        Core/ ..................... central entities of the system (Project, Task, Model)
        Data/ ..................... generic database-independent data access classes
        Data.Hibernate/ ........... NHibernate-based implementation of the data access
        DataObjects/ .............. main data object entities (Quantities, Functions, etc.)
        GeometryObjects/ .......... geometry / spatial entities (Grid, Cell)
        Gui/ ...................... gui facade logic (gui toolkit independent)
        Gui.Windows/ .............. gui implementation using System.Windows.Forms 2.0
        Utils/ .................... common utilities
     test/ ........................ test units and other tests
        TestUnits/
           Core/
           DataAccess/
           DataObjects/
           GeometryObjects/
           Gui/
           Utils/
   DataAnalysisTool/ ............... data analysis and visualization tool (a new CAT, ODSVIEW)
     src/
        DataAnalysisTool/ ......... main application (exe)
     test/
        TestUnits/
            DataAnalysisTool.Tests/

  ModellingEnvironment/ ........... desktop modelling system (planning)
     src/
        <Source Project>
     test/
        <Test Project>/

doc/ .............................. documentation (autogenerated)

sandbox/ .......................... experimental projects
   Spring.Net/
   NHibernate/

shared/ ........................... distribution of third-party tools
   Log4net/
   Spring.Net/
   NHibernate/
   ...

Note

real projects never depend on these directories directly, the required .dll files are copied in corresponding lib/ directories of each application instead. As result it will be possible to see what libraries are used by each application.

TODO: update this list regularly when new folders are added and keep other folders in the project clean (no garbage).

Third-party Components and Libraries used in development

The next components will be re-uses during development:

NUnit - use for all kind of tests.
Spring.Net
- Declarative (config file based) linking of different system components. Dramatically decreases dependency and simplifies testing if used properly.
- Persistency - probably will use to have a possibility to substitute different data access libraries, however NHibernate seems to be also sufficient.
- Remoting - also nice but will be used later, for example to implement Task-related logic as a remote component.
log4net - logging toolkit.
NHibernate - SQLite, FireBird or other database.
Gui control libraries:
- XtraPivotTable - tables.
- ComponentOne FlexGrid - seems to be the best for tables (TODO: compare with Xtra grid).
- DotNetBar - docking and other user-interface enchancements.
- Dundas Chart?.
- NPlot - good free charting library but maybe not so mature as others.
- JFreeChart - can be used also using IKVM but then Java Graphics implementation should be finilized for .NET (a lot already exists in IKVM).

Uml Diagrams

Delta.DataObjects Classes

Delta.Core

Diagrams are exported from Visual Studio solution.

Latesd Update: Don 15:21, 17 August 2006 (CEST).

To update diagrams:

Open project solution any open any diagram.
Right-click Export Diagram as Image ...
Select all and save to doc/uml/
Re-upload all UML diagrams below (add ModellingSystem_ prefix to all images while uploading).

Child pages

Modelling System Ideas