Cloud Notary 

In our field of hydrodynamic modelling, it is common to have sensitive and/or large and expensive datasets, which are vital to the business and treated as secret. These datasets however should still be accessible to downstream algorithms and (external) models to analyze and improve these models. In other words, Party A has data x and Party B has model y and both do not want to or cannot share their property. However, they are both interested in the result z = operation(x,y). We envision the use of cloud computing as a facilitator for the operation. 

An example of such a cooperation is Van Oord and Deltares. Van Oord has Buoy data which could be useful to improve the Global Tide and Surge Model (GTSM) from Deltares. Both parties cannot share their property but are interested in the improved model results. This use case is taken to create an architecture in Microsoft Azure as described below. 

Key concepts 

In our brainstorming sessions, a few key concepts came up. First, that both parties still need to trust both the data and the algorithm to be valid and to do only what they need to do and nothing else. Therefore, a legal contract should be drawn up. Second, the environment that will hold both the data and the algorithm should be transparent as possible, i.e., it should be clear what it will do beforehand. This is best solved by presenting the infrastructure as code beforehand. Third and last, the envisioned operation should run as a functional account that ideally prevents any access by both parties. 

• Legal contract in which both parties trusts both data and algorithm. 

• Infrastructure as code, transparent how the data and algorithm will run. 

• Use functional accounts to prevent user tampering. 

Continuous integration and continuous deployment (CI/CD), normally used to test and deploy code on each commit, is a suitable candidate to provision a secure cloud environment. Using Github and Azure DevOps (both from Microsoft) we can store the environment as code and setup a cloud environment. In the Azure DevOps setup, a service level user is created that will gain access to both the Deltares Azure environment as the van Oord Azure environment, but without either parties seeing those keys or gaining access themselves. 

Image Added 

Figure 1 Cloud notary architecture in Azure 

Architecture 

The implementation expects an initial setup at Microsoft Azure for the involved parties. This is illustrated in Figure 1. 

Van Oord has a resource group within their subscription with a storage account and a key vault. The storage account contains their secret dataset, and the key vault stores a key which provides access to the storage account. 

Deltares has a resource group within their subscription with a container registry and a key vault. The container registry contains their container with numerical model and in the key vault a key is stored which provides access to the storage account. 

This initial setup is the input for an Azure DevOps pipeline. This pipeline creates a Kubernetes cluster with the keys stored in secrets. This Kubernetes cluster is then used to start a workflow which brings data and model together. A more detailed description of the pipeline is given next.  

Image Added 

The actions in the pipeline are illustrated in the image below. A service account from both Van Oord and Azure are used in the pipeline for access to the relevant items in the recourse groups as described above. 

• Several keys are extracted from the Deltares key vault 

• Access key to a storage account with input data for the model. 

• Access key to the container registry in Microsoft Azure for the pre- and post-processing containers. 

• Access key to Docker hub delft3dfm registry for the model container 

• Service principal for the Kubernetes cluster is extracted from the Deltares key vault. This service account will be used to setup the cluster. 

• The access key for the Van Oord storage account is extracted from the Van Oord key vault. 

At this point all information to setup a new model run is available in de pipeline. In the next steps the Kubernetes cluster will be prepared to run the model. Ideally the cluster will run in a completely independent Microsoft Azure account. Unfortunately, this is not a service provided by Microsoft Azure, so for this moment it is created in a separated resource group in the Deltares account.  

For the cluster setup we need the following steps: 

• Create Azure Kubernetes Cluster with the following configuration options: 

• Number of nodes 

• Node size 

• Install Argo workflow manager. Argo allows is to run a workflow with sequential and parallel steps. 

• Install secrets. The secrets for the storage accounts and container registries are installed in the cluster. 

• Mount volumes. For the storage accounts Persistent Volumes and Persistent Volume Claims are created to mount these volumes in the containers 

• In the last step the Argo workflow is submitted to run the Global Tide and Surge Model. 

Image Added 

Operation 

Once the above architecture is setup, any user can trigger the workflow by making changes to the code, for example to point to a new input path or timeframe. The CI will trigger and run the complete pipeline, both to setup the architecture as to run the workflow. Larger changes to the infrastructure can be reviewed in a merge request like manner. 

The last step of the pipeline is to generate a comparison between the model and the observational data, in our case between Buoy data and our GTSM model. Such a result is seen in Figure 2. Note that the output has to strike a balance between being detailed enough to improve the model and coarse enough that the original input cannot be retrieved anymore. 

Image Added 

Figure 2 Output of the complete processing pipeline. The model is overestimating the magnitude of the velocity in this case, especially on lower velocities. 

Future work 

The above architecture and usage are a working proof of concept. We must note however that while the use of DevOps as a CI service is useful for the transparent setup of the architecture, it’s less so for actually triggering the pipeline, for which a standalone service should be made. This service can however be provisioned by the above pipeline. 

This concept also cannot guarantee a complete inability to access the model or data by the other party when only two parties are involved, as the service account under which the pipeline runs is still owned by of them. We also couldn’t guarantee that such access, while non-trivial, would show up in logs visible to the other party. 

We’ll keep in touch with Microsoft to keep track of possible solutions for the above problems, as they do develop similar services for secure computing. The code for the above pipeline and preparations in Azure can be found online at https://github.com/openearth/tki-cloud-notary 

Image Removed

Continuous integration aims to improve the quality of software by continuously testing and validating software. In context of hydro- and morphodynamic model development, testing and validating typically involves model/data comparisons for various parameter settings, i.e. calibration. If either the model code changes, or additional measurement data become available, ideally the model is recalibrated and validated against data. But also if neither the model code nor that available data changes, recalibration and validation might be necessary. Calibration and validation is specific for a certain period of time, area and required amount of detail. For example, a global model, calibrated and validated on a worldwide dataset, might very well deviate significantly from reality in a particular location or at a particular moment in time. Global models should be used with care for local purposes.

The project DEL063 - Continuous Integration of Models and Data aims at the development of a standardized calibration and validation framework for hydro- and morphodynamic models that allows for continous calibration and validation of these models at particular datasets for particular purposes. The project focuses on global operational models. An interactive web-based shell is developed through which operational global models can be validated and calibrated agains user-defined data. The interactive shell also allows for user-defined queries on the operational model forecasts.

Global operational modeling

Deltares joined the worldwide tendency to provide services instead of stand-alone software packages. A recent example is the GLObal Storm Surge Information System (GLOSSIS) that provides global 10-day operational flood forecasts every 6 hours based on a global Delf3D FM model in a FEWS framework. Currently, GLOSSIS provides hourly global data maps and 10-min data time series at predefined locations free of charge. In addition, customers can subscribe to a paid data service to obtain customized data packages every 6 hours (currently in pilot phase). GLOSSIS is launched with the explicit purpose of providing a solid basis for operational forecast systems that can be expanded as customer demand increases.

A possible expansion of GLOSSIS is interactivity. In an interactive system, customers have the ability to interact with the system by providing additional input. In the case of GLOSSIS, customers might provide the system with their own local datasets in order to validate or calibrate the model output locally. As a global model is always optimized on a global scale, locally the model results might still deviate significantly. Providing local data to the model system has multiple benefits for both customers and Deltares: 1) the local model error can be quantified, 2) the model error can be locally corrected, and 3) the global model can be optimized.

The transition of GLOSSIS towards an interactive global operational model system is subject to this project.

Interaction in global operational models is an unexplored area in the field of numerical modelling. Therefore, it is the intention to initiate the transition towards an interactive global operational model system one step at the time. Each step pursues added value to the operational system that can be deployed operationally against minimal investment. Customer feedback should guide development to the next steps and explore the potential user base as to closely tie technical advances to customer demand.

Potential users

The potential user base of interactive global operational model systems is yet unknown. An explicit objective of the development process is to explore the potential user base. Nevertheless, the potential user base can roughly be characterised as high-demand users (i.e. high resolution and/or frequency, SLA) with infrequent or changing data requirements.

For low-demand customers (i.e. low resolution and/or frequency, no SLA) or interested users without specific purpose for the data provided, the existing free data services (i.e. providing graphs of time series through www.globalfloodforecast.com) are likely sufficient. High-demand customers with high-frequent, but constant data requirements can benefit from the paid subscription. But for high-demand customers with infrequent or changing data, neither the free data service nor the paid subscription likely fit their needs and an interactive global operational model system can be of added value.

High-demand customers with infrequent or changing data requirements can be end-users, like contractors with irregular operating times and location, or business-to-business users and resellers, like rapid-response or search-and-rescue service providers that operate on a global scale. Nowadays, these include the big tech companies (e.g. Google, Facebook, Amazon) that increasingly provide societal data services in areas struck by disaster and/or conflict. Amongst others, one could think about search-and-rescue, oil spills, but also navigation optimization.

The service will be provided as a public service. Public services should be cost-effective, but not necessarily profitable. Minimum revenue should be generated to prevent the service to be loss-making. The service should be scriptable to allow third-parties to build commercial services upon the public service.

Partners

• Deltares
• Van Oord Dredging
• Microsoft

Activity

2018 Q1

• Aligned project plans with FEWS/GLOSSIS developments for 2018.
• Designed interactive shell.
• $10.000 computational credit on Microsoft Azure from AI for Earth Grant.

2018 Q2

...