Delft-FEWS components are being deployed on many different architectures and hardware. The three main deployment types are:
When organizations are in the initial stage of (re-)defining their IT infrastructure, it is commonly recognized that after virtualization, containerization is the next logical step in the evolution of IT infrastructure. A container is a "lightweight" abstraction layer on top of the host operating system. Multiple containers share the machine’s operating system kernel and do not require the overhead of associating an operating system within each application. While we simplify Delft-FEWS for use with containers, running Delft-FEWS on physical hardware / virtual machines remains fully supported.
Delft-FEWS system installations on physical hardware / VMs are currently done by organizing a central database, installing RPMs / MSIs / unzipping the binaries, setting OS environment variables and starting a launcher service. Installations in the cloud are not much different. A general difference is that cloud installations are controlled using data driven yaml / json configuration files to apply the needed actions. In comparison with VMs, containers bring reduced start-up time, more compute capacity, more flexibility, fault isolation, ease of management, simplified security and reduced costs. The operational benefits for Delft-FEWS systems are also in line with the Roadmap plans for automation of installations with less needless customization, better auto-scaling and more flexible testing. We prefer using linux containers as much as possible. Whether linux containers can be used may depend on the requirements of the forecast model. Any Windows-based forecast models can be separately run on Windows hardware, Windows VMs (or in a Windows docker container).
| component | cloud readiness status | Room for improvements |
|---|---|---|
| Database | Both db docker containers as well as managed instances are already possible. Managed instances require minor adjustments of the db scripts. | Support one set of database scripts for all db flavors managed and unmanaged. |
Master Controller | Yes | Enable service replication |
Admin Interface | Yes | |
Operator Client / SA | Use Database proxy (Azure: Azure Virtual Desktop) | |
| ConfigManager | See Operator Client, in addition the AdminInterface API can be used. | |
Forecasting Shell Server | Yes | Facilitate auto scaling. |
WebServices | Yes | |
DatabaseProxy | Yes | |
OpenArchive | Yes |
The Delft-FEWS Hardware and software requirements for on-premise hardware / VMs also apply to deployment in the cloud. We recommend all containers to be linux unless Windows containers are specifically required. For Windows containers HW virtualization is required.
A dual Master Controller setup provides redundancy at the cost of additional compute resources. This concept is also recommended in the cloud when high availability is required.
| non-exhaustive list of options | remarks |
|---|---|
| database http proxy using SSL | |
| ssh + mobaxterm | |
| Citrix | integrates with cloud providers |
| Apache Guacamole |
There is no actual requirement for the Delft-FEWS components to use managed services. Managed services can be used as long performance is not affected. As an example, customers that are using SQLServer database replication between different geographical locations reported database timeouts. In response, we've improved our software, removed database indexes and added reconnection strategy for these problems. Since we expect Delft-FEWS users add many more simultaneous running Forecasting Shell servers in the future, we expect / foresee more challenges in this area. It is much better not to use the automated placement of database indexes.
Most container solutions have Kubernetes under the hood. For autoscaling we will use the Kubernetes API because in our view Kubernetes is the most commonly accepted and best supported cloud computing solution for this. Kubernetes uses Docker containers.
An estimate for the cost for a basic/medium-sized Delft-FEWS system in the cloud would be around 12k€- 15k€. Many cloud providers offer a "cloud calculator" to calculate, upfront, the expected cost. The estimate differs per Delft-FEWS system.
Egress, i.e. data traffic transferred from the cloud environment, is not free. The egress costs depend on the type of connection and on the amount of egress data. Ingress, e.g. data traffic into the cloud environment is often free of cost. Egress can be, partly, configured in the Delft-FEWS. Also the location of the Operator Client (in the cloud or on-premise) effects the egress.
The prices for for example a 32GB virtual machine and two 16GB virtual machines (VM) are quite similar. However, depending on your requirements, each separate VM requires additional and individual services like for example back-up and security.
Data can be stored in different cloud solutions each with it's own price and functionalities. Depending on your requirements you can add managed disks to your machine or a dedicated storage solution.
Cloud providers offer managed services like for example a managed database. The price for a managed solution is higher than an unmanaged solution. However, a managed solution requires a lot less of your own IT-staffing hours regarding managing the system.
When talking about high availability and scaling, the following concepts are important:
There can only be one Master Controller running at a time and is a single point of failure. This can only be avoided by deploying a dual MC system. Probably starting with Delft-FEWS 2022.02 multiple Master Controllers can be run on one database. As a result of this, the Master Controller cannot be scaled horizontally. Since the Master Controller is a single point of failure, it is important to monitor the health of the Master Controller. Scaling the Master Controller vertically can be done by redeploying the Master Controller.
Forecasting Shell Servers can be scaled up both horizontally and vertically. Deploying multiple Forecasting Shell Servers will avoid a single point of failure.
The Admin Interface can be scaled up both horizontally and vertically. Deploying multiple Admin Interfaces will avoid a single point of failure.
The Web Services can be scaled up both horizontally and vertically. Deploying multiple Web Services will avoid a single point of failure.
The Archive Server can be scaled up both horizontally and vertically, but more manual actions are required. Deploying multiple archive servers will avoid a single point of failure.
In case a single VM needs to be recovered the VM image will be restored using the latest functional backup.
For each of the Delft-FEWS components, the following post recovery actions are required:
In case of a disaster where the database and VMs have to be moved to a new data center or region, the VMs can be recovered from backup. After the Virtual Machines have been restored to a new location, database specific configurations will have to be adjusted on the virtual machines. This is a manual process. The specific requirements can be found in the Delft-FEWS System Administration Guide. On a high level, the following changes need to be performed for the Delft-FEWS components:
Master Controller: update the ENV variables with the changed database connection.
Admin Interface: update the ENV variables with the changed database connection.
Forecasting Shell Server:
Web Services: Update the ENV variables with the changed database connection.
Archive Server: update the storage location in the archive configuration file.
It is possible to have a synchronizing operator client on-premise. In case the database is no longer available, the Operator Client can still be used with the synchronized data in the local datastore. In case a new database has been installed after a disaster recovery, the Operator Client has to be reconfigured to access the new database.
In case a new database has been installed after a disaster recovery, the Operator Client has to be reconfigured to access the new database.
Delft-FEWS logs all events from workflows in the central database.
The Operator Client provides some access to information on the status of the system components, file imports and workflows.
The Web Based Delft-FEWS Admin Interface provides a dashboard for the FEWS Administrators to view the status of the Delft-FEWS components and workflows. Errors and events are logged within the central database and log extracts can be downloaded via the browser to provide to Deltares in the event of issues which can't be resolved internally. The Admin Interface also provides a series of APIs to enable access to the events and status information and the audit logs. Audit Logs of user actions are also stored in the central database and the Admin Interface API can be used to access these events.
For Delft-FEWS in the cloud the same principles apply for security as on premise: Security - Shared responsibility model for Delft-FEWS system installations. Securing your cloud assets requires continuous investment in keeping your containers safe. An infamous example of malconfigured cloud environment has been Tesla's unsecured admin console for a Kubernetes cluster. This led to malicious actors getting hold of credentials for Tesla's wider AWS environment who used it for cryptomining. Tesla highlighted that it was a test instance "only", but this incident shows why it's really important to secure both production and pre-production resources as far as possible.
Bottom line is to ensure / check any cloud instances you manage are appropriately secured. Use of cloud managed Kubernetes platforms (AKS, EKS, GKE) will generally make this easier and give you more confidence compared to situations where you have to run your own cluster, as the cloud provider will take care of many aspects of configuration. But regardless, be aware that running a cloud instance well and securely is a big undertaking that requires serious, proactive and ongoing effort to keep things secure and maintained.
https://cheatsheetseries.owasp.org/cheatsheets/Kubernetes_Security_Cheat_Sheet.html
https://cloudsecdocs.com/container_security/defensive/kubernetes/k8s_production_checklist/
https://media.defense.gov/2021/Aug/03/2002820425/-1/-1/1/CTR_KUBERNETES%20HARDENING%20GUIDANCE.PDF
Deltares has done several migrations and implementations of Delft-FEWS in the cloud. Microsoft Azure is a popular provider among the community, see also Delft-FEWS and Azure. Delft-FEWS can be run in almost any cloud-environment.
Deltares has successfully completed Delft-FEWS projects in the cloud with virtual machines using standard installation scripts, using virtual machines with AWS Elastic Beanstalk.
Deltares has successfully completed Delft-FEWS projects in the cloud with virtual machines using standard installation scripts, using virtual machines with Azure ARM templates. A good example is MDBA, they have a high level of knowledge of both the Delft-FEWS systems as well as the new technologies offered by cloud solutions. Example ARM templates have been provided by MDBA and can be found here: MDBA ARM templates download. NB. These scripts are provided as-is but are not supported by Delft-FEWS support.
For more info contact Delft-FEWS product management.