Page History

...

Delft-FEWS components are being deployed on many different architectures and hardware. A considerable amount of Delft-FEWS users use an IT infrastructure with virtual machines. The usual goal of virtualization is to centralize administrative tasks while improving The three main deployment types are:

1. physical servers - traditional server installation on top of OS.
2. virtual machines - improved scalability and overall hardware-resource utilization and centralized administrative tasks.
3. containers - cloud virtualization benefiting from a more efficient abstraction layer than virtual machines.

When organizations are in the initial stage of (re-)defining their IT infrastructure, it is commonly recognized that after virtualization, containerization is the next logical step in the evolution of IT infrastructure. It will remain possible to install . A container is a "lightweight" abstraction layer on top of the host operating system. Multiple containers share the machine’s operating system kernel and do not require the overhead of associating an operating system within each application. While we simplify Delft-FEWS for use with containers, running Delft-FEWS on on-premise hardware, or in physical hardware / virtual machines remains fully supported.

Similar installation steps to regular installation

Delft-FEWS system installation installations on regular physical hardware / VMS is VMs are currently done by organizing a central database, installing RPMs / MSIs / unzipping the binaries, setting OS environment variables and starting a launcher service. For installation in Kubernetes this is not going to be Installations in the cloud are not much different. Usually this is A general difference is that cloud installations are controlled using data driven yaml / json configuration files to apply the needed actions.  In comparison with VMs, containers bring reduced start-up time, more compute capacity, more flexibility, fault isolation, ease of management, simplified security and reduced costs. The operational benefits for Delft-FEWS systems are also in line with the Roadmap plans for automation of installations with less needless customization, better auto-scaling and more flexible testing. We prefer using linux containers as much as possible. Whether linux containers can be used may depend on the requirements of the forecast model.  Any Windows-based forecast models can be separately run on Windows hardware, Windows VMs (or in a Windows docker container). 

Autoscaling

...

Delft-FEWS Hardware and software requirements

...

...

1. for file-based imports, use Network File Service (NFS) or Windows shares
2. for server imports serving public data, ftp / http can be used (encryption would provide unnecessary overhead), other services in need of passwords should should use a secure connection / https

Autoscaling with Kubernetes

Kubernetes uses Docker containers. A container is a "lightweight" abstraction layer on top of the host operating system. Multiple containers share the machine’s operating system kernel and do not require the overhead of associating an operating system within each application. In comparison with VMs, containers bring reduced start-up time, more compute capacity, more flexibility, fault isolation, ease of management, simplified security and reduced costs. The operational benefits for Delft-FEWS systems are also in line with the Roadmap plans for automation of installations with less needless customization, better auto-scaling and more flexible testing. We prefer using linux containers as much as possible. Whether linux containers can be used may depend on the requirements of the forecast model.  Any Windows-based forecast models can be separately run on Windows hardware, Windows VMs (or in a Windows docker container)Most container solutions have Kubernetes under the hood.  For autoscaling we will use the Kubernetes API because in our view Kubernetes is the most commonly accepted and best supported cloud computing solution for this. Kubernetes uses Docker containers.

Important cost variables

An estimate for the cost for a basic/medium-sized Delft-FEWS system in the cloud would be around 12k€- 15k€. Many cloud providers offer a "cloud calculator" to calculate, upfront, the expected cost. The estimate differs per Delft-FEWS system.

...

For Delft-FEWS in the cloud the same principles apply for security as on premise: Security - Shared responsibility model for Delft-FEWS system installations. Securing your cloud assets requires continuous investment in keeping your containers safe. An infamous example of malconfigured Kubernetes cloud environment has been Tesla's unsecured admin console for a Kubernetes cluster.  This led to malicious actors getting hold of credentials for Tesla's wider AWS environment who used it for cryptomining. Tesla highlighted that it was a test instance "only", but this incident shows why it's really important to secure both production and pre-production resources as far as possible. 

...

Bottom line is to ensure / check any Kubernetes cloud instances you manage are appropriately secured. Use of cloud managed Kubernetes platforms (AKS, EKS, GKE) will generally make this easier and give you more confidence compared to situations where you have to run your own cluster, as the cloud provider will take care of many aspects of configuration.  But regardless, be aware that running a Kubernetes cluster cloud instance well and securely is a big undertaking that requires serious, proactive and ongoing effort to keep things secure and maintained.

...

• https://learnk8s.io/production-best-practices

• https://cheatsheetseries.owasp.org/cheatsheets/Kubernetes_Security_Cheat_Sheet.html

• https://cloudsecdocs.com/container_security/defensive/kubernetes/k8s_production_checklist/

• https://media.defense.gov/2021/Aug/03/2002820425/-1/-1/1/CTR_KUBERNETES%20HARDENING%20GUIDANCE.PDF

Examples

Deltares has done several migrations and implementations of Delft-FEWS in the cloud. Microsoft Azure is a popular provider among the community, see also Delft-FEWS and Azure. Delft-FEWS can be run in almost any cloud-environment.

AWS Elastic Beanstalk

Deltares has successfully completed Delft-FEWS projects in the cloud with virtual machines using standard installation scripts, using virtual machines with AWS Elastic Beanstalk.

ARM templates

Deltares has successfully completed Delft-FEWS projects in the cloud with virtual machines using standard installation scripts, using virtual machines with Azure ARM templatesand AWS Elastic Beanstalk. For practical reasons, will keep our requirements / installation instructions as cloud neutral as possible.

ARM templates

. A good example is MDBA, they have a high level of knowledge of both the Delft-FEWS systems as well as the new technologies offered by cloud solutions. Example ARM templates have been provided by MDBA and can be found here: MDBA ARM templates download

Getting started with the Cloud

Deltares has done several migrations and implementations of Delft-FEWS in the cloud. Microsoft Azure is a popular provider among the community but Delft-FEWS will run in any cloud-environment. See also Delft-FEWS and Azure.
Based on our experience with successful migration and implementations like MDBA (link) we drafted a "how to get started" bullet list.

. NB. These scripts are provided as-is but are not supported by Delft-FEWS support.

Getting started with the Cloud

1. Involve your IT solution provider Make sure your IT solution provider is involved from the beginning of the project.
2. Train / recruite staff / organisation so that there is a good understanding and knowledge of the specific cloud you want to host your system in. Mapping the functional requirements of the possible cloud solutions can be done much faster with people skilled in the cloud domain. A good example is MDBA, they have a high level of knowledge of both the Delft-FEWS systems as well as the new technologies offered by cloud solutions.in the cloud solution.
3. Define functional and technical requirements: scalability, Create a list of requirements. Both functional and technical. Also incorporate requirements like performance, uptime, disaster recovery, high availability etc. Make sure that you also are aware of your company rules regarding using and migrating to the cloud.
4. Verify forecast model requirements / licences are suitable for containers.Check which forecast models need to be run and if these can be run in the cloud (and, if applicable, under which licences)
5. Organise a couple of workshops with Deltares (or another partner) to map the requirements of the cloud solutions.   
6. Create an implementation or migration plan.
7. Implement a dry run phase. In this phase, the whole system is up and running but not for operational use. During this phase, the users can use the system like an operational system to test whether everything is functioning as expected. 

...