Delft-FEWS and the cloud

Introduction

Delft-FEWS components are being deployed on many different architectures and hardware. A considerable amount of Delft-FEWS users use an IT infrastructure with virtual machines. The usual goal of virtualization is to centralize administrative tasks while improving scalability and overall hardware-resource utilization. When organizations are in the initial stage of (re-)defining their IT infrastructure, it is commonly recognized that after virtualization, containerization is the next logical step in the evolution of IT infrastructure. Deltares will provide guidance. This guidance shall be focused on Kubernetes because in our view Kubernetes is the most commonly accepted and best supported cloud computing solution.

Kubernetes uses Docker containers. A container is a "lightweight" abstraction layer on top of the host operating system. Multiple containers share the machine’s operating system kernel and do not require the overhead of associating an operating system within each application. In comparison with VMs, containers bring reduced start-up time, more compute capacity, more flexibility, fault isolation, ease of management, simplified security and reduced costs. The operational benefits for Delft-FEWS systems are also in line with the Roadmap plans for automation of installations with less needless customization, better auto-scaling and more flexible testing. We prefer using linux containers as much as possible. Whether linux containers can be used may depend on the requirements of the forecast model.  Any Windows-based forecast models can be separately run on Windows hardware, Windows VMs (or in a Windows docker container).

Delft-FEWS Software: A cloud agnostic approach

Delft-FEWS system installation on regular hardware / VMS is currently done by installing RPMs, unzipping the binaries, setting OS environment variables and starting a launcher service. For installation in Kubernetes this is not going to be much different. Usually this is controlled using data driven yaml / json configuration files to apply the needed actions.

Delft-FEWS in the cloud: reference architectures

Explain and visualize reference architectures

• Single MC
• Dual MC (Multi MC?)

Hard- software requirements

Indications of hardware specs for installing the different VM's / containers.

The memory requirements in the cloud are similar as in a VM or on-premise. We recommend all containers to be linux unless Windows containers are specifically required. For Windows containers HW virtualization is required.

Typical cloud related choices (cloud FAQs)

Based on Webinar content / known FAQs specify a number of sub-topics, like

• Costs
  • An estimate for the cost for a basic/medium-sized Delft-FEWS system in the cloud would be around 12k€- 15k€ .
    Many cloud providers offer a "cloud calculator" to calculate, upfront, the expected cost (i.e. Azure calculator).|
    The estimate will differ per Delft-FEWS system. Important cost variables are:
    • data (size and egress)
    • size of the whole Delft-FEWS system (sizing of the VM's, number of the VM's, number of Delft-FEWS components per VM)
    • the number of users (depending on the location of the Operator client; in the cloud v.s. on-premise)
    • requirements regarding redundancy, High Availability, Disaster Recovery, performance 
  • Data ingress and egress (egress exits the cloud and ingress enters the cloud)
    Ingress and data traffic in the cloud environment is often free of cost.
    Egress however is not free. The egress cost depend on the type of connection and on the amount of egress data.
    Some examples/comparisons for Azure can be found on this webpage: How to calculate the right Azure outbound capacity and choose the best egress option
  • Type and number of machines
    The price difference between for example a 32GB virtual machine and two 16GB virtual machines (VM) is quite the same. However, depending on your requirements, each separate VM requires additional and individual services like for example back-up and security. 
  • Data storage
    Data can be stored in different cloud solutions each with it's own price and functionalities. Depending on your requirements you can add managed disks to your machine or a dedicated storage solution like Azure files or Blob storage solutions
  • Managed v.s unmanaged
    Cloud providers offer managed services like for example a managed database. The price for a managed solution is higher than an unmanaged solution. However, a managed solution requires a lot less of your own IT-staffing hours regarding managing the system.

Scalability

         Kubernetes/Containers

          DevOps (Infrastructure as Code, Automatic deployments of config changes)

Installation of Operator Clients

Use of managed services

There is no actual requirement for the Delft-FEWS components to use managed services. Managed services can be used as long performance is not affected. As an example, customers that are using SQLServer database replication between different geographical locations reported database timeouts. In response, we've adjusted our database indexes and reconnection strategy for these problems. Since we expect Delft-FEWS users add many more simultaneous running Forecasting Shell servers in the future, we expect / foresee more challenges in this area. It is much better not to use the automated placement of database indexes.

How to deal with (incoming, outgoing) data feeds

1. for file-based imports, use Network File Service (NFS) or Windows shares.
2. for server imports serving public data, ftp / http can be used (encryption would provide unnecessary overhead), other services in need of passwords should should use a secure connection / https

Security

Securing your cloud assets requires continuous investment in keeping your containers safe. An infamous example of malconfigured Kubernetes has been Tesla's unsecured admin console for a Kubernetes cluster (Lessons from the Cryptojacking Attack at Tesla).  This led to malicious actors getting hold of credentials for Tesla's wider AWS environment who used it for cryptomining. Tesla highlighted that it was a test instance "only", but this incident shows why it's really important to secure both production and pre-production resources as far as possible. 

• do not use insecure keys
• do not inappropriately open network configuration on test instances because they are "just" test instances.

Bottom line is to ensure / check any Kubernetes instances you manage are appropriately secured. Use of cloud managed Kubernetes platforms (AKS, EKS, GKE) will generally make this easier and give you more confidence compared to situations where you have to run your own cluster, as the cloud provider will take care of many aspects of configuration.  But regardless, be aware that running a Kubernetes cluster well and securely is a big undertaking that requires serious, proactive and ongoing effort to keep things secure and maintained.

Deltares recommends a three-layered approach:

1. The inner layer is the central database (and optionally Deltares Open Archive).
2. The middle layer are the components that communicate directly with the database using encryption.
3. The third layer (optional) is a reverse proxy to the database that can be accessed externally.

How to get started

Deltares has done several migrations and implementations of Delft-FEWS in the cloud. Microsoft Azure is the most popular provider among the community but Delft-FEWS will run in any cloud-environment.
Based on our experience with successful migration and implementations like  MDBA (link) we drafted a "how to get started" bullet list.

1. Make sure your IT solution provider is involved from the beginning of the project
2. It's a big plus in case your organisation or IT solution provider has a good understanding and knowledge of the specific cloud you want to host your system
   Mapping the functional requirements of the possible cloud solutions can be done much faster with people skilled in the cloud domain. A good example is MDBA, they have a high level of knowledge of both the Delft-FEWS systems as well as the new technologies offered by cloud solutions.
3. Create a list of requirements.
   Both functional and technical. Also incorporate requirements like performance, uptime, disaster recovery, high availability etc.
   Make sure that you also are aware of your company rules regarding using and migrating to the cloud.
4. Organise a couple of workshops with Deltares (or another partner) to map the requirements on the cloud solutions.   
5. Create an implementation or migration plan
6. Implement a dry run phase
   In this phase, the whole system is up and running but not for operational use. During this phase, the users can use the system like an operational system to test whether everything is functioning as expected. 

Best practices & recommendations

• https://learnk8s.io/production-best-practices

• https://cheatsheetseries.owasp.org/cheatsheets/Kubernetes_Security_Cheat_Sheet.html

• https://cloudsecdocs.com/container_security/defensive/kubernetes/k8s_production_checklist/

• https://media.defense.gov/2021/Aug/03/2002820425/-1/-1/1/CTR_KUBERNETES%20HARDENING%20GUIDANCE.PDF

• Project references
• Technical knowledge Deltares has successfully completed Delft-FEWS projects with Azure ARM templates and AWS Elastic Beanstalk. For practical reasons, will keep our requirements / installation instructions as cloud neutral as possible.
• Involved people
• More info / Deltares contact(s)