...
Cloud providers offer managed services like for example a managed database. The price for a managed solution is higher than an unmanaged solution. However, a managed solution requires a lot less of your own IT-staffing hours regarding managing the system.
Additional requirements
...
High Availability and scaling
Introduction
When talking about high availability and scaling, the following concepts are important:
- Scale horizontally
- Scale out: add more virtual machines or containers of the same component
- Scale in: remove virtual machines or containers of the same component
- Scale vertically
- Scale up: increase the resources of a virtual machine or container like CPU or RAM
- Scale down: decrease the resources of a virtual machine or container like CPU or RAM
Scaling up of Azure components can be done manually. Some possibilities are:
- Create a new Virtual machine by restoring an existing backup.
- Use a Azure Automation Runbook in combination with DSC to install the Deflt-FEWS software
- Use a devops solution tor creating a new virtual machine by deploying ARM templates of the components.
For each Delft-FEWS component the scaling capabilities are discussed. Delft-FEWS does not auto scale components itself. If required, auto scaling must be managed by the Azure environment. Azure has autoscaling features like VM Scale sets (which require a custom VM image), App Services and Kubernetes to support autoscaling.
Availability Set
In Azure all Virtual machines can be deployed as part of an Availability Set. This will assure that VMs that are part of the same Availability Set won’t be affected at the same time by Azure maintenance, windows updates etc.
Availability and scaling the Master Controller
There can only be one Master Controller running at a time and is a single point of failure. This can only be avoided by deploying a dual MC system.
As a result of this, the Master Controller cannot be scaled horizontally. Since the Master Controller is a single point of failure, it is important to monitor the health of the Master Controller.
Scaling the Master Controller vertically can be done by redeploying the Master Controller.
Availability and scaling the Forecasting Shell Servers
Forecasting Shell Servers can be scaled up both horizontally and vertically.
Scale horizontally: deploying a new Forecasting Shell Server virtual machine for an FSS Group that needs more forecasting shells.
Scale vertically: update an existing Forecasting Shell Server Virtual Machine with more resources. To allow a Forecasting Shell to use more memory, the Delft-FEWS FSS client config file can be customized. It is also possible to run multiple Forecasting Shells on the same Virtual machine.
Deploying multiple Forecasting Shell Servers will avoid a single point of failure.
Availability and scaling the Admin Interface
The Admin Interface can be scaled up both horizontally and vertically.
Scale horizontally: deploying a new Admin Interface virtual machine. A load balancer or application gateway is used in front of the Admin Interface and new VMs must be registered with the loadbalancer.
Scale vertically: update an existing Admin Interface Virtual Machine with more resources.
Deploying multiple Admin Interfaces will avoid a single point of failure.
Availability and scaling the Web Services
The Web Services can be scaled up both horizontally and vertically.
Scale horizontally: deploying a new Web Services ARM template. A load balancer or application gateway is used in front of the Web Services and new VMs must be registered with the loadbalancer.
Scale vertically: Redeploy the Web Services Virtual Machine with more resources.
Deploying multiple Web Services will avoid a single point of failure.
Availability and scaling the Archive Server
The Archive Server can be scaled up both horizontally and vertically, but more manual actions are required.
Scale horizontally: Deploy a new Archive Server Virtual Machine. A load balancer or application gateway is used in front of the Archive and new VMs must be registered with the loadbalancer.
The scheduling of the harvester must be duplicated to the new VM. Since the harvester builds the index it takes some time before a new archive server is in sync. The load balancer should use sticky sessions to make sure consistent results are given all the time for the same user.
Scale vertically: Redeploy the Archive Server Virtual Machine with more resources.
Deploying multiple archive servers will avoid a single point of failure.
Disaster Recovery
This section will describe possible Disaster Recovery solutions for the different parts of a Virtual Machine based Azure system:
• Azure Managed Database
• Virtual machines
• Azure Storage Accounts
Azure Managed Database Disaster Recovery
Azure Managed databases are backed up. Make sure database backups are stored using geo redundancy. In case of a disaster, the database needs to be restored to another data center in the same region or to another region in case the whole region is affected.
Since restoring a backup to another environment changes the database hostname, this impacts all Delft-FEWS components that refer to the database. This will impact both the server components and the Operator Client.
Virtual machines disaster recovery
Azure backup can be used to restore a Virtual Machines that are backed up using a storage account that support geo-redundant storage (GRS).
Single VM recovery
In case a single VM needs to be recovered the VM image will be restored using the latest functional backup.
For each of the Delft-FEWS components, the following post recovery actions are required:
- Master Controller: no actions
- Admin Interface: no actions
- Forecasting Shell Server: no actions
- Web Services: no actions
- Archive Server:
- Archive Configuration files that were uploaded after the backup, need to be uploaded again.
- The archive harvester should be run to make sure the indexes are up-to-date. This is done in the Archive server web application.
VM recovery with new database
In case of a disaster where the database and VMs have to be moved to a new data center or region, the VMs can be recovered in 2 ways:
Restore Virtual Machine from backup
After the Virtual Machines have been restored to a new location, database specific configurations will have to be adjusted on the virtual machines. This is a manual process. The specific requirements can be found in the Delft-FEWS System Administration Guide.
On a high level, the following changes need to be performed for the Delft-FEWS components:
Master Controller: update the ENV variables with the changed database connection.
Admin Interface: update the ENV variables with the changed database connection.
Forecasting Shell Server:
- Update the FSS ENV variables with the changed database connection, username and password.
- Update the global.properties of the FSS in the Delft-FEWS configuration and upload them with the Config Manager. Typical changes are: URL to the archive server, location of the Azure File Shares for imports and the archive.
Web Services: Update the EVN variables with the changed database connection.
Archive Server: update the location of the Azure File Share in the archive configuration file.
performance
Security
For Delft-FEWS in the cloud the same principles apply for security as on premise: Security - Shared responsibility model for Delft-FEWS system installations. Securing your cloud assets requires continuous investment in keeping your containers safe. An infamous example of malconfigured Kubernetes has been Tesla's unsecured admin console for a Kubernetes cluster. This led to malicious actors getting hold of credentials for Tesla's wider AWS environment who used it for cryptomining. Tesla highlighted that it was a test instance "only", but this incident shows why it's really important to secure both production and pre-production resources as far as possible.
...