...
Type and number of machines
The price difference between prices for for example a 32GB virtual machine and two 16GB virtual machines (VM) is are quite the samesimilar. However, depending on your requirements, each separate VM requires additional and individual services like for example back-up and security.
...
- Create a new Virtual machine by restoring an existing backup.
- Use a Azure Automation Runbook in combination with Desired State Configuration (DSC) to install the Deflt-FEWS software
- Use a devops devOps solution tor creating a new virtual machine by deploying ARM templates of the components.
...
There can only be one Master Controller running at a time and is a single point of failure. This can only be avoided by deploying a dual MC system. Probably starting with Delft-FEWS 2022.02 multiple Master Controllers can be run on one database.
As a result of this, the Master Controller cannot be scaled horizontally. Since the Master Controller is a single point of failure, it is important to monitor the health of the Master Controller.
Scaling the Master Controller vertically can be done by redeploying the Master Controller.
...
Forecasting Shell Servers can be scaled up both horizontally and vertically. Deploying multiple Forecasting Shell Servers will avoid a single point of failure.
- Scale horizontally: deploying a new Forecasting Shell Server virtual machine for an FSS Group that needs more forecasting shells.
- Scale vertically: update an existing Forecasting Shell Server Virtual Machine with more resources. To allow a Forecasting Shell to use more memory, the Delft-FEWS FSS client config file can be customized. It is also possible to run multiple Forecasting Shells on the same Virtual machine.
...
Admin Interface
The Admin Interface can be scaled up both horizontally and vertically. Deploying multiple Admin Interfaces will avoid a single point of failure.
- Scale horizontally: deploying a new Admin Interface virtual machine. A load balancer or application gateway is used in front of the Admin Interface and new VMs must be registered with the loadbalancer.
- Scale vertically: update an existing Admin Interface Virtual Machine with more resources
...
- .
Web Services
The Web Services can be scaled up both horizontally and vertically. Deploying multiple Web Services will avoid a single point of failure.
- Scale horizontally: deploying a new Web Services ARM template. A load balancer or application gateway is used in front of the Web Services and new VMs must be registered with the loadbalancer.
- Scale vertically: Redeploy the Web Services Virtual Machine with more resources
...
- .
Archive Server
The Archive Server can be scaled up both horizontally and vertically, but more manual actions are requiredmore manual actions are required. Deploying multiple archive servers will avoid a single point of failure.
- Scale horizontally: Deploy a new Archive Server Virtual Machine. A load balancer or application gateway is used in front of the Archive and new VMs must be registered with the loadbalancer.
The scheduling of the harvester must be duplicated to the new VM. Since the harvester builds the index it takes some time before a new archive server is in sync. The load balancer should use sticky sessions to make sure consistent results are given all the time for the same user. - Scale vertically: Redeploy the Archive Server Virtual Machine with more resources.
...
High available scenarios
On a high level the different scenarios for running Delft-FEWS in a highly available manner when the primary Azure region fails, from most high available to least available
...
This section will describe possible Disaster Recovery solutions for the different parts of a Virtual Machine based Azure system:• Azure
- Azure Managed Database
...
- Virtual machines
...
- Azure Storage Accounts
It is also possible to use Azure Site Recovery. This is the disaster recovery solution offered by Azure. Depending on the database solution chosen, it may still be required to do post recovery configurations.
...
Azure Managed databases are backed up. Make sure database backups are stored using geo redundancy. In case of a disaster, the database needs to be restored to another data center in the same region or to another region in case the whole region is affected.
Since restoring a backup to another environment changes the database hostname, this impacts all Delft-FEWS components that refer to the database. This will impact both the server components and the Operator Client.
...
In case a single VM needs to be recovered the VM image will be restored using the latest functional backup.
For each of the Delft-FEWS components, the following post recovery actions are required:
...
In case of a disaster where the database and VMs have to be moved to a new data center or region, the VMs can be recovered from backup. After the Virtual Machines have been restored to a new location, database specific configurations will have to be adjusted on the virtual machines. This is a manual process. The specific requirements can be found in the Delft-FEWS System Administration Guide.
On a high level, the following changes need to be performed for the Delft-FEWS components:
...
Web Services: Update the EVN ENV variables with the changed database connection.
Archive Server: update the location of the Azure File Share in the archive configuration file.
...
It is possible to have a synchronizing operator client on-premise. In case the Azure Database is no longer available, the Operator Client can still be used with the synchronized data in the local datastore.
In case a new database has been installed after a disaster recovery, the Operator Client has to be reconfigured to access the new database.
...
The Web Based Delft-FEWS Admin Interface provides a dashboard for the FEWS Administrators to view the status of the Delft-FEWS components and workflows. Errors and events are logged within the central database and log extracts can be downloaded via the browser to provide to Deltares in the event of issues which can't be resolved internally.
The Admin Interface also provides a series of APIs to enable access to the events and status information and the audit logs.
Audit Logs of user actions are also stored in the central database and the Admin Interface API can be used to access these events.
...
Using the Log Analytics service (part of Azure Monitoring) of Azure allows collecting log events from the different Delft-FEWS components.
The Master Controller, Forecasting Shell Server and Admin Interface support sending error log events to the Windows Event Logger or Linux syslog. Log Analytics can be connected to the Windows Event Logger and syslog. This requires installing the MicrosoftMonitoringAgent extension on the virtual machine.
To connect other Delft-FEWS component to Log Analytics a custom connector can be defined. All Delft-FEWS components log files to the local file system.
...
Infrastructure monitoring of the Delft-FEWS Virtual Machines is done with Azure Monitoring Services. This requires installing the IaaSDiagnostics extension on the virtual machine.
Azure Service Bus
It is possible to configure the The Delft-FEWS System Alerter can be configtured to sent send events to the Azure Service Bus. This allows triggering external applications (for example a azure an Azure function) based on events in the Delft-FEWS system. For more information, see: Azure Service Bus Alerts
...
Deltares has successfully completed Delft-FEWS projects in the cloud with virtual machines using standard installation scripts, using virtual machines with Azure ARM templates and AWS Elastic Beanstalk. For practical reasons, will keep our requirements / installation instructions as cloud neutral as possible.
ARM templates
Example ARM templates have been provided by MDBA and can be found here: MDBA ARM templates download
...
For more info contact Delft-FEWS product management.