Reference Architecture
The following diagram is a typical architecture that is used in the Azure cloud when deploying a Single MC system using traditional virtual machines.
Installation of Operator Clients
| non-exhaustive list of options | remarks |
|---|---|
| Azure Virtual Desktop | See also: Azure Virtual Desktop for the Operator Client |
| database http proxy using SSL | |
| Citrix | Can be integrated with most cloud providers |
Use of managed services
There is no actual requirement for the Delft-FEWS components to use managed services. Managed services can be used as long performance is not affected. As an example, customers that are using SQLServer database replication between different geographical locations reported database timeouts. In response, we've improved our software, removed database indexes and added reconnection strategy for these problems. Since we expect Delft-FEWS users add many more simultaneous running Forecasting Shell servers in the future, we expect / foresee more challenges in this area. It is much better not to use the automated placement of database indexes.
How to deal with (incoming, outgoing) data feeds
Important cost variables
An estimate for the cost for a basic/medium-sized Delft-FEWS system in the cloud would be around 12k€- 15k€. For Azure use the Azure calculator. The estimate differs per Delft-FEWS system.
Data size and egress
See How to calculate the right Azure outbound capacity and choose the best egress option.
Data storage
Data can be stored in different cloud solutions each with it's own price and functionalities. Depending on your requirements you can add managed disks to your machine or a dedicated storage solution like Azure files.
High Availability and scaling
Scaling up of Azure components can be done manually. Some possibilities are:
- Create a new Virtual machine by restoring an existing backup.
- Use a Azure Automation Runbook in combination with Desired State Configuration (DSC) to install the Deflt-FEWS software
- Use a devOps solution tor creating a new virtual machine by deploying ARM templates of the components.
For each Delft-FEWS component the scaling capabilities are discussed. Delft-FEWS does not auto scale components itself. If required, auto scaling must be managed by the Azure environment. Azure has autoscaling features like VM Scale sets (which require a custom VM image), App Services and Kubernetes to support autoscaling.
Availability Set
In Azure all Virtual machines can be deployed as part of an Availability Set. This will assure that VMs that are part of the same Availability Set won’t be affected at the same time by Azure maintenance, windows updates etc.
High available scenarios
On a high level the different scenarios for running Delft-FEWS in a highly available manner when the primary Azure region fails, from most high available to least available
Scenario | Description | Disaster recovery if primary regions fails | Manual interventions | Recover Time/Data Loss for Delft-FEWS components |
Dual MC in 2 Azure regions | 2 different databases in two regions, databases will be synchronized by Delft-FEWS. Use Geo redundant storage for file shares. | Automatically by Delft-FEWS. | Operator clients need to reconnect to secondary MC. This has been pre-configured. | No Recovery Time. (Hot Stand By) Data loss is minimal and can be restored by rerunning workflows. |
Single MC with Database in 2 Azure regions | Azure auto failover group Managed Instance Database with Azure Site Recovery for VMS. Use Geo redundant storage for file shares. | Use Azure Site Recovery to restore VMS in secondary region. Synchronized database will be made primary by Azure.
| Azure Site Recovery has to be enabled. | Recovery Time depends on the time it takes for Microsoft to enable ASR. Data loss is minimal and can be restored by rerunning workflows. |
Single MC with locally redundant Database | Single database with Delft-FEWS components | Database backups have to be restored to a new database in a secondary region. VM backups have to be restored to a secondary region | Database configurations on restored VMs have to be adjusted to new DB. Delft-FEWS configuration has to be adjusted to use new Database. | Recovery time is a few days. Data loss depends on the backup schedules. |
Disaster Recovery
This section will describe possible Disaster Recovery solutions for the different parts of a Virtual Machine based Azure system:
- Azure Managed Database
- Virtual machines
- Azure Storage Accounts
It is also possible to use Azure Site Recovery. This is the disaster recovery solution offered by Azure. Depending on the database solution chosen, it may still be required to do post recovery configurations.
Azure Managed Database Disaster Recovery
Azure Managed databases are backed up. Make sure database backups are stored using geo redundancy. In case of a disaster, the database needs to be restored to another data center in the same region or to another region in case the whole region is affected. Since restoring a backup to another environment changes the database hostname, this impacts all Delft-FEWS components that refer to the database. This will impact both the server components and the Operator Client.
Virtual machines disaster recovery
Azure backup can be used to restore a Virtual Machines that are backed up using a storage account that support geo-redundant storage (GRS).
VM recovery with new database
- Update the MC / FSS ENV variables with the changed database connection, username and password.
- Update the global.properties of the FSS in the Delft-FEWS configuration and upload them with the Config Manager. Typical changes are: URL to the archive server, location of the Azure File Shares for imports and the archive.
Web Services: Update the ENV variables with the changed database connection.
Archive Server: update the location of the Azure File Share in the archive configuration file.
Azure File Shares Archive recovery
In case of a disaster or a human error with the archived data on an Azure Files Share, Azure backup can be used to restore the archived files. It is recommended to use a geo redundant backup for archived data.
Monitoring and Alerting
Log Analytics
Using the Log Analytics service (part of Azure Monitoring) of Azure allows collecting log events from the different Delft-FEWS components. The Master Controller, Forecasting Shell Server and Admin Interface support sending error log events to the Windows Event Logger or Linux syslog. Log Analytics can be connected to the Windows Event Logger and syslog. This requires installing the MicrosoftMonitoringAgent extension on the virtual machine. To connect other Delft-FEWS component to Log Analytics a custom connector can be defined. All Delft-FEWS components log files to the local file system.
Malware protection
To protect the VMs from malware, it is possible to install the IaaSAntimalware extension on the Virtual Macghine. Enabling malware protection may have a negative impact on the performance of the Delft-FEWS components. It may be required to add some of the Delft-FEWS directories to the exclusion list.
Azure Infrastructure Monitoring
Infrastructure monitoring of the Delft-FEWS Virtual Machines is done with Azure Monitoring Services. This requires installing the IaaSDiagnostics extension on the virtual machine.
Azure Service Bus
The Delft-FEWS System Alerter can be configtured to send events to the Azure Service Bus. This allows triggering external applications (for example an Azure function) based on events in the Delft-FEWS system. For more information, see: Azure Service Bus Alerts
DevOps
It is possible to use DevOps techniques to automaticlly test and deploy Delf-FEWS configuration changes in the cloud. See the following example on how to use Azure DevOps in combincation with the Workflow Test Runner and the Admin Interface API to automatically deploy a tested configuration:
