Introduction
It is possible to automate testing of Delft-FEWS configurations and use CI/CD technology to facility automatic deployments of changed Delft-FEWS configurations.
At a high level the following steps can be followed:
- Use the Delft-FEWS Workflow Test Runner to run workflows.
- Use the python tooling to run test cases and generate local test reports.
- Use Azure DevOps to create a build pipeline that can run the python tests on configuration changes.
- Use Azure DevOps Release Pipelines to automatically deploy a new configuration and validate the live system status.
Architectural Overview
The different components that are required to run the workflow test runner in the context of Azure DevOps are shown in the following architecture:
Workflow Test Runner
The Workflow Test Runner is a general tool that can run workflows using a Standalone Delft-FEWS on the command line. See also: How to configure a workflow test run. Running workflows from the command-line
Using this tool, a well-known set of files can be used to run a workflow. The results of the workflow can be verified with previous results.
A JUnit XML report can be generated to create a test report and integrate into Continuus Integration platforms. The XML report can be published to Azure DevOps as a Test Artifact.
Python
Python can be used (but is not required), to run the workflow test runner, merge all test reports and generate a HTML report with the test results.
When completed, files that have been generated with the configured Delft-FEWS workflow, will be compared with a well-known result file. In case of any differences, the test will fail and reported in a test report file.
Azure DevOps build pipeline
Once all tests are created, they can be used in a build pipeline. Azure DevOps is a CI/CD platform that can be used to build and deploy software. See also: https://dev.azure.com
The build pipeline will be triggered if a change to the Delft-FEWS configuration is committed.
- The workflow test runner tests will be run with this new configuration.
- On any failures, the failed tests will be reported, and the pipeline stops.
- If all tests are successful, the build pipeline will provide the test configuration to the release pipeline.
The following is an example of a build pipeline configuration in Azure Devops:
Azure DevOps release pipeline
In a release pipeline, multiple stages can be defined. We assume a preprod and prod stage will be configured.
In the preprod stage, the new Delft-FEWS configuration will be uploaded to the Delft-FEWS database using the Admin Interface API. Once the upload completes, some live system tests will be performed.
- The Admin Interface API will be used to do a health check. For example:
- there should not be any ERRORS in the logs.
- The MC and all Forecasting shell servers should report they are fine.
- If any check fails, the pipeline will end.
- If all checks are successful, the pipeline will move to the next stage: prod.
- Optionally a gate can be configured that required a manual approval to deploy the configuration to production. An email can be sent to the responsible person.
If the gate is passed, the admin interface API will be used to deploy the approved configuration to production.
The following is an example of a release pipeline in Azure DevOps. For an example on how to deploy a configuration with the admin interface API see: Upload Delft-FEWS configuration zip with Azure DevOps
The following Python script is an example on how to use the workflow test runner, merge all xml results and publish the results as an HTML report using the junit2html tool:
# # Python 3 is required. # # python3 workflowtests/scripts/workflowtestrunner.py # # The following python packages are required. # # pip install junitparser # pip install junit2html # pip install requests # -*- coding: utf-8 -*- import sys import os import shutil import requests, zipfile, io from datetime import datetime from junitparser import JUnitXml # Make sure the basebuild is available. def download_base_build(): r = requests.get( "https://public.url.with.basebuild.zip") z = zipfile.ZipFile(io.BytesIO(r.content)) z.extractall("bin") def absolute_file_paths(directory): path = os.path.abspath(directory) return [entry.path for entry in os.scandir(path) if entry.is_file()] def merge_xml_reports(reportDir): junit_report_dir = f"{reportDir}/junit" report = "" junit_files = absolute_file_paths(junit_report_dir) for file in junit_files: if file.endswith(".xml"): xml = JUnitXml.fromfile(file) if report == "": report = xml else: report = report + xml report_xml_file = f"{reportDir}/junit/report-all.xml" report.write(report_xml_file, True) return report_xml_file def main(argv): working_dir = os.getcwd() if not os.path.isdir(f"{working_dir}/bin"): download_base_build() report_dir = f"{working_dir}/report" report_html_file = f"{report_dir}/report.html" shutil.rmtree(report_dir, ignore_errors=True) workflow_tests_command = f"{working_dir}/bin/windows/Delft-FEWSc.exe -Dregion.home={working_dir}/region_home -DautoRollingBarrel=false -Xmx20484m -DoldPID=13096_1555070573358 -Djava.locale.providers=SPI,JRE -Dstart.time=1630454400000 -Djava.library.path={working_dir}/bin/windows -Wvm.location={working_dir}/bin/windows/jre/bin/server/jvm.dll -Wclasspath.1={working_dir}/region_home/patch.jar -Wclasspath.2={working_dir}/bin/*.jar -Wmain.class=nl.wldelft.fews.system.workflowtestrun.WorkflowTestRun -Warg.1={working_dir}/region_home -Warg.2={working_dir}/workflowtests/configuration" print("%s", workflow_tests_command) # Run the workflow tests. os.system(workflow_tests_command) # Merge all xml reports. report_xml_file = merge_xml_reports(report_dir) os.system(f"junit2html {report_xml_file} {report_html_file}") # ------------------------------------------------------------------------------ if __name__ == "__main__": start_time = datetime.now() print('Start Workflow Test Runner: %s\n' % start_time.strftime("%Y-%m-%d %H:%M:%S")) main(sys.argv[0:]) print('End : %s' % datetime.now().strftime("%Y-%m-%d %H:%M:%S")) print('Finished')
An example of a Azure Devops Build Pipeline, can be found in the following YAML definition.
# Build Pipeline to test Delft-FEWS configurations using the Workflow Test Runner # Any commit in te configuration, will trigger the pipeline. # Only trigger the build on commits in the region_home folder. trigger: branches: include: - main paths: include: - region_home pool: vmImage: windows-2019 steps: - checkout: self path: fews # Checkout the configuration repo to the fews folder. - checkout: git://MDBADelftFewsConfigurationPipeline/MDBADelftFewsConfigurationPipeline path: testfiles - task: UsePythonVersion@0 inputs: versionSpec: '3.10' - script: | echo %cd% pip install pytest pytest-azurepipelines pip install junitparser pip install junit2html pip install requests cd ../fews echo %cd% python workflowtests/scripts/workflowtestrunner.py displayName: 'Run Delft-FEWS Workflow Test Runner.' # Publish test results to Azure Pipelines - task: PublishTestResults@2 inputs: testResultsFormat: 'JUnit' testResultsFiles: '../fews/report/junit/report-all.xml' failTaskOnFailedTests: true # Optional #testRunTitle: # Optional #buildPlatform: # Optional #buildConfiguration: # Optional publishRunAttachments: true displayName: 'Publish Delft-FEWS Workflow Test Runner results.' - task: ArchiveFiles@2 inputs: rootFolderOrFile: '../fews/region_home/Config' includeRootFolder: true archiveType: 'zip' archiveFile: '$(Build.ArtifactStagingDirectory)/Config.zip' replaceExistingArchive: true displayName: 'Zip Delft-FEWS config.' - task: PublishBuildArtifacts@1 displayName: 'Publish Delft-FEWS Config.zip'