Warning |
---|
Important architectural changes concerning MC workflows in 2017.02: For 2017.02, MC workflows such as MC_MarkedRecordManager, MC_Synchronisation and MC_RollingBarrel have been made obsolete and are automatically removed by the DatabaseInitialization tool. These components are now subprocesses of the Master-Controller and run continuously.The only remaining MC workflow shall be the Mc SystemAlerter. The McRollingBarrel runs every 10 minutes by default, and its frequency can be controlled by editing the fews.master.mc.conf and restarting the MasterController. Code Block |
---|
<rollingbarrel>
<heartbeat time="60" /> |
In 2017.02 Mc Synchronisation synchLevels can be tuned using the mcSynchronisation xml as below, the used synchlevels will be reported mastercontroller_log.txt in DEBUG mode. Code Block |
---|
<remotemc mcid="...">
<database>
...
</database>
<mcSynchronisation>
<recordtype type="TimeSeries">
<modifier>
<synchlevel level="0"/>
<synchlevel level="1"/>
<synchlevel level="2"/>
<synchlevel level="3"/>
<synchlevel level="4"/>
<synchlevel level="5"/>
<synchlevel level="6"/>
<synchlevel level="7"/>
<synchlevel level="8"/>
<synchlevel level="11"/>
<synchlevel level="16"/>
</modifier>
</recordtype>
</mcSynchronisation> |
McSynchronization from a specific MC in 2017.02 can be disabled by outcommenting the remotemc section and restarting the MasterController. Code Block |
---|
<!--
<remotemc mcid="...">
...
</remotemc>
--> |
See 13 RollingBarrel_FSS (Compact index and cache files) - EOL for the FSS maintenance workflows. |
Introduction
In order for the master controller (and Forecasting Shell Server) to run properly several maintenance workflows must be scheduled. These workflows make sure the database does not overflow (they run the rolling barrel) and that configuration marked to be removed using the Configuration Manager actually gets deleted. In addition, each FSS must also run a rolling barrel task, usually once a day. This task runs the rolling barrel on the FSS and deletes configuration marked for deletion. Also the cache files are compacted which is essential for performance and preventing OutOfMemory errors. Both for synchronisation as for direct database access this maintenance of the cache files is important for optimal performance.
- MC_MarkedRecordManager - Removes records marked for deletion. This removes static entries only.
- MC_Synchronisation - Synchronise between several master controllers, only needed on duty standby systems
- MC_RollingBarrel - Removes expired records from the MC database (e.g. timeseries, model states etc)
- MC_SystemAlerter - Optional, used for triggering emails on log event codes.
- RollingBarrel_FSS (one for each FSS), recommended up to 2014.01 - synchronises the localDatastore and removes expired records from the FSS database. RollingBarrelFSS won't work unless module descriptors file is available and properly configured. Please use Compact_FSSCache.xml instead from 2014.02 onwards.
- Compact_FSSCache.xml (one for each FSS), preferred worfklow replacing RollingBarrel_FSS since 2014.02 .
Scheduling the workflows using the Administrator Interface (AI)
...
...
Info |
---|
Please note that the Rolling Barrel for the FSS is a 'real' workflow (it can be run stand-alone) and should be uploaded using the Configuration Manager. |
Info |
---|
Master Controller related tasks and workflows do not require upload of workflows anymore but are automatically created. They can be scheduled as described here. Following the steps described below to upload MC worklows is no longer requiredSee the Admin Interface User Guide how to schedule them. The Rolling Barrel for the FSS is a 'real' workflow, and still requires a workflow file (which should be uploaded using the Configuration Manager) |
...
Now the workflows can be scheduled using the Forecast Tasks -> Schedules tasks -> Schedule new task menu item. When scheduling you should make sure you specify the appropriate properties file for each workflow type:
![](/download/attachments/8683586/ai4.png?version=1&modificationDate=1190588184000&api=v2)
Listing of workflows and properties
MC:MarkedRecordManager
Suggested scheduling interval: Every 30 minutes, depending on the needs of the system.
Code Block |
---|
|
<?xml version="1.0" encoding="UTF-8"?><Blank/>
|
Code Block |
---|
|
<?xml version="1.0" encoding="UTF-8"?><Blank/>
|
MC:Synchronisation
Suggested scheduling interval: Every 5 minutes, depending on the needs of the system.
Code Block |
---|
|
<?xml version="1.0" encoding="UTF-8"?><Blank/>
|
Code Block |
---|
|
<?xml version="1.0" encoding="UTF-8"?><synchronisation>
<synchparams>
<!--
The remote MC name must be replaced with the actual name of the
remote master controller. (It references the remote MCId in the
MC configuration
-->
<remotemcid name="EAMIMC01"/>
<synchid id="MainMcSynch"/>
</synchparams>
<recordtype type="WorkflowFiles"/>
<recordtype type="DefaultWorkflowFiles"/>
<recordtype type="SystemConfigurations"/>
<recordtype type="DefaultSystemConfigurations"/>
<recordtype type="ModuleInstanceConfigs"/>
<recordtype type="DefaultModuleInstanceConfigs"/>
<recordtype type="ModuleInstanceDatasets"/>
<recordtype type="DefaultModuleInstanceDatasets"/>
<recordtype type="SysReportTemplates"/>
<recordtype type="DefaultSysReportTemplates"/>
<recordtype type="SysReportStyles"/>
<recordtype type="DefaultSysReportStyles"/>
<recordtype type="RegionConfigurations"/>
<recordtype type="DefaultRegionConfigurations"/>
<recordtype type="DisplayConfigurations"/>
<recordtype type="DefaultDisplayConfigurations"/>
<recordtype type="FlagConversions"/>
<recordtype type="DefaultFlagConversions"/>
<recordtype type="IdMaps"/>
<recordtype type="DefaultIdMaps"/>
<recordtype type="UnitConversions"/>
<recordtype type="DefaultUnitConversions"/>
<recordtype type="ReportTemplates"/>
<recordtype type="DefaultReportTemplates"/>
<recordtype type="ConfigManConfigurations"/>
<recordtype type="DefaultConfigManConfigurations"/>
<recordtype type="CorrelationEventSets"/>
<recordtype type="DefaultCorrelationEventSets"/>
<recordtype type="CorrelationTravelTimes"/>
<recordtype type="DefaultCorrelationTravelTimes"/>
<recordtype type="ModuleParameters"/>
<recordtype type="DefaultModuleParameters"/>
<recordtype type="MapLayers"/>
<recordtype type="DefaultMapLayers"/>
<recordtype type="Icons"/>
<recordtype type="DefaultIcons"/>
<recordtype type="ReportImages"/>
<recordtype type="DefaultReportImages"/>
<recordtype type="RootConfigFiles"/>
<recordtype type="DefaultRootConfigFiles"/>
<recordtype type="PiServiceConfigurations"/>
<recordtype type="DefaultPiServiceConfigurations"/>
<recordtype type="PiClientConfigurations"/>
<recordtype type="DefaultPiClientConfigurations"/>
<recordtype type="WhatIfScenarios"/>
<recordtype type="Tasks"/>
<recordtype type="TaskRuns"/>
<recordtype type="FewsSessions"/>
<recordtype type="MCCpts"/>
<recordtype type="LogEntries"/>
<recordtype type="ModuleInstanceRuns"/>
<recordtype type="TaskRunCompletions"/>
<recordtype type="ArchiveMetadata"/>
<recordtype type="CurrentModuleInstanceRuns"/>
<recordtype type="RecordsPendingDeletion"/>
<recordtype type="Reports"/>
<recordtype type="WarmStates"/>
<recordtype type="ColdStates"/>
<recordtype type="TimeSeries">
<modifier>
<synchlevel level="0"/>
<synchlevel level="1"/>
<synchlevel level="2"/>
<synchlevel level="3"/>
<synchlevel level="4"/>
<synchlevel level="5"/>
<synchlevel level="6"/>
<synchlevel level="7"/>
<synchlevel level="8"/>
</modifier>
</recordtype>
<recordtype type="ConfigRevisions" />
<recordtype type="ConfigRevisionStore" />
<recordtype type="ThresholdEvents"/>
<recordtype type="Modifiers"/>
<recordtype type="RecordsPendingUpdateExpiry"/>
</synchronisation>
|
MC:RollingBarrel
Suggested scheduling interval: Every 30 minutes, depending on the needs of the system.
Code Block |
---|
|
<?xml version="1.0" encoding="UTF-8"?><Blank/>
|
Code Block |
---|
|
<?xml version="1.0" encoding="UTF-8"?><rollingbarrel action="remove,setexpire">
<table name="LogEntries"/>
<table name="Reports"/>
<table name="TaskRuns"/>
<table name="TimeSeries"/>
<table name="SystemStatusDigests"/>
<table name="Tasks"/>
<table name="ThresholdEvents"/>
<table name="WarmStates"/>
<table name="RecordsPendingDeletion"/>
<table name="ImportStatus"/>
<table name="Modifiers"/>
<table name="RecordsPendingUpdateExpiry"/>
</rollingbarrel>
|
RollingBarrel
Anchor |
---|
| FSSRollingBarrel |
---|
| FSSRollingBarrel |
---|
|
Suggested scheduling interval: Every 360 minutes, depending on the needs of the system.
Please note that with the new release a SINGLE rolling barrel workflow should be scheduled. When scheduling this should be set to run as "all-staggered" in the admin interface
Contents of the workflow RollingBarrel 1.00 default.xml
Code Block |
---|
|
<?xml version="1.0" encoding="UTF-8"?>
<workflow version="1.1" xmlns="http://www.wldelft.nl/fews"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.wldelft.nl/fews
http://fews.wldelft.nl/schemas/version1.0/workflow.xsd">
<!--Rolling barrel workflow-->
<activity>
<runIndependent>true</runIndependent>
<moduleInstanceId>RollingBarrel</moduleInstanceId>
</activity>
<!--Delete records pending deletion-->
<activity>
<runIndependent>true</runIndependent>
<moduleInstanceId>MarkedRecordManager</moduleInstanceId>
</activity>
</workflow>
|
Sample contents of
...
Compact_FSSCache.xml (Recommended instead of RollingBarrel_FSS since 2014.02)
Code Block |
---|
|
<?xml version="1.0" encoding="UTF-8"?>
<taskProperties<workflow version="1.1" xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<description>Rolling Barrel on FSS</description>
<workflowId>RollingBarrel</workflowId>
<taskSelection>
<scheduledTask>
<schedulingPeriod>
<startDate>2006-06-03T03:30:00.000Z</startDate>
<endDate>2050-08-11T12:00:00.000Z</endDate>
</schedulingPeriod>
<schedulingInterval multiplier="360" unit="minute"/>
<schedulingTime0Shift multiplier="0" unit="minute"/>
</scheduledTask>
</taskSelection>
<forecastPriority>Normal</forecastPriority>
<makeForcastCurrent>false</makeForcastCurrent>
<makeStateCurrent>false</makeStateCurrent>
</taskProperties>
|
Sample contents of properties for upload in Admin Interface.
Code Block |
---|
xml | xml | <?xml version="1.0" encoding="UTF-8"?>
<taskList xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/taskListworkflow.xsd" xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<task><activity>
<taskStatus>P</taskStatus>
<runOnFailOver>true</runOnFailOver>
<taskProperties>
<workflowId>RollingBarrel_FSS</workflowId>
<taskSelection><!--Removes rows from cache and index files that no longer exist in the database, every FSS has its own cache and index files.
For legacy configuration the "nl.wldelft.fews.system.plugin.rollingbarrel.RollingBarrelModule" class is redirected to this activity
There is no need <scheduledTask>
to schedule the rolling barrel on a FSS, this happens automatically on <schedulingPeriod>
<startDate>2010-01-01T00:00:00.000Z</startDate>the FSS.
<endDate>2050-08-11T12:00:00.000Z</endDate>
</schedulingPeriod>
<schedulingInterval unit="hour" multiplier="6"/>
<predefinedActivity>compact </scheduledTask>
cache files</predefinedActivity>
</taskSelection>
<forecastPriority>Normal</forecastPriority>
</taskProperties>
</task>
</taskList>
|
...