You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 11 Next »

Function:

runInLoopParallelProcessorCount, set the amount of cores available to Delft-FEWS when running ensemble workflows

Module Name:

runInLoopParallelProcessorCount

Where to Use?

global properties file

Why to Use?

to speed-up ensemble runs on multi core machines

Description:

The runInLoopParallelProcessorCount en try in the global properties files indicated the number of cores Delft-FEWS may use when running ensemble members in a loop

Preconditions:

2009-02 release, multi core cpu or multi cpu computer

Outcome(s):

speed-up of the computations

Scheendump(s):

link to attached screendump(s) for displays only

Remark(s):

The speedup that may be obtained is highly dependent on the type of module you are running

Available since:

DelftFEWS200902

Contents

Overview

Delft-FEWS can split ensemble workflows (that have the runInLoop element set to true) over multiple cores. Based on the available amount of cores a number of queues is made, one for each core. When running the activity the different ensemble members are added to the different queues. An example of a workflow that can use this feature is shown below:

	<activity>
		<runIndependent>true</runIndependent>
		<moduleInstanceId>MOGREPS_Spatial_Interpolation</moduleInstanceId>
		<ensemble>
			<ensembleId>MOGREPS</ensembleId>
			<runInLoop>true</runInLoop>
		</ensemble>
	</activity>

Configuration

By default Delft-FEWS will only use one core and all tasks are run one after another. To enable the parallel running of ensemble members the runInLoopParallelProcessorCount entry must be set in the global properties file. Here you either specify the number of cores to use or specify 100 to use all available cores.

In the global properties

Config Example

 
# to use all available cores/cpu's:
runInLoopParallelProcessorCount=100

For all internal Delft-FEWS modules that have been tested no changes are needed to the configuration. For external modules that are run using the General adapter some changes may be needed to the configuration.

Tested modules

Module

Remarks

Transformation (old)

Test ok

Interpolation

Test ok. Interpolation via DLL not tested

TransformationModule (new)

Test ok. Interpolation via DLL not tested

pcrTransformation

Chrashes FEWS completely. Apparently not tread-save

Sample input and output

Sample input and output

Error and warning messages

Description of errors and warnings that may be generated

Error:

Error message

Action:

Action to fix

Known issues

  • Running modules in parallel means you will use more memory

In some cases, the increase in speed may be very limited. Although it depends on a case by case basis the following simple rules may be used to determine the experted increase in execution speed:

  • execution time of an individual module <= 1 sec: expected increase < 20%
  • execution time of an individual module > 1 sec < 10: expected increase between 20 and 50%
  • execution time of an individual module > 10 sec: expected increase > 50 % and < 100 %

The percentage given in the list above should be scaled using the amount of cores used. The 100% in the example above is a two-fold increase using two cores.

Other factors that influence this are the amount of data being retrieved and store in the FEWS database in relation to the total execution time and (in the case of an external module) the amount of data written to and read from the file system.

Related modules and documentation

Links to related parts of the system

Technical reference

Entry in moduleDescriptors:

none

Link to schema:

none
  • No labels

Disclaimer