With Delft-FEWS systems getting larger and more data stored in the central database it is sometimes good to review the memory settings of your Operator Client and Forecasting Shell client components. Changing these settings to more realistic values can greatly improve the performance of your system. The default settings of the Delft-FEWS settings are tuned for medium size systems, these default settings can be changed in the root configuration files (this only applies to Delft-FEWS version 2018.02 and later). There are two files that are used for these settings:
- clientConfig.xml (for OC, FSS and SA)
- global.properties (for OC, FSS and SA)
Client config.xml (for Delft-FEWS version 2018.02 and later)
The clientConfig.xml file can be used for memory allocation (RAM) and disk space allocation for local cache files. The default memory setting is <jvmOption>-Xmx1024m</jvmOption>. Increased memory allocation settings need to be configured using <jvmOption></jvmOption> elements.
<jvmOption>-Xms1024m</jvmOption> <jvmOption>-Xmx2048m</jvmOption>
The flag -Xmx specifies the maximum memory allocation, while -Xms specifies the initial (and minimum) memory allocation. In this example the initial memory allocated is 1024 MB and the maximum memory is 2048 MB. The value for -Xmx is the most significant value of the two, the value for -Xms has some influence on the startup time of the application. The choice of -Xmx should be balanced. A too small -Xmx memory setting has a lower memory footprint on an Operating System with limited memory, but may cause OutOfMemoryErrors to appear.
When rolling out a Delft-FEWS application, it is best practice is to test the application with a low memory setting (e.g. 1024MB) and perform some performance tests with intensive viewing of grids first. If the application's performance becomes sluggish when using a low -Xmx setting, it is likely that the Java garbage collection memory management process is responsible. In that case it is time to tune to a higher -Xmx setting. For testing a configuration with 64 bit Java (from FEWS 2018.02), -Xmx1500m is a good starting point, but -Xmx2048m is also commonly used, higher numbers can be used if necessary. The final rolled out Delft-FEWS application should be able to run smoothly with the specified -Xmx value.
To analyze the currently used amount of memory in the user interface, you can use the F12 debug functionality to log the memory used by the Operator Client (from FEWS 2021.01):
- Open the Logs panel and press F12
- Select "P: database" and "log database info"
The logs will show similar information as listed below. In this example the -Xmx value is set to -Xmx1500m. The heap memory in the logs is related to the Xmx memory.
Wed 16-03-2022 10:46 INFO - max memory 1.9 GB
Wed 16-03-2022 10:46 INFO - allocated memory 1.9 GB
Wed 16-03-2022 10:46 INFO - native memory 247 MB
Wed 16-03-2022 10:46 INFO - java heap memory (without cache) 842 MB
Wed 16-03-2022 10:46 INFO - java non heap memory 188 MB
Wed 16-03-2022 10:46 INFO - java threads: count 63 / memory 41 MB
Wed 16-03-2022 10:46 INFO - java cache memory: 147 MB (released on memory shortage)
The tooltip on the FEWS Explorer memory element also provides information on memory usage.
From these figures and the available RAM of the hardware it is possible to give some guidance for the value of the -Xmx memory allocation.
- The Xmx is configured as 1500 MB in this example. The "java heap memory (without cache)" is the Xmx memory used; = 842 MB. This means there is still memory left.
- Additional to the heap memory, there is the "non heap memory" and the "native memory"; total is about 400 MB.
- The max memory used is the Xmx + this additional memory; this is 1500 + 400 = 1.9 GB.
- There is also java threads memory in use. When more displays are opened there are also more threads and more memory is used. This is normally in the order of 50-100 MB.
With these settings of -Xmx1500m and 400 MB additional memory, the FEWS application will use about 2GB of RAM.
Some additional information on the memory settings and the use of this for an OC or FSS.
- If you reserve more RAM than available on a VM/Computer in either Windows or Linux, the performance will drop considerably as the Operating System will start swapping memory.
- We advise to use Xms setting on linux with enough memory reserved initially so the OC/FSS will run without problems. On linux it is better to set the Xms equal to Xmx.
- If you have more RAM on a VM it is always better to increase the Xmx. The unused heap (Xmx) will be used for temporarily caching time series info, this will increase performance.
- Do not make the Xmx larger than 32 GB.
Especially on FSS, when you run external models it is important that there is more RAM available.
- If you run external models on FSS additional memory is used. Models take on average 500 MB of RAM.
- When you start using larger models, it is possible you need more RAM outside the reserved FEWS allocated memory.
In the clientConfig.XML there is another element that can be used to improve performance of a FSS or OC, this is the size of the local time series cache files.
<localCacheSizeMB>2000</localCacheSizeMB>
- The default of the localCacheSize is 500 MB.
- The localCacheSizeMB is the cache size of the data in the FEWS \Temp folder. This prevents the OC or FSS to download the data several times during a run or when the OC is open. You can set this to 2000 MB or even 10000 MB when needed.
- Setting this CacheSize to larger values than 500 MB will not have a negative impact on the performance, make sure there is enough disk space.
Changes to global.properties
In the global.properties it is possible to set the timeSeriesWriteCacheSizeMB size. This is the local cache for writing time series to the database during a workflow run.
timeSeriesWriteCacheSizeMB=100
- The default is 10 MB
- Best to increase this to 100MB for the FSS, this can reduce the number of small records in the time series database table.
- The database will flush the buffer to the time series database tables once the cache size is reached.
- When a workflow produces less that 100 MB time series data, all times series values are stored at the end of the workflow run.
