Exporting large files and further processing

With data files becoming larger, in combination with further processing and/or moving files exported from Delft-FEWS it can occur that downstream processes already try to access the exported data before the export has finished. Simultaneous access of files by different processes can lead to errors and crashes which are hard to reproduce and can influence the stability and robustness of downstream processes. Therefore, this page explains how Delft-FEWS deals with exports of large files and how you can set up your downstream processes for subsequent actions on the exported files.

Generic solution

When Delft-FEWS writes a file during an export it will always append the following:

• A tmp prefix
• A *.new extension

For example, tmpLargeFile.nc.new Thus you can have your downstream process checking for files with a tmp prefix and/or *.new extension and ignore those files.

After writing Delft-FEWS will rename the file by removing the prefix and additinal extension tmpLargeFile.nc.new → LargeFile.nc

Types of exports from Delft-FEWS

The above described generic solution is implemented for the following exports that are available in Delft-FEWS: