With data files becoming larger, in combination with further processing and/or moving files exported from Delft-FEWS it can occur that downstream processes already try to access the exported data before the export has finished. Simultaneous access of files by different processes can lead to errors and crashes which are hard to reproduce and can influence the stability and robustness of downstream processes. Therefore, this page explains how Delft-FEWS deals with exports of large files and how you can set up your downstream processes for subsequent actions on the exported files.
Generic solution
When Delft-FEWS writes a file during an export it will always append the following:
- A tmp prefix
- A *.new extension
For example, tmpLargeFile.nc.new Thus you can have your downstream process checking for files with a tmp prefix and/or *.new extension and ignore those files.
After writing Delft-FEWS will rename the file by removing the prefix and additinal extension tmpLargeFile.nc.new → LargeFile.nc
Types of exports from Delft-FEWS
The above described generic solution is implemented for the following exports that are available in Delft-FEWS:
| Module | Prefix | Extension | Example | Description/Explanation |
|---|---|---|---|---|
| Export module | tmp | .new | NETCDF-CF_GRID: tmp202401010000_LargeFileTest.nc.new pi 1.31: tmp202401010000_LargeFileTestPiXml.xml.new pi_ascii_map_stack: tmpGfs0007.231.new asciigrid: tmp202412190000_LargeFileTestAscii.txt.new | Written to the export directory with tmp prefix and .new extension, after writing is complete, the file will be renamed to the original name. |
| General adapter export | None | None | LargeFileGA.nc | Files exported from the general adapter are expected to be used by the model, and not picked up by any other software package to copy/process. Therefore, no tmp prefix/.new extension is used. |
| General adapter DumpFiles | tmp | .new | tmpLargeFileExportGADumpFile_GeneralAdapter_2024.01.01.00.00.00.zip.new | Written to the DumpFileDir with tmp prefix and .new extension, after writing is complete, the file will be renamed to the original name. |
| General adapter ShutDownActivities (zip) | tmp | .new | tmpLargeFile.zip.new | Written to the zip file with tmp prefix and .new extension, after writing is complete, the file will be renamed to the original name. |
| Archive export | None | .tmp | LargeFileArchive.nc.tmp | Files exported to the Archive are written in the temp folder, before being moved to the Archive. When written they will have an .tmp extension, the file will be renamed after writing is finished. Only then will it be move to the Archive directory. You should never try to move files from the temp folder of Delft-FEWS. |
| Report module | None | None | Report.txt/Report.html | |
| Manual export (File > "Export timeseries") | tmp | .new | tmpLargeManualExport.xml.new tmpLargeManualExport.csv.new | Written to the selected directory with tmp prefix and .new extension, after writing is complete, the file will be renamed to the original name. |
