...
- Consistent folder structure
- Readable scripts
- Documentation
- File formats and data organization
- Version of your code and data
Since scripting becomes increasingly important in our projects, the rules of software development also start to apply to us. This has the downside that it requires extra effort from us, hydrologists, to learn new tools, but the upside is that there already is a wealth of properly tested tools and documentation available from the software development world. Having reproducible code is one of these important things, for which version control systems have been developed. It is very useful to use these systems in our projects since they provide the following advantages:
You keep track of the history of a project. In this way you keep a journal of decisions made in a project.
In this chapter we will give a quick introduction to versioning and point to some more elaborate guides.
If you mess up something, you can always revert to a previous state.
It allows for collaborative development, where individuals can create their own branch to safely work on new developments and later merge their changes. - Manage the worklow
A collection of scripts doesn't make a project reproducible per se. Quite often the output of script A serves as the input of script B. The order in which you execute scripts therefore matters: after a data update, first script A needs to be run, after which script B needs to be run, while an independent script C does not need to be re-run. Running script B before script A would not result in an update of the end result. Data dependencies between scripts can get complex quickly when multiple data sources exist, which commonly happens in projects. Managing workflows therefore is a common task in your numerical modeling project is a common task but not always as straight forward straightforward as it seems.
Workflow managers can help with this. In this chapter we will give a quick introduction to using the worklfow workflow manager snakemake Snakemake and point to some more elaborate guides.
...