Github is an ideal place to work together with colleagues and also external parties. In the new organization https://github.com/Deltares-research, there are a few ways to work with outside collaborators:

1. Use PULL REQUESTS. Allow your repository to accept pull requests. This way collaborators do not require to be added (and managed) as 'Outside collaborators'. This works well for PUBLIC repositories.
2. Invite an outside collaborator to your organization and then assign them to your repository. For PUBLIC repositories no license is claimed for PRIVATE repositories 1 license is claimed. No additional costs for adding the same collaborator to multiple repositories.

Inviting 'outside collaborators' to your repository:

Login to GITHUB and go to the settings of your repository then continue to 'Collaborators and teams'. Here you can 'Add people'. In the popup form enter either the GITHUB username or the e-mails address of the user you wish to invite and set the role for this user.

The user will receive an invitation by e-mail and also under the users GITHUB settings in the section Organizations. The user must accept the invitation (either in e-mail or under settings) to become outside collaborator.

MyDeltares account

An outside collaborator is not required to authenticate through the MyDeltares identity provider and therefore does not require a MyDeltares account

Firstly, what is the difference between a Member and Outside-collaborator? 

Information that can help answer to this question can be found here: About conversion of organization members to outside collaborators. In short outside collaborators are more limited than members.

Inviting Members

Inviting users as members to your organization is managed from the 'People' menu in your Organization. Here outside-collaborators can be promoted to members or members can be invited directly. This is something you can probably not do yourself and you will have to send a request to: github-support@deltares.nl or on the public WIKI Submit a Question.

The user will receive an invitation by e-mail or will be able to Accept the invitation under users GITHUB settings in the section Organizations (see How to invite outside-collaborators question above).

On accepting the invitation the user will be asked to join the organization which will require that the users logs into MyDeltares:

If the user does not have a MyDeltares account the the user can create an account (free-of-charge) by following the link 'Create a MyDeltares account' and providing the requested information. 

I want to use my own Github account that I've used thus far (linked to my private e-mail address). Can I do that and if yes, how?

It is possible to use your existing GITHUB account. However if you work for Deltares, you will have to assign your Deltares email to your GITHUB account. All Deltares employees will be invited using their Deltares email address.

Problem

I want to request a new Github repository for my project or software product, but I do not know which of the two Deltares organizations I must choose: Deltares or Deltares-research

Solution

You do not need to choose anything, this will be done for you based on the information provided during the registration process.

What is the difference between the two organizations?

Deltares organization: This organization is meant to house all repositories that are linked to a Product Management Team (PMT) and contain production ripe software.

Deltares-research: All repositories that do not fit into the Deltares organization.

Problem

After your repository is moved from https://github.com/Deltares to the new organization https://github.com/Deltares-research, it is possible that you are required to re-authorize your account in one or more of your local applications, such as VS Code, TortoiseGIT or on the GIT command line.

Solution

Follow the following steps to refresh you authorization token:

1. Open the Credential Manager in Widows. ( type 'Credential Manager')
2. In Credential Manager select 'Windows Credentials'
   
   
   
3. Lookup the credentials for git:https://github.com and remove this.
4. Re-initialise the cred manager in git bash: git config --global credential.helper manager-core
5. Re-run git pull and follow the pop-up instructions to authenticate in a browser (which happened automatically for me with SSO).

No there is not.

When you have questions regarding the new GITHUB environment you can post them to the GITHUB support team. We can then try to connect you to people who already have experience with GITHUB. Otherwise we suggest you contact your department head for funding.

There is no fixed definition of 'production-ripe' software. It is up to the Product Management Teams (PMT) to decide if they wish to designate certain repositories / products as production ripe or as research. 

All repositories under the Deltares organisations should be linked to one of the PMT;s. If unsure about the status of your repository please contact the PMT and discuss with them where your repository fits best.

If you have been working on a software product for Deltares, but you have created the repository under your own personal Github account, you might want to move that repository to a Deltares Github organization. This can be achieved by transferring the ownership of your repository. 

Before transferring ownership, there are some preconditions that need to be met:

Once these preconditions have been met, you can start the transfer process by opening the Settings for your repository:

In the General section, scroll all the way down to the Danger Zone. Here you will find the option Transfer

As New owner you must choose Deltares-research. This option should be visible from the drop-down list.

In the bottom filed type the text stated in the message and press the button 'I understand, transfer this repository'

In the current Deltares organization, co-pilot has been enabled. There are 24 licenses available, all of which are currently taken. At present co-pilot is not enabled in the Deltares-research organization. This will change once the Deltares organization has been added to the Deltares enterprise.

How we plan to use co-pilot in the future and how we plan to manage the licenses has not yet been worked out. For more information please contact: github-owners@deltares.nl .

Apparently some users can no longer see their repositories in Zenodo. Even when their repositories have not been moved from the Deltares organization. Why this is the case is still unclear however the resolution is that one of the organization owners logs into Zenodo using ones Github account and activates the required repositories.

Deltares does not support the use of GITHUB's own Large File Storage (LFS) environment. However it is possible to connect your repository to the Deltares S3 bucket (MinIO) using GIT-LFS.

Please read the information on the following page: GitHub - How to connect your repository to a LFS?:

If you wish to connect your repository to the Data Version Control (DVC) tool, then please read the information on the following page: GitHub - How to connect your repository to a LFS?:

When you create a personal access token , this needs to be approved by one of the Github owners. To assure that this is done, you are requested to send an email to github.support@deltares.nl explaining your request.

Codecov is not updating anymore since it requires a token now. This can be fixed by replacing:

   
 - uses: codecov/codecov-action@v1

with:

 - uses: codecov/codecov-action@v4
   env:
   CODECOV_TOKEN: ${{secrets.CODECOV_TOKEN}}

Setup your GITHUB repository:

First you must setup your GITHUB repository by cloning this to your computer. If you do not yet have a GITHUB repository, you can request one 'Request a repository' page.

Once your GITHUB repository is in place and up-to-date, continue with the GIT-LFS instructions.

Setup GIT-LFS:

Go to the GIT-LFS website and follow the instructions on how to download and install GIT-LFS. A good starting point is the 'Getting Started' section on the home page.

GIT-LFS configuration files:

.gitattributes:    Stored in the root folder of your repository. This file contains patterns of all files that GIT-LFS should track and manage as 'large files'.

*.bin filter=lfs diff=lfs merge=lfs -text

Useful examples for .gitattributes can be found here

.lfsconfig:   Stored in the root folder of your repository. This file is necessary to point GIT-LFS to the MinIO server of Deltares instead of the default GITHUB LFS.

[lfs]
url = "http://localhost:8080"

The URL points to the GIT-LFS Proxy endpoint. In this example the proxy listens on port 8080 but this will is configurable in the proxy configuration file config.json

Setup GIT-LFS API Proxy

Besides installing GIT-LFS you will also need to install the GIT LFS Proxy that is shown in the figure below. This proxy is needed to seamlessly integrate GIT-LFS with the S3-based MinIO API. The proxy serves as a bridge between GIT-LFS and the S3 storage protocol, and will translate Git LFS API calls into S3 API calls, ensuring that files tracked by Git LFS are correctly stored on S3.

How this works in practice is as follows:

1. User adds a file to the repository that is identified as a file that should be tracked (see .gitattributes)
2. The file is committed in local repository

3. Once user pushes this file to GITHUB the GIT LFS Proxy intercepts the call and replaces the targeted file by a reference file. This reference file has the same name as the targeted file but the content has been replaced with reference information as shown in the example below
   
   

   version https://git-lfs.github.com/spec/v1
   oid sha256:3056ea3aa2461c9f149ff2c6b62ced81bf396e8bdbbdd0ddb6e8180350f7d715
   size 545791164

4. The actual targeted file is pushed to your configured MinIO bucket with a name that matches the oid sha256 value.
   

The proxy installation files can be found here:

N:\Deltabox\Publications\2023\ict\git-lfs-minio\ for Windows and Linux executables with example config.json

The proxy installation can be placed anywhere on your computer. You can start the proxy by opening a command window and executing git-lfs-minio.exe. A more robust solution is to run the executable as a Windows Service:

How to install a Windows Service:

1. Open command prompt with Admin privileges.

2. Run command: 
   
   

   sc create <Display name> binPath="<Path to>\git-lfs-minio.exe"
   
   Expected result:	[SC] CreateService SUCCESS

3. Open Windows Services to further configure the new service.

GIT-LFS Proxy configuration files:

config.json:    Stored outside of the root folder of your repository. This file contains the MinIO end-point server of Deltares

{
    "serverListenAddr": ":<your proxy port> default :8080",
    "minioHost": "s3.deltares.nl",
    "minioAccessKey": "<your bucket access key>",
    "minioSecretKey": "<your bucket secret key>",
    "minioBucket": "<your bucket name>",
    "minioURLExpires": 3600
}

Minio access- and secretkey can be created in the MinIO console https://s3-console.deltares.nl.

You will also require a MinIO account and a Bucket. You can make request for this together with your request for a GITHUB repository

Setup your GITHUB repository:

First you must setup your GITHUB repository by cloning this to your computer. If you do not yet have a GITHUB repository, you can request one 'Request a repository' page.

Once your GITHUB repository is in place and up-to-date, continue with the DVC instructions.

Setup DVC:

Go to the DVC website and follow the instructions on how to download and install DVC. A good starting point is the 'Get Started' page.

DVC configuration files:

.dvcignore:    Stored in the root folder of your DVC project. This file contains patterns of all files that DVC should ignore.

# Add patterns of files dvc should ignore, which could improve
# the performance. Learn more at
# https://dvc.org/doc/user-guide/dvcignore

# Ignore secrets file
.dvc/config.local

.dvc/.gitignore:    Stored in the .dvc folder. This file is similar to the .dvcignore file however this file contains patterns of all files that GIT should ignore.

/config.local
/tmp
/cache

.dvc/config:    Stored in the .dvc folder. Contains all DVC configuration that can be shared and can be uploaded into your repository.

 [core]
    remote = miniostorage
    autostage = true
['remote "miniostorage"']
    url = s3://<path to your bucket>
    endpointurl = https://s3.avi.deltares.nl
    ssl_verify = false

.dvc/config.local:    Stored in the .dvc folder. Contains all DVC configuration that cannot be shared nor uploaded into your repository

['remote "miniostorage"']
    access_key_id = 
    secret_access_key = 

How you will setup your scripting environment will strongly depend on the codding language of the source code in your GITHUB repository. But in all cases you can take advantage of the REWIND functionality of MinIO. This functionality allows you to restore your data folder of files to a given point in time. 

For Python an example can be found here: https://github.com/robin-deltares/minio-py-rewind/blob/main/minio_rewind.py

from minio import Minio
import minio_rewind

# For access
myMinioServer = 'my.minio.server'
myAccessKey   = 'my_access_key'
mySecretKey   = 'my_secret_key'

# The path that will be recursively downloaded
myBucketName = 'my_bucket_name'
myPathName   = 'my_path_name'
myRewind     = '2023.05.10T16:00' # Notation that mc uses

# Minio client connection
myClient = Minio(myMinioServer,
                 access_key=myAccessKey,
                 secret_key=mySecretKey)

# Prepare the rewind-settings
rewinder = minio_rewind.Rewinder(myClient,myRewind)

# Download the objects
rewinder.download(myBucketName,myPathName)

GIT-LFS

git-lfs is intended to be transparent to git, therefore it requires a customized server. Its learning process is short and fast. Some configuration commands, and bang! it is running, storing large files independently of the git repository. That's its only function, and it does it fine. Having an additional server is not a drawback, but instead a requirement for such transparency. Once configured, files are just handled by git, by means of git hooks (endpoints that are activated after git operations).

Limitations of GIT-LFS can are documented here.

DVC

dvc is intended to provide independent management of large files for the final user. What dvc basically does is this: it just makes git ignore the files that you wish to control (adding them to .gitignore) and instead, it generates an additional file with the same name and the extension .dvc. So, in order to push a commit with its corresponding files, the user is required to manually "add" (equivalent to git commit, not to git add; there's no equivalent for the git stage in dvc) and "push" to both systems. This is not a drawback, but a necessary level of control. In exchange, the remote large-files-holder is just any remote filesystem, accessible directly by its path, via ssh or via multiple drivers (google drive, amazon, etc.). Anyway, hooks are also available for dvc, which would simplify the use of large files, if having additional files is not annoying to one, and saving files to the remote would require additional operations, remember that they are .gitignored! So, if you modify a file stored in dvc, such change will not be noticed by git status, and you might lose such change, except if you make the additional check with dvc.

Some comparisons with related technologies can be found here.

Custom scripts

scripting is the most flexible way to go. It allows you to access all MinIO's API functionality in the language of your preference. To help you get started MinIO offers the user a variety of SDKs. Scripting does imply that you as developer have enough coding skills  are also the maintainer. However with enough real-word examples this option should not be too difficult.

Note to developers: Please provide your examples to github-support@deltares.nl so we can incorporate them into this manual.