Import type NETCDF-CF_GRID_S3 imports grids from nc files stored in  S3 buckets.

To find  the URL’s to the required S3 buckets  and to list the nc files stored in the buckets,  the Amazon Web Services (AWS) are used.
The listed nc files  are  read using the existing import type NETCDF-CF_GRID . It means that all features of NETCDF-CF_GRID  are also available for reading from S3 buckets.

Importing data  for certain period

To import data from multiple S3 Url’s for certain period,  configure:

  • keyword DATE_TIME together with date/time pattern in the serverUrl, for example %DATE_TIME(yyyy)%
  • relativeViewPeriod to indicate for which period the data should be imported
  • optional fileNameObservationDateTimePattern if only selected files should be imported, for example only last 10 days before T0

An example:

<serverUrl>https://s3.waw3-1.cloudferro.com/mdl-native-14/native/GLOBAL_ANALYSISFORECAST_PHY_001_024/cmems_mod_glo_phy_anfc_0.083deg_PT1H-m_202211/%DATE_TIME(yyyy)%/%DATE_TIME(MM)%</serverUrl>	
<fileNameObservationDateTimePattern>'glo12_rg_1h-m_'yyyyMMdd'?'</fileNameObservationDateTimePattern> 		
<relativeViewPeriod unit="day" start="-10" end="0"/>

If the T0 is 01-May-2024 23:00:00 , then this example imports data for 10 days, from 21-04-2024 to 01-05-2024, from these S3 folders ./2024/04 and ./2024/05


Importing data  without period specification

Configuration example 1:  single file  glo12_rg_1h.nc will be imported

<importType>NETCDF-CF_GRID_S3</importType>
<serverUrl>https://s3.waw3-1.cloudferro.com/mdl-native-14/native/GLOBAL_ANALYSISFORECAST_PHY_001_024/cmems_mod_glo_phy_anfc_0.083deg_PT1H-m_202211/2024/03/ glo12_rg_1h.nc</serverUrl>

Configuration example 2:  all nc files under the folder /03 will be listed and imported. In this example all files in month March

<importType>NETCDF-CF_GRID_S3</importType>
<serverUrl>https://s3.waw3-1.cloudferro.com/mdl-native-14/native/GLOBAL_ANALYSISFORECAST_PHY_001_024/cmems_mod_glo_phy_anfc_0.083deg_PT1H-m_202211/2024/03</serverUrl>

Configuration example 3:  all nc files under the folder /2024  will be listed and imported. In this example all files available for year 2024 

<importType>NETCDF-CF_GRID_S3</importType>
<serverUrl>https://s3.waw3-1.cloudferro.com/mdl-native-14/native/GLOBAL_ANALYSISFORECAST_PHY_001_024/cmems_mod_glo_phy_anfc_0.083deg_PT1H-m_202211/2024</serverUrl>

Configuration example 4:  all nc files under the folder /cmems_mod_glo_phy_anfc_0.083deg_PT1H-m_202211  will be listed and imported. In this example all files available for all years under the mentioned  folder

<importType>NETCDF-CF_GRID_S3</importType>
<serverUrl>https://s3.waw3-1.cloudferro.com/mdl-native-14/native/GLOBAL_ANALYSISFORECAST_PHY_001_024/cmems_mod_glo_phy_anfc_0.083deg_PT1H-m_202211</serverUrl>


  • No labels

Disclaimer