...
To find the URL’s to the required S3 buckets and to list the nc files stored in the buckets, the Amazon Web Services (AWS) are used.
The listed nc files are read read using the existing import type NETCDF-CF_GRID . It means that all features of NETCDF-CF_GRID are also available for reading from S3 buckets.
Importing data for certain period
To import data from multiple S3 Url’s for certain period, configure:
- keyword DATE_TIME together with date/time pattern in the serverUrl, for example %DATE_TIME(yyyy)%
- relativeViewPeriod to indicate for which period the data should be imported
- optional fileNameObservationDateTimePattern if only selected files should be imported, for example only last 10 days before T0
An example:
Code Block |
---|
<serverUrl>https://s3.waw3-1.cloudferro.com/mdl-native-14/native/GLOBAL_ANALYSISFORECAST_PHY_001_024/cmems_mod_glo_phy_anfc_0.083deg_PT1H-m_202211/%DATE_TIME(yyyy)%/%DATE_TIME(MM)%</serverUrl>
<fileNameObservationDateTimePattern>'glo12_rg_1h-m_'yyyyMMdd'?'</fileNameObservationDateTimePattern>
<relativeViewPeriod unit="day" start="-10" end="0"/> |
If the T0 is 01-May-2024 23:00:00 , then this example imports data for 10 days, from 21-04-2024 to 01-05-2024, from these S3 folders ./2024/04 and ./2024/05
Importing data without period specification
Configuration example 1: single file glo12_rg_1h.nc will be imported
Code Block |
---|
<importType>NETCDF-CF_GRID_S3</importType> <serverUrl>https://s3.waw3-1.cloudferro.com/mdl-native-14/native/GLOBAL_ANALYSISFORECAST_PHY_001_024/cmems_mod_glo_phy_anfc_0.083deg_PT1H-m_202211/2024/03/ glo12_rg_1h.nc</serverUrl> |
Configuration example 2: all nc files under the folder /03 will be listed and imported. In this example all files in month March
Code Block |
---|
<importType>NETCDF-CF_GRID_S3</importType>
<serverUrl>https://s3.waw3-1.cloudferro.com/mdl-native-14/native/GLOBAL_ANALYSISFORECAST_PHY_001_024/cmems_mod_glo_phy_anfc_0.083deg_PT1H-m_202211/2024/03</serverUrl> |
Configuration example 3: all nc files under the folder /2024 will be listed and imported. In this example all files available for year 2024
Code Block |
---|
<importType>NETCDF-CF_GRID_S3</importType>
<serverUrl>https://s3.waw3-1.cloudferro.com/mdl-native-14/native/GLOBAL_ANALYSISFORECAST_PHY_001_024/cmems_mod_glo_phy_anfc_0.083deg_PT1H-m_202211/2024</serverUrl> |
Configuration example 4: all nc files under the folder /cmems_mod_glo_phy_anfc_0.083deg_PT1H-m_202211 will be listed and imported. In this example all files available for all years under the mentioned folder
Code Block |
---|
<importType>NETCDF-CF_GRID_S3</importType>
<serverUrl>https://s3.waw3-1.cloudferro.com/mdl-native-14/native/GLOBAL_ANALYSISFORECAST_PHY_001_024/cmems_mod_glo_phy_anfc_0.083deg_PT1H-m_202211</serverUrl> |