Import type NETCDF-CF_GRID_S3 imports grids from nc (and sometimes grib2) files stored in S3 buckets.

To find the URLs to the required S3 buckets and to list the nc files stored in the buckets, the Amazon Web Services (AWS) are used.
The listed nc files are read using the existing import type NETCDF-CF_GRID. It means that all features of NETCDF-CF_GRID are also available for reading from S3 buckets.

Importing data  for certain period

To import data from multiple S3 Url’s for certain period, configure:

  • keyword DATE_TIME together with date/time pattern in the serverUrl, for example %DATE_TIME(yyyy)%
  • relativeViewPeriod to indicate for which period the data should be imported
  • optional fileNameObservationDateTimePattern if only selected files should be imported, for example only last 10 days before T0


S3 specific import properties


Property nameTypePurposeNote
s3Region StringSpecify the AWS import region if required, if not specified the region 
AWS_GLOBAL is used.

anonymousS3Credentials 
BooleanTo make unauthenticated, anonymous requests to Amazon S3 using the AWS SDK, you must configure the client to bypass credential resolution. This is only supported for S3 buckets and objects that have been explicitly configured with public-read or anonymous access policies.
s3forcePathStyleBooleanWhen true, force a path-style endpoint interpretation by the AWS SDK. To be used where the bucket name is part of the path.

See https://docs.aws.amazon.com/sdk-for-swift/latest/api/awss3/documentation/awss3/endpointparams/forcepathstyle/

variable_identification_method

stringFor ECMWF import see This How-to pageFor ECMWF import
s3endpointOverrideUrlStringTo be used when: 

  • Working with local or third-party service implementations (such as LocalStack)

  • Connecting to AWS services through a proxy or VPC endpoint

  • Testing against beta or pre-release service endpoints

https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/endpoint-config.html

s3MaxForecastHours

intTo specify relative end time. When used the Import Routine will skip files outside the RVP endtime. see This How-to page



Property use example 
		<general>
			<importType>NETCDF-CF_GRID_S3</importType>
			<serverUrl>$EcmwfApiUrl$/ecmwf-forecasts/%TIME_ZERO(yyyyMMdd)%/%TIME_ZERO(HH)%z/ifs/0p25/$EcmwfDataSet$/</serverUrl>
			<connectionTimeOutMillis>1200000</connectionTimeOutMillis>
			<relativeViewPeriod unit="hour" start="$StartInHours$" end="$EndInHours$" startOverrulable="$StartOverrulable$" endOverrulable="$EndOverrulable$"/>
			<idMapId>IdMapFromEcmwf</idMapId>
			<unitConversionsId>ImportUnitConversions</unitConversionsId>
			<importTimeZone>
				<timeZoneName>GMT</timeZoneName>
			</importTimeZone>
			<dataFeedId>Ecmwf.$AnalysisTimeIdentifier$</dataFeedId>
			<actionLogEventTypeId>New.Forecast.Ecmwf.$AnalysisTimeIdentifier$</actionLogEventTypeId>
			<expiryTime unit="day" multiplier="$ExpiryTimeExternalForecastingGridInDays$"/>
		</general>
		<properties>
			<int key="s3MaxForecastHours" value="$EndInHours$"/>
			<bool key="anonymousS3Credentials" value="true"/>
			<bool key="s3forcePathStyle" value="true"/>
			<string key="variable_identification_method" value="grib2_parameter_code"/>
			<string key="s3endpointOverrideUrl" value="$EcmwfApiUrl$"/>
		</properties>


Example for CMEMS data

<serverUrl>https://s3.waw3-1.cloudferro.com/mdl-native-14/native/GLOBAL_ANALYSISFORECAST_PHY_001_024/cmems_mod_glo_phy_anfc_0.083deg_PT1H-m_202211/%DATE_TIME(yyyy)%/%DATE_TIME(MM)%</serverUrl>	
<fileNameObservationDateTimePattern>'glo12_rg_1h-m_'yyyyMMdd'?'</fileNameObservationDateTimePattern> 		
<relativeViewPeriod unit="day" start="-10" end="0"/>

If the T0 is 01-May-2024 23:00:00 , then this example imports data for 10 days, from 21-04-2024 to 01-05-2024, from these S3 folders ./2024/04 and ./2024/05

Importing data  without period specification

Configuration example 1:  single file glo12_rg_1h.nc will be imported

<importType>NETCDF-CF_GRID_S3</importType>
<serverUrl>https://s3.waw3-1.cloudferro.com/mdl-native-14/native/GLOBAL_ANALYSISFORECAST_PHY_001_024/cmems_mod_glo_phy_anfc_0.083deg_PT1H-m_202211/2024/03/glo12_rg_1h.nc</serverUrl>

Configuration example 2:  all nc files under the folder /03 will be listed and imported. In this example all files in month March

<importType>NETCDF-CF_GRID_S3</importType>
<serverUrl>https://s3.waw3-1.cloudferro.com/mdl-native-14/native/GLOBAL_ANALYSISFORECAST_PHY_001_024/cmems_mod_glo_phy_anfc_0.083deg_PT1H-m_202211/2024/03</serverUrl>

Configuration example 3:  all nc files under the folder /2024  will be listed and imported. In this example all files available for year 2024 

<importType>NETCDF-CF_GRID_S3</importType>
<serverUrl>https://s3.waw3-1.cloudferro.com/mdl-native-14/native/GLOBAL_ANALYSISFORECAST_PHY_001_024/cmems_mod_glo_phy_anfc_0.083deg_PT1H-m_202211/2024</serverUrl>

Configuration example 4:  all nc files under the folder /cmems_mod_glo_phy_anfc_0.083deg_PT1H-m_202211  will be listed and imported. In this example all files available for all years under the mentioned  folder

<importType>NETCDF-CF_GRID_S3</importType>
<serverUrl>https://s3.waw3-1.cloudferro.com/mdl-native-14/native/GLOBAL_ANALYSISFORECAST_PHY_001_024/cmems_mod_glo_phy_anfc_0.083deg_PT1H-m_202211</serverUrl>

Example for ECMWF open data

This How-to page provides a configuration example for importing ECMWF open data using the NETCDF-CF_GRID_S3 importType.

  • No labels

Disclaimer