How to import ECMWF open data

Introduction

ECMWF makes real-time forecast data from the IFS and AIFS models available to the public free of charge. More information about this dataset is available on the ECMWF open data page and ECMWF open data user documentation. This wiki page provides example config for adding this import to your Delft-FEWS application. A configuration example of the import is also available in the FEWS-Conform repository.

This import uses importType NETCDF-CF_GRID_S3.

This import is available Delft-FEWS versions 2024.02 (basebuild 202402-50714143 or higher) and later.

The second time you run the import for the same T0, the import will be much faster (as long as you don't restart FEWS), because FEWS caches the .gpx files in the temp folder. This makes the import easier to test/debug.

Known issues/warnings

The first time importing a certain forecast (with a specific T0), you may occasionally miss a timestep and get this warning: "WARN - Bad GRIB2 record in file cdms3://ecmwf-forecasts.s3.eu-central-1.amazonaws.com/ecmwf-forecasts?20250807/00z/ifs/0p25/oper/20250807000000-129h-oper-fc.grib2, skipping pos=85720605 cause=Please reduce your request rate. (Service: S3, Status Code: 503, Request ID: GT6E0AJF2J2GT1CZ, Extended Request ID: 0dj/YW262Zwm6s8wg8n4mKOwp1yzZqxNS38x9GnoGQ3o2hlBKu02xOzgHs/IdVT3wl3n5comEM1d/z2FF14+MFrmr7tntDg+9aY85Jns1KA=)". This is a problem and we're working on this (see FEWS-32610).
Import runtimes can vary a lot (
The first time importing a certain forecast (with a specific T0), you'll see several warnings like "WARN - Table ecmf/name line scaledValueOfSecondWavelength = missing() ;" → This is not a problem, you can ignore this warning.
When importing accumulated parameters, the first timestep (00h) will always give a warning like "WARN - Exception while parsing NetCDF data from url cdms3://ecmwf-forecasts.s3.eu-central-1.amazonaws.com/ecmwf-forecasts?20250807/00z/ifs/0p25/oper/20250807000000-0h-oper-fc.grib2. This url will be skipped. Message was: Allocation size must be greater than zero". → This is not a problem, you can ignore this warning.
The units defined in the .grib2 files don't always seem to be correct (e.g. radiation flux says W/m2 in the .grib2, but it actually seems to be a 3-hour accumulated J/m2 value).
We have observed that certain parameters (e.g. Pressure_msl) take much longer than other parameters to import. This still needs to be investigated further.

Make the IdMap

Start by manually downloading an example .grib2 file from the ECMWF data browser and opening it in a NetCDF viewer (e.g. Panoply). Identify the parameters you would like to import. How to map the parameters depends on whether a parameter has a z-level (*_height_above_ground) and whether a parameter is accumulated or not (i.e. variables whose name changes in the grib2 file for each timestep).:

Parameters without a z-level can be mapped directly, without defining locations. Use the main variable name listed in the grib2 file as external id.
Parameters with a z-level should be mapped to an internalLocation that has a z-level defined (see next paragraph) combined with externalLocation="0". Use the main variable name listed in the grib2 file as external id.
For parameters whose name changes at each timestep, such as accumulated variables (*_Accumulation), use the "Grib2_Parameter_Name" from the grib2 file as external id.
For isobaric parameters (*_isobaric), map the FEWS location to the layer index in the grib2 file.

Example IdMap for 2D parameters, 3D parameters using a z-level, and parameters whose names change at each timestep

<idMap xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/idMap.xsd" version="1.1">
	
	<!--3D variables with a fixed grib2 parameter name and a z-level ("*_height_above_ground") should be linked to a location with that height.-->
	<map internalParameter="WSu.fcst" internalLocation="Ecmwf.10m" externalParameter="u-component_of_wind_height_above_ground" externalLocation="0"/>
	<map internalParameter="WSv.fcst" internalLocation="Ecmwf.10m" externalParameter="v-component_of_wind_height_above_ground" externalLocation="0"/>
	<map internalParameter="T.fcst" internalLocation="Ecmwf.2m" externalParameter="Temperature_height_above_ground" externalLocation="0"/>

	<!--2D variables with a fixed grib2 parameter name only need to be linked to the internal parameter-->
	<parameter internal="Pmsl.fcst" external="Pressure_msl"/>
	<parameter internal="Rain.rate.tot" external="Total_precipitation_rate_surface"/>

    <!--2D variables with a varying grib2 parameter name only need to be linked to the internal parameter. Use the "Grib2_Parameter_name" as external.-->
    <parameter internal="Rs.fcst" external="Net short wave radiation flux"/>
</idMap>

Example IdMap for 3D parameters using isobaric levels

See next section for how locationSet Ecmwf.Ecmwf_levels is configured.

<idMap xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/idMap.xsd" version="1.1">
    
	<!-- 3D variables with a fixed grib2 parameter name and an isobaric level ("*_isobaric") should be linked to a location with that height.-->
	<parameter internal="T.air.3D" external="Temperature_isobaric"/>
	<parameter internal="Wind.u.3D" external="u-component_of_wind_isobaric"/>
	<parameter internal="Wind.v.3D" external="v-component_of_wind_isobaric"/>
	<parameter internal="Humidity.rel.3D" external="Relative_humidity_isobaric"/>
	<parameter internal="Geopot.height.3D" external="Geopotential_height_isobaric"/>	
	
	<locationIdFunction internalLocationSet="Ecmwf.Ecmwf_levels" externalLocationFunction="@LAYER_INDEX@"/>
</idMap>

Define the locations

Next, define the locationId for the grid(s). You'll need to configure a separate locationId for each height above the ground (and one for variables that don't have a height specified). For *_height_above_ground variables, you'll need to identify the height coordinate defined for the variable, and then check the associated coordinate variable to identify the height. Configure this height in Locations.xml as shown below.

Example locationIds for 2D and 3D data (using z-levels)

	<location id="Ecmwf" name="Grid for ECMWF">
		<x>0</x>
		<y>0</y>
	</location>
	<location id="Ecmwf.2m" name="Grid for ECMWF (2 m above ground)">
		<x>0</x>
		<y>0</y>
		<z>2</z>
	</location>
	<location id="Ecmwf.10m" name="Grid for ECMWF (10 m above ground)">
		<x>0</x>
		<y>0</y>
		<z>10</z>
	</location>

Example locationSet for 3D data (using isobaric levels)

	<locationSet id="Ecmwf.Ecmwf_levels">
		<csvFile>
			<file>ECMWF_layers.csv</file>
			<geoDatum>WGS 1984</geoDatum>
			<id>ECMWF.%LEVEL_MB%mb</id>
			<name>ECMWF (%LEVEL_MB% mb)</name>
			<parentLocationId>Ecmwf</parentLocationId>
			<x>0</x>
			<y>0</y>
			<z>%LEVEL_MB%</z>
			<attribute id="LAYER_INDEX">
				<description>Level ID (0-12), out of 13 layers available in ECMWF</description>
				<text>%LAYER_INDEX%</text>
			</attribute>
			<attribute id="LEVEL_MB">
				<description>Pressure associated with level (in mb)</description>
				<text>%LEVEL_MB%</text>
			</attribute>
		</csvFile>
	</locationSet>

Where ECMWF_layers.csv looks like this (in this example we only import 5 of the 13 layers available in ECMWF):

LAYER_INDEX,LEVEL_MB
5,300
7,500
8,600
9,700
10,850

Define the grids

Next, define the grid you would like to import - see the example below. The ECMWF data are available for the entire globe, but with the NETCDF-CF_GRID_S3 you can directly import a smaller section of the grid for your area of interest.

<regular locationId="Ecmwf">
		<rows>136</rows>
		<columns>168</columns>
		<geoDatum>WGS 1984</geoDatum>
		<firstCellCenter>
			<x>112</x>
			<y>-10</y>
		</firstCellCenter>
		<xCellSize>0.25</xCellSize>
		<yCellSize>0.25</yCellSize>
	</regular>
	<regular locationId="Ecmwf.2m">
		<rows>136</rows>
		<columns>168</columns>
		<geoDatum>WGS 1984</geoDatum>
		<firstCellCenter>
			<x>112</x>
			<y>-10</y>
		</firstCellCenter>
		<xCellSize>0.25</xCellSize>
		<yCellSize>0.25</yCellSize>
	</regular>
	<regular locationId="Ecmwf.10m">
		<rows>136</rows>
		<columns>168</columns>
		<geoDatum>WGS 1984</geoDatum>
		<firstCellCenter>
			<x>112</x>
			<y>-10</y>
		</firstCellCenter>
		<xCellSize>0.25</xCellSize>
		<yCellSize>0.25</yCellSize>
	</regular>

Configure the import module

Now you can configure your import module.

All imports require the property anonymousS3Credentials = true
Some of the ECMWF open data (e.g. IFS oper meteo data) uses a variable timestep - e.g. 3-hourly for the first 144 hours, followed by 6-hourly timesteps, but only for 2 of the 4 forecasts per day. To address, this, the example below includes setting the option interpolateSerie to true which automatically imports the data to 3-hourly timesteps upon import, avoiding the need for a processing module. If you import 6 days or less, then you don't need this setting.
To import data every 6 hours, duplicate the <import> section in your import module file and replace "/oper/" with "/scda/" for meteo data. Similarly, for wave data, substitute /wave/ with /scwv/ to get the 06:00 and 18:00 forecasts.
The import settings depend on the type of parameter you're trying to import - the three known examples are shown below (there may be more)
This import has not been tested with the aifs-single or aifs-ens data yet, but the formats appear to be similar to the ifs data.

Variables without a z-level (i.e. without *_height_above_ground in the variable name)

<import>
		<general>
			<importType>NETCDF-CF_GRID_S3</importType>
			<serverUrl>https://ecmwf-forecasts.s3.eu-central-1.amazonaws.com/ecmwf-forecasts/%TIME_ZERO(yyyyMMdd)%/%TIME_ZERO(HH)%z/ifs/0p25/oper/</serverUrl>
			<connectionTimeOutMillis>1200000</connectionTimeOutMillis>
			<relativeViewPeriod unit="day" start="0" end="10"/>
			<idMapId>IdMapFromEcmwf</idMapId>
			<unitConversionsId>ImportUnitConversions</unitConversionsId>
			<importTimeZone>
				<timeZoneName>GMT</timeZoneName>
			</importTimeZone>
			<dataFeedId>Ecmwf.meteo</dataFeedId>
		</general>
		<properties>
			<bool key="anonymousS3Credentials" value="true"/>
		</properties>
		<timeSeriesSet>
			<moduleInstanceId>ImportEcmwfMeteo</moduleInstanceId>
			<valueType>grid</valueType>
			<parameterId>Pmsl.fcst</parameterId>
			<locationId>Ecmwf</locationId>
			<timeSeriesType>external forecasting</timeSeriesType>
			<timeStep unit="hour" multiplier="3"/>
			<readWriteMode>add originals</readWriteMode>
		</timeSeriesSet>
		<timeSeriesSet>
			<moduleInstanceId>ImportEcmwfMeteo</moduleInstanceId>
			<valueType>grid</valueType>
			<parameterId>Rain.rate.tot</parameterId>
			<locationId>Ecmwf</locationId>
			<timeSeriesType>external forecasting</timeSeriesType>
			<timeStep unit="hour" multiplier="3"/>
			<readWriteMode>add originals</readWriteMode>
		</timeSeriesSet>
		<interpolateSerie parameterId="Pmsl.fcst" interpolate="true"/>
		<interpolateSerie parameterId="Rain.rate.tot" interpolate="true"/>
    </import>

3D variables with a z-level (i.e. with *_height_above_ground)

Importing these variables requires setting layerIndexAsQualifierId to true (and using the locationId with a z-level defined, see above).

<import>
		<general>
			<importType>NETCDF-CF_GRID_S3</importType>
			<serverUrl>https://ecmwf-forecasts.s3.eu-central-1.amazonaws.com/ecmwf-forecasts/%TIME_ZERO(yyyyMMdd)%/%TIME_ZERO(HH)%z/ifs/0p25/oper/</serverUrl>
			<connectionTimeOutMillis>1200000</connectionTimeOutMillis>
			<relativeViewPeriod unit="day" start="0" end="10"/>
			<idMapId>IdMapFromEcmwf</idMapId>
			<unitConversionsId>ImportUnitConversions</unitConversionsId>
			<importTimeZone>
				<timeZoneName>GMT</timeZoneName>
			</importTimeZone>
			<dataFeedId>Ecmwf.meteo</dataFeedId>
		</general>
		<properties>
			<bool key="anonymousS3Credentials" value="true"/>
			<bool key="layerIndexAsQualifierId" value="true"/>
		</properties>
		<timeSeriesSet>
			<moduleInstanceId>ImportEcmwfMeteo</moduleInstanceId>
			<valueType>grid</valueType>
			<parameterId>T.fcst</parameterId>
			<locationId>Ecmwf.2m</locationId>
			<timeSeriesType>external forecasting</timeSeriesType>
			<timeStep unit="hour" multiplier="3"/>
			<readWriteMode>add originals</readWriteMode>
		</timeSeriesSet>
		<timeSeriesSet>
			<moduleInstanceId>ImportEcmwfMeteo</moduleInstanceId>
			<valueType>grid</valueType>
			<parameterId>WSu.fcst</parameterId>
			<locationId>Ecmwf.10m</locationId>
			<timeSeriesType>external forecasting</timeSeriesType>
			<timeStep unit="hour" multiplier="3"/>
			<readWriteMode>add originals</readWriteMode>
		</timeSeriesSet>
		<timeSeriesSet>
			<moduleInstanceId>ImportEcmwfMeteo</moduleInstanceId>
			<valueType>grid</valueType>
			<parameterId>WSv.fcst</parameterId>
			<locationId>Ecmwf.10m</locationId>
			<timeSeriesType>external forecasting</timeSeriesType>
			<timeStep unit="hour" multiplier="3"/>
			<readWriteMode>add originals</readWriteMode>
		</timeSeriesSet>
		<interpolateSerie parameterId="T.fcst" interpolate="true"/>
		<interpolateSerie parameterId="WSu.fcst" interpolate="true"/>
		<interpolateSerie parameterId="WSv.fcst" interpolate="true"/>
	</import>

2D variables with a grib2 parameter name that changes per timestep (e.g. accumulated parameters)

This option requires setting the variable_identification_method property to grib2_parameter_name.

<import>
		<general>
			<importType>NETCDF-CF_GRID_S3</importType>
			<serverUrl>https://ecmwf-forecasts.s3.eu-central-1.amazonaws.com/ecmwf-forecasts/%TIME_ZERO(yyyyMMdd)%/%TIME_ZERO(HH)%z/ifs/0p25/oper/</serverUrl>
			<connectionTimeOutMillis>1200000</connectionTimeOutMillis>
			<relativeViewPeriod unit="day" start="0" end="10"/>
			<idMapId>IdMapFromEcmwf</idMapId>
			<unitConversionsId>ImportUnitConversions</unitConversionsId>
			<importTimeZone>
				<timeZoneName>GMT</timeZoneName>
			</importTimeZone>
			<dataFeedId>Ecmwf.meteo</dataFeedId>
		</general>
		<properties>
			<bool key="anonymousS3Credentials" value="true"/>
			<string key="variable_identification_method" value="grib2_parameter_name"/>
		</properties>
		<timeSeriesSet>
			<moduleInstanceId>ImportEcmwfMeteo</moduleInstanceId>
			<valueType>grid</valueType>
			<parameterId>Rs.fcst</parameterId>
			<locationId>Ecmwf</locationId>
			<timeSeriesType>external forecasting</timeSeriesType>
			<timeStep unit="hour" multiplier="3"/>
			<readWriteMode>add originals</readWriteMode>
		</timeSeriesSet>
		<externUnit parameterId="Rs.fcst" unit="Jm-2/3hr" cumulativeSum="true"/>
        <interpolateSerie parameterId="Rs.fcst" interpolate="true"/>
	</import>

3D variables with an isobaric level (i.e. with *_isobaric)

This option requires setting the laterIndexAsLocationId property to true.

<import>
		<general>
			<importType>NETCDF-CF_GRID_S3</importType>
			<serverUrl>https://ecmwf-forecasts.s3.eu-central-1.amazonaws.com/ecmwf-forecasts/%TIME_ZERO(yyyyMMdd)%/%TIME_ZERO(HH)%z/ifs/0p25/oper/</serverUrl>
			<connectionTimeOutMillis>1200000</connectionTimeOutMillis>
			<relativeViewPeriod unit="day" start="0" end="10"/>
			<idMapId>IdMapFromEcmwf</idMapId>
			<unitConversionsId>ImportUnitConversions</unitConversionsId>
			<importTimeZone>
				<timeZoneName>GMT</timeZoneName>
			</importTimeZone>
			<dataFeedId>Ecmwf.meteo</dataFeedId>
		</general>
		<properties>
			<bool key="anonymousS3Credentials" value="true"/>
            <bool key="layerIndexAsLocationId" value="true"/>
        </properties>
		<timeSeriesSet>
			<moduleInstanceId>ImportEcmwfMeteo</moduleInstanceId>
			<valueType>grid</valueType>
			<parameterId>T.air.3D</parameterId>
			<locationId>Ecmwf.Ecmwf_levels</locationId>
			<timeSeriesType>external forecasting</timeSeriesType>
			<timeStep unit="hour" multiplier="3"/>
			<readWriteMode>add originals</readWriteMode>
		</timeSeriesSet>
        <interpolateSerie parameterId="T.air.3D" interpolate="true"/>
	</import>

Page tree