Page History

...

ECMWF makes real-time forecast data from the IFS and AIFS models available to the public free of charge. More information about this dataset is available on the ECMWF websiteopen data page and ECMWF open data user documentation. This wiki page provides example config for adding this import to your Delft-FEWS application. An up-to-date A configuration example of the import is also available in the FEWS-Conform repository. 

...

This import is available Delft-FEWS versions 2024.02 (basebuild 202402-50714143 or higher) and later.

The second time you run the import for the same T0, the import will be much faster (as long as you don't restart FEWS), because FEWS caches the .gpx files in the temp folder. This makes the import easier to test/debug/debug.

Known issues/warnings

1. The first time importing a certain forecast (with a specific T0), you may occasionally miss a timestep and get this warning: "WARN - Bad GRIB2 record in file cdms3://ecmwf-forecasts.s3.eu-central-1.amazonaws.com/ecmwf-forecasts?20250807/00z/ifs/0p25/oper/20250807000000-129h-oper-fc.grib2, skipping pos=85720605 cause=Please reduce your request rate. (Service: S3, Status Code: 503, Request ID: GT6E0AJF2J2GT1CZ, Extended Request ID: 0dj/YW262Zwm6s8wg8n4mKOwp1yzZqxNS38x9GnoGQ3o2hlBKu02xOzgHs/IdVT3wl3n5comEM1d/z2FF14+MFrmr7tntDg+9aY85Jns1KA=)". This is a problem and we're working on this (see FEWS-32610). 
2. Import runtimes can vary a lot (
3. The first time importing a certain forecast (with a specific T0), you'll see several warnings like "WARN - Table ecmf/name line      scaledValueOfSecondWavelength = missing() ;" → This is not a problem, you can ignore this warning.
4. When importing accumulated parameters, the first timestep (00h) will always give a warning like "WARN - Exception while parsing NetCDF data from url cdms3://ecmwf-forecasts.s3.eu-central-1.amazonaws.com/ecmwf-forecasts?20250807/00z/ifs/0p25/oper/20250807000000-0h-oper-fc.grib2. This url will be skipped. Message was: Allocation size must be greater than zero". → This is not a problem, you can ignore this warning.
5. The units defined in the .grib2 files don't always seem to be correct (e.g. radiation flux says W/m2 in the .grib2, but it actually seems to be a 3-hour accumulated J/m2 value). 
6. We have observed that certain parameters (e.g. Pressure_msl) take much longer than other parameters to import. This still needs to be investigated further.

Make the IdMap

Start by manually downloading an example .grib2 file from the ECMWF data browser and opening it in a NetCDF viewer (e.g. Panoply). Identify the parameters you would like to import. How to map the parameters depends on whether a parameter has a z-level (*_height_above_ground) and whether a parameter is accumulated or not (i.e. variables whose name changes in the grib2 file for each timestep).:

• Parameters without a z-level can be mapped directly, without defining locations. Use the main variable name listed in the grib2 file as external id.
• Parameters with a z-level should be mapped to an internalLocation that has a z-level defined (see next paragraph) combined with externalLocation="0". Use the main variable name listed in the grib2 file as external id.
• For parameters whose name changes at each timestep, such as accumulated variables (*_Accumulation), use the "Grib2_Parameter_Name" from the grib2 file as external id. 
• This import has not yet been tested for other types of parameters such as *_isobaric variables.

...

• For isobaric parameters (*_isobaric), map the FEWS location to the layer index in the grib2 file. 

Example IdMap for 2D parameters, 3D parameters using a z-level, and parameters whose names change at each timestep

<idMap xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/idMap.xsd" version="1.1">
	
	<!--3D variables with a fixed grib2 parameter name and a z-level ("*_height_above_ground") should be linked to a location with that height.-->
	<map internalParameter="WSu.fcst" internalLocation="Ecmwf.10m" externalParameter="u-component_of_wind_height_above_ground" externalLocation="0"/>
	<map internalParameter="WSv.fcst" internalLocation="Ecmwf.10m" externalParameter="v-component_of_wind_height_above_ground" externalLocation="0"/>
	<map internalParameter="T.fcst" internalLocation="Ecmwf.2m" externalParameter="Temperature_height_above_ground" externalLocation="0"/>

	<!--2D variables with a fixed grib2 parameter name only need to be linked to the internal parameter-- internal parameter-->
	<parameter internal="Pmsl.fcst" external="Pressure_msl"/>
	<parameter internal="PmslRain.rate.fcsttot" external="Pressure_mslTotal_precipitation_rate_surface"/>

	    <!--2D variables with a varying grib2 parameter name only need to be linked to the internal parameter. Use the "Grib2_Parameter_name" as external.-->
    <parameter internal="Rs.fcst" external="Net short wave radiation flux"/>
</idMap>

...

Example locationIds for 2D and 3D data (using z-levels)

	<location id="Ecmwf" name="Grid for ECMWF">
		<x>0</x>
		<y>0</y>
	</location>
	<location id="Ecmwf.2m" name="Grid for ECMWF (2 m above ground)">
		<x>0</x>
		<y>0</y>
		<z>2</z>
	</location>
	<location id="Ecmwf.10m" name="Grid for ECMWF (10 m above ground)">
		<x>0</x>
		<y>0</y>
		<z>10</z>
	</location>

...

• All imports require the property anonymousS3Credentials = true
• Some of the ECMWF open data (e.g. IFS oper meteo data) uses a variable timestep - e.g. 3-hourly for the first 144 hours, followed by 6-hourly timesteps, but only for 2 of the 4 forecasts per day. To address, this, the example below includes setting the option interpolateSerie to true which automatically imports the data to 3-hourly timesteps upon import, avoiding the need for a processing module. If you import 6 days or less, then you don't need this setting.
• To import data every 6 hours, duplicate the <import> section in your import module file and replace "/oper/" with "/scda/" for meteo data. Similarly, for wave data, substitute /wave/ with /scwv/ to get the 06:00 and 18:00 forecasts.
• The import settings depend on the type of parameter you're trying to import - the three known examples are shown below (there may be more))
• This import has not been tested with the aifs-single or aifs-ens data yet, but the formats appear to be similar to the ifs data. 

Variables without a z-level (i.e. without *_height_above_ground in the variable name)

<import>
		<general>
			<importType>NETCDF-CF_GRID_S3</importType>
			<serverUrl>https://ecmwf-forecasts.s3.eu-central-1.amazonaws.com/ecmwf-forecasts/%TIME_ZERO(yyyyMMdd)%/%TIME_ZERO(HH)%z/ifs/0p25/oper/</serverUrl>
			<connectionTimeOutMillis>1200000</connectionTimeOutMillis>
			<relativeViewPeriod unit="day" start="0" end="10"/>
			<idMapId>IdMapFromEcmwf</idMapId>
			<unitConversionsId>ImportUnitConversions</unitConversionsId>
			<importTimeZone>
				<timeZoneName>GMT</timeZoneName>
			</importTimeZone>
			<dataFeedId>Ecmwf.meteo</dataFeedId>
		</general>
		<properties>
			<bool key="anonymousS3Credentials" value="true"/>
		</properties>
		<timeSeriesSet>
			<moduleInstanceId>ImportEcmwfMeteo</moduleInstanceId>
			<valueType>grid</valueType>
			<parameterId>Pmsl.fcst</parameterId>
			<<locationId>Ecmwf</general>locationId>
			<properties><timeSeriesType>external forecasting</timeSeriesType>
			<bool<timeStep keyunit="anonymousS3Credentialshour" valuemultiplier="true3"/>
			<readWriteMode>add originals</readWriteMode>
		</properties>timeSeriesSet>
		<timeSeriesSet>
			<moduleInstanceId>ImportEcmwfMeteo</moduleInstanceId>
			<valueType>grid</valueType>
			<parameterId>Pmsl<parameterId>Rain.rate.fcst<tot</parameterId>
			<locationId>Ecmwf</locationId>
			<timeSeriesType>external forecasting</timeSeriesType>
			<timeStep unit="hour" multiplier="3"/>
			<readWriteMode>add originals</readWriteMode>
		</timeSeriesSet>
		<interpolateSerie parameterId="Pmsl.fcst" interpolate="true"/>
		<interpolateSerie parameterId="Rain.rate.tot" interpolate="true"/>
    </import>

3D variables with a z-level (i.e. with *_height_above_ground)

...