Overview

Imports grid time series data from CMEMS NetCDF files. Data is imported per z-layer, cut into smaller time steps to avoid getting "data file too big" error.

Import Configuration

An example of the CMEMS import will be given here. 

The type of import should be configured using the DATA_TYPE property:

  • DATA_TYPE should be set to "HOURLY" for hourly data.
  • DATA_TYPE should be set to "DAILY" for daily data.
  • DATA_TYPE should be set to "MONTHLY" for monthly data.

Properties "product" and "service" are compulsory fields. Please fill them in exactly the same as it should be in the request url. (See the examples below) The product (i.e. ''Product Identifier" and service (found in "Services tab" on CMEMS website) can be found in the product explorer on the Copernicus website: http://marine.copernicus.eu/services-portfolio/access-to-products/.

FEWS will automatically add x_lo, x_hi, y_lo, y_hi, z_lo, z_hi values to the url. The data will be extracted from the grid configuration.  Please make sure that the configured grid layer depth (z-layer) matches exactly the z_lo layer values the library provides. If it is not an exact match the data request will be refused.  The z_hi values will be calculated by FEWS (z_lo+0.01). Do not use more than two decimals to configure the layers. When shortening the numbers, do not round them: just remove extra digits beyond 2 decimals. Note: Even for 2D imports you need to configure the correct z-value, e.g. 0.493, of the grid layer in Locations.xml.


Attempting to import too many data points at once can cause an error on the data provider (server) side. The default way of importing data is per time step and per layer. If the grids are very small, it can be possible to import several time steps or layers at once. For this, properties importAllLayersAtOnce and importTimePeriodAtOnce should be set to true. The properties can be used independently of each other. If importAllLayersAtOnce is set to true, data from all layers available between the highest and lowest will be downloaded, even for ones that are not configured in FEWS and not needed.

Z-layer configuration example
LAYER,Z
1,-0.49
8,-9.57
18,-47.37
23,-109.72
32,-541.08
36,-1062.43
46,-3992.48

The parameters needed will be added to the request url by FEWS. It uses the external paramter ID for this, so please make sure it matches the parameter that should be in the url exactly.

This example is configured to import 2D Hourly grids for the last 24 hours: 

ImportNetcdf_Grid 1.00 default.xml
<?xml version="1.0" encoding="UTF-8"?>
<timeSeriesImportRun xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/timeSeriesImportRun.xsd">

	<import>
		<general>
			<importType>Cmems</importType>		
			<serverUrl>http://nrt.cmems-du.eu/motu-web/Motu?action=productdownload</serverUrl>
			<user>dummy_username</user>
			<password>dummy_password</password>
			<relativeViewPeriod unit="hour" start="-24" end="0" startOverrulable="true"/>
			<idMapId>IdImportCmems</idMapId>
			<unitConversionsId>ImportUnitConversions</unitConversionsId>
			<expiryTime unit="week" multiplier="500"/>
		</general>
		<properties>
			<string key="DATA_TYPE" value="HOURLY"></string>
			<string key="TICKET_URL" value="https://cmems-cas.cls.fr/cas/v1/tickets"/>
			<string key="product" value="global-analysis-forecast-phy-001-024-hourly-t-u-v-ssh"/>
			<string key="service" value="GLOBAL_ANALYSIS_FORECAST_PHY_001_024-TDS"/>
<!--          <bool key="importAllLayersAtOnce" value="true"/>-->
<!--         <bool key="importTimePeriodAtOnce" value="true"/>-->
		</properties>
		<timeSeriesSet>
			<moduleInstanceId>Import_E2O_Server</moduleInstanceId>
			<valueType>grid</valueType>
			<parameterId>Wind.u</parameterId>
			<locationId>CMEMS.L1</locationId>
			<timeSeriesType>external historical</timeSeriesType>
		    <timeStep unit="nonequidistant"/>
			<readWriteMode>add originals</readWriteMode>
		</timeSeriesSet>		
	</import>
</timeSeriesImportRun></timeSer


The following example is configured for importing daily 3D grids. 3D data should be imported one layer at a time using the $LOOP_LOCATION_ID$ tag. 
<import>
		<general>
			<importType>CMEMS</importType>
			<serverUrl>http://nrt.cmems-du.eu/motu-web/Motu?action=productdownload</serverUrl>
			<connectionTimeOutMillis>300000</connectionTimeOutMillis>
			<user>$CMEMS_USER$</user>
			<password>$CMEMS_PASSWORD$</password>
			<relativeViewPeriod unit="day" start="-2" end="10" startOverrulable="true"/>
			<idMapId>IdImport_CMEMS</idMapId>
			<maximumVerticalSnapDistance>0.1</maximumVerticalSnapDistance>
			<unitConversionsId>ImportUnitConversions</unitConversionsId>
			<dataFeedId>CMEMS_3D</dataFeedId>
		</general>
		<properties>
			<string key="DATA_TYPE" value="DAILY"/>
			<string key="TICKET_URL" value="https://cmems-cas.cls.fr/cas/v1/tickets"/>
			<string key="product" value="global-analysis-forecast-phy-001-024"/>
			<string key="service" value="GLOBAL_ANALYSIS_FORECAST_PHY_001_024-TDS"/>
		</properties>
		<timeSeriesSet>
			<moduleInstanceId>Import_CMEMS_3D</moduleInstanceId>
			<valueType>grid</valueType>
			<parameterId>T.simulated</parameterId>
			<locationId>$LOOP_LOCATION_ID$</locationId>
			<timeSeriesType>external historical</timeSeriesType>
			<timeStep unit="nonequidistant"/>
			<readWriteMode>add originals</readWriteMode>
			<expiryTime unit="day" multiplier="$EXPIRY_DAYS_EXTERNAL_FORECAST_GRID$"/>
		</timeSeriesSet>
	</import>



The following example is configured for importing monthly 3D grids:
	<import>
		<general>
			<importType>Cmems</importType>
			<serverUrl>http://nrt.cmems-du.eu/motu-web/Motu?action=productdownload</serverUrl>
            <connectionTimeOutMillis>300000</connectionTimeOutMillis>
			<user>$CMEMS_USER$</user>
			<password>$CMEMS_PASSWORD$</password>
			<relativeViewPeriod unit="day" start="-1" end="1" startOverrulable="true"/>
			<idMapId>IdImport_CMEMS</idMapId>
			<unitConversionsId>ImportUnitConversions</unitConversionsId>
			<expiryTime unit="day" multiplier="356"/>
		</general>
		<properties>
			<string key="TICKET_URL" value="https://cmems-cas.cls.fr/cas/v1/tickets"/>
			<string key="DATA_TYPE" value="MONTHLY"/>
			<string key="product" value="global-analysis-forecast-phy-001-024-monthly"/>
			<string key="service" value="GLOBAL_ANALYSIS_FORECAST_PHY_001_024-TDS"/>
		</properties>
		<timeSeriesSet>
			<moduleInstanceId>Import_CMEMS</moduleInstanceId>
			<valueType>grid</valueType>
			<parameterId>T.simulated.CMEMS</parameterId>
			<locationId>$LOOP_LOCATION_ID$</locationId>
			<timeSeriesType>external forecasting</timeSeriesType>
			<timeStep unit="nonequidistant"/>
			<readWriteMode>add originals</readWriteMode>
		</timeSeriesSet>
	</import>


Id Map Configuration

An example of the IdMapping used for the NETCDF-CF_GRID import is shown below.

IdImportNetCDF 1.00 default.xml
<idMap version="1.1" xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/idMap.xsd">
	<parameter internal="Wind.u" external="uo"/>
	<parameter internal="Wind.v" external="uv"/>
</idMap>

Grids Configuration

When importing grids in the FEWS database it is required to configure the grid characteristics in the grids.XML file. The grid characteristics must be similar to the grid imported from the NetCDF file.

Grids 1.00 default.xml
	<regular locationId="CMEMS_2D">
		<rows>241</rows>
		<columns>265</columns>
		<geoDatum>WGS 1984</geoDatum>
		<firstCellCenter>
			<x>43</x>
			<y>28</y>
		</firstCellCenter>
		<xCellSize>0.0833</xCellSize>
		<yCellSize>0.0833</yCellSize>
	</regular>
  • No labels

Disclaimer