Imports grid data from CMEMS

Overview

Imports grid time series data from CMEMS NetCDF files. Data is imported per z-layer, cut into smaller time steps to avoid getting "data file too big" error.

Import Configuration

An example of the CMEMS import will be given here. 

The type of import should be configured using the DATA_TYPE property:

Properties "product" and "service" are compulsory fields. Please fill them in exactly the same as it should be in the request url. (See the examples below) The product (i.e. ''Product Identifier" and service (found in "Services tab" on CMEMS website) can be found in the product explorer on the Copernicus website: http://marine.copernicus.eu/services-portfolio/access-to-products/.

FEWS will automatically add x_lo, x_hi, y_lo, y_hi, z_lo, z_hi values to the url. The data will be extracted from the grid configuration.  Please make sure that the configured grid layer depth (z-layer) matches exactly the z_lo layer values the library provides. If it is not an exact match the data request will be refused.  The z_hi values will be calculated by FEWS (z_lo+0.01). Do not use more than two decimals to configure the layers. When shortening the numbers, do not round them: just remove extra digits beyond 2 decimals. Note: Even for 2D imports you need to configure the correct z-value, e.g. 0.493, of the grid layer in Locations.xml.

LAYER,Z
1,-0.49
8,-9.57
18,-47.37
23,-109.72
32,-541.08
36,-1062.43
46,-3992.48

The parameters needed will be added to the request url by FEWS. It uses the external paramter ID for this, so please make sure it matches the parameter that should be in the url exactly.

This example is configured to import 2D Hourly grids for the last 24 hours: 

<?xml version="1.0" encoding="UTF-8"?>
<timeSeriesImportRun xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/timeSeriesImportRun.xsd">

	<import>
		<general>
			<importType>Cmems</importType>		
			<serverUrl>http://nrt.cmems-du.eu/motu-web/Motu?action=productdownload</serverUrl>
			<user>USERNAME</user>
			<password>PASSWORD</password>
			<relativeViewPeriod unit="hour" start="-24" end="0" startOverrulable="true"/>
			<idMapId>IdImportCmems</idMapId>
			<unitConversionsId>ImportUnitConversions</unitConversionsId>
			<expiryTime unit="week" multiplier="500"/>
		</general>
		<properties>
			<string key="DATA_TYPE" value="HOURLY"></string>
			<string key="TICKET_URL" value="https://cmems-cas.cls.fr/cas/v1/tickets"/>
			<string key="product" value="global-analysis-forecast-phy-001-024-hourly-t-u-v-ssh"/>
			<string key="service" value="GLOBAL_ANALYSIS_FORECAST_PHY_001_024-TDS"/>
		</properties>
		<timeSeriesSet>
			<moduleInstanceId>Import_E2O_Server</moduleInstanceId>
			<valueType>grid</valueType>
			<parameterId>Wind.u</parameterId>
			<locationId>CMEMS.L1</locationId>
			<timeSeriesType>external historical</timeSeriesType>
		    <timeStep unit="nonequidistant"/>
			<readWriteMode>add originals</readWriteMode>
		</timeSeriesSet>		
	</import>
</timeSeriesImportRun></timeSer
 
The following example is configured for importing daily 3D grids. 3D data should be imported one layer at a time using the $LOOP_LOCATION_ID$ tag. 
<import>
		<general>
			<importType>CMEMS</importType>
			<serverUrl>http://nrt.cmems-du.eu/motu-web/Motu?action=productdownload</serverUrl>
			<connectionTimeOutMillis>300000</connectionTimeOutMillis>
			<user>$CMEMS_USER$</user>
			<password>$CMEMS_PASSWORD$</password>
			<relativeViewPeriod unit="day" start="-2" end="10" startOverrulable="true"/>
			<idMapId>IdImport_CMEMS</idMapId>
			<maximumVerticalSnapDistance>0.1</maximumVerticalSnapDistance>
			<unitConversionsId>ImportUnitConversions</unitConversionsId>
			<dataFeedId>CMEMS_3D</dataFeedId>
		</general>
		<properties>
			<string key="DATA_TYPE" value="DAILY"/>
			<string key="TICKET_URL" value="https://cmems-cas.cls.fr/cas/v1/tickets"/>
			<string key="product" value="global-analysis-forecast-phy-001-024"/>
			<string key="service" value="GLOBAL_ANALYSIS_FORECAST_PHY_001_024-TDS"/>
		</properties>
		<timeSeriesSet>
			<moduleInstanceId>Import_CMEMS_3D</moduleInstanceId>
			<valueType>grid</valueType>
			<parameterId>T.simulated</parameterId>
			<locationId>$LOOP_LOCATION_ID$</locationId>
			<timeSeriesType>external historical</timeSeriesType>
			<timeStep unit="nonequidistant"/>
			<readWriteMode>add originals</readWriteMode>
			<expiryTime unit="day" multiplier="$EXPIRY_DAYS_EXTERNAL_FORECAST_GRID$"/>
		</timeSeriesSet>
	</import>

 
The following example is configured for importing monthly 3D grids:
	<import>
		<general>
			<importType>Cmems</importType>
			<serverUrl>http://nrt.cmems-du.eu/motu-web/Motu?action=productdownload</serverUrl>
            <connectionTimeOutMillis>300000</connectionTimeOutMillis>
			<user>$CMEMS_USER$</user>
			<password>$CMEMS_PASSWORD$</password>
			<relativeViewPeriod unit="day" start="-1" end="1" startOverrulable="true"/>
			<idMapId>IdImport_CMEMS</idMapId>
			<unitConversionsId>ImportUnitConversions</unitConversionsId>
			<expiryTime unit="day" multiplier="356"/>
		</general>
		<properties>
			<string key="TICKET_URL" value="https://cmems-cas.cls.fr/cas/v1/tickets"/>
			<string key="DATA_TYPE" value="MONTHLY"/>
			<string key="product" value="global-analysis-forecast-phy-001-024-monthly"/>
			<string key="service" value="GLOBAL_ANALYSIS_FORECAST_PHY_001_024-TDS"/>
		</properties>
		<timeSeriesSet>
			<moduleInstanceId>Import_CMEMS</moduleInstanceId>
			<valueType>grid</valueType>
			<parameterId>T.simulated.CMEMS</parameterId>
			<locationId>$LOOP_LOCATION_ID$</locationId>
			<timeSeriesType>external forecasting</timeSeriesType>
			<timeStep unit="nonequidistant"/>
			<readWriteMode>add originals</readWriteMode>
		</timeSeriesSet>
	</import>


Id Map Configuration

An example of the IdMapping used for the NETCDF-CF_GRID import is shown below.

<idMap version="1.1" xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/idMap.xsd">
	<parameter internal="Wind.u" external="uo"/>
	<parameter internal="Wind.v" external="uv"/>
</idMap>

Grids Configuration

When importing grids in the FEWS database it is required to configure the grid characteristics in the grids.XML file. The grid characteristics must be similar to the grid imported from the NetCDF file.

	<regular locationId="CMEMS_2D">
		<rows>241</rows>
		<columns>265</columns>
		<geoDatum>WGS 1984</geoDatum>
		<firstCellCenter>
			<x>43</x>
			<y>28</y>
		</firstCellCenter>
		<xCellSize>0.0833</xCellSize>
		<yCellSize>0.0833</yCellSize>
	</regular>