Page History

...

The data folder defines the root directory in which all the netcdf files are stored. Note that it is allowed to use subfolders. The tag timeSeriesType defines which type of data (external forcast or external historical) is stored in this folder.

Timeseries mapping

To map the timeseries in the netcdf file to a FEWS timeseries in FEWS definition attributes are used. In the attributeMapping can be defined which attribute wil will be used to define the module instance id of the timeseries in the netcdf file.

It is always mandatory to define which attributes will define the module instance id. The location id and the parameter id is defined in the netcdf-files itself. It is therefore not needed to define a attribute for the location id or the parameter id.

If you only define an attribute mapping for a module instance here then the following defaults will be assumed. The ensemble id is main, the time series has no qualifiers, the time If you only define an attribute mapping for a module instance here then the following will be assumed. The ensemble id is main, the time series has no qualifiers, the time step is non equidistant and the data is scalar data.

Timeseries mapping

If you want to define which attributes define the time step, qualifiers and the ensemble id then this should be added to the attributeMapping. Below an example.

Code Block

language	xml

<?xml version="1.0" encoding="UTF-8"?>
<externalStorage xsi:schemaLocation="http://www.wldelft.nl/fews/archive http://fews.wldelft.nl/schemas//version1.0/archive-schemas/externalStorage.xsd" xmlns="http://www.wldelft.nl/fews/archive" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
	<externalCFCompliantNetCDFStorage id="Maps1D_Rivieren">
		<dataFolder>D:\FEWS\archive\matroos\maps1d</dataFolder>
		<timeSeriesType>externalForecast</timeSeriesType>
		<attributeMapping>
			<moduleInstanceId>system</moduleInstanceId>
			<timeStepId>SETS360<<timeStepId>timeStepAttribute</timeStepId>
			<qualifiersId>q1<<qualifiersId>qualifierAttribute</qualifiersId>
			<ensembleId>era5<<ensembleId>ensembleAttribute</ensembleId>
		</attributeMapping>
	</externalCFCompliantNetCDFStorage>
</externalStorage>

Area id and source id

The archive display searches the archive by using an area id and a source id. Both are used to facilitate quick searching in your archive. It is possible to define which attributes of a netcdf file define the area id and source id.

They can also be defined in the attributeMapping sectionIf you are using netcdf-files in which these attributes are not available then you can configure attribute values in the xml-file itself. Below an example.

xml

Code Block

language

<?xml version="1.0" encoding="UTF-8"?>
<externalStorage xsi:schemaLocation="http://www.wldelft.nl/fews/archive http://fews.wldelft.nl/schemas//version1.0/archive-schemas/externalStorage.xsd" xmlns="http://www.wldelft.nl/fews/archive" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
	<externalCFCompliantNetCDFStorage id="Maps1D_Rivieren">
		<dataFolder>D:\FEWS\archive\matroos\maps1d</dataFolder>
		<attribute id="timeStepAttribute" name="SETS360"/>
		<timeSeriesType>externalForecast</timeSeriesType>
		<attributeMapping>
			<moduleInstanceId>system</moduleInstanceId>
			<timeStepId>SETS360<<timeStepId>timeStepAttribute</timeStepId>
			<qualifiersId>q1<<qualifiersId>qualifierAttribute</qualifiersId>
			<ensembleId>era5<<ensembleId>ensembleAttribute</ensembleId>
			<areaId>myAreaId</areaId>
			<sourceId>mySourceId</sourceId>
		<</attributeMapping>
	</externalCFCompliantNetCDFStorage>
</externalStorage>

Grid id

In FEWS a grid has location id. However such a definition in not available for grids in Netcdf. The location id must therefore be defined by an attribute to facilitate that a time series in a netcdf file can be mapped to a time series in FEWS.

This can be done by defining an attribute which contains the location id for the grid. Below an example.

Area id and source id

The archive display searches the archive by using an area id and a source id. Both are used to facilitate quick searching in your archive. It is possible to define which attributes of a netcdf file define the area id and source id.

They can also be defined in the attributeMapping section. Below an example.

Code Block

language	xml

<?xml version="1.0" encoding="UTF-8"?>
<externalStorage xsi:schemaLocation="http://www.wldelft.nl/fews/archive http://fews.wldelft.nl/schemas//version1.0/archive-schemas/externalStorage.xsd" xmlns="http://www.wldelft.nl/fews/archive" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
	<externalCFCompliantNetCDFStorage id="Maps1D_Rivieren">
		<dataFolder>D:\FEWS\archive\matroos\maps1d</dataFolder>
		<timeSeriesType>externalForecast</timeSeriesType>
		<attributeMapping>
			<moduleInstanceId>system</moduleInstanceId>
			<timeStepId>SETS360</timeStepId>
			<qualifiersId>q1</qualifiersId>
			<gridId>locationForGrid</gridId>
			<ensembleId>era5</ensembleId>
			<areaId>myAreaId</areaId>
			<sourceId>mySourceId</sourceId>
		</attributeMapping>
	</externalCFCompliantNetCDFStorage>
</externalStorage>

Data folder and file name filter

Usually a data folder for the external netcdf storage contains a lot of different data. It is usually inconvenient or even impossible to assign all the data to a single netcdf storage.

If for example a data folder contains scalar data and grid data then the data cannot be assigned to a single external storage because a single external storage always contains only scalar or only grid data.

It is therefore common to assign only a part of the data to a single netcdf storage. To define which netcdf file should be assigned to a specific netcdf storage you can use data folder filters and file name filters

They can be used to define which folders and which files belong to the netcdf storage. Below an example.

Grid id

In FEWS a grid has a location id. However such a definition in not available for grids in Netcdf. The location id must therefore be defined by an attribute to facilitate that a time series in a netcdf file can be mapped to a time series in FEWS.

This can be done by defining an attribute which contains the location id for the grid. Below an example.

Code Block

language	xml

<?xml version="1.0" encoding="UTF-8"?>
<externalStorage xsi:schemaLocation="http://www.wldelft.nl/fews/archive http://fews.

Code Block

language	xml

<?xml version="1.0" encoding="UTF-8"?>
<externalStorage xsi:schemaLocation="http://www.wldelft.nl/fews/archive http://fews.wldelft.nl/schemas//version1.0/archive-schemas/externalStorage.xsd" xmlns="http://www.wldelft.nl/fews/archive" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
	<externalCFCompliantNetCDFStorage id="Maps1D_Rivieren">
		<dataFolder>D:\FEWS\archive\matroos\maps1d</dataFolder>
		<timeSeriesType>externalForecast</timeSeriesType>
		<dataFolderFilter><attributeMapping>
			<allValid><moduleInstanceId>system</moduleInstanceId>
			<timeStepId>SETS360</timeStepId>
			<dataFolderContains contains="areaId"/><qualifiersId>q1</qualifiersId>
			<<gridId>locationForGrid</allValid>gridId>
			<anyValid><ensembleId>era5</ensembleId>
				<dataFolderContains contains="abc"/><areaId>myAreaId</areaId>
				<dataFolderContains contains="def"/><sourceId>mySourceId</sourceId>
		</attributeMapping>
	</anyValid>
			<not>
				<anyValid>
					<dataFolderContains contains="123"/>
				</anyValid>
			</not>
		</dataFolderFilter>
		<fileNameFilter>
			<allValid>
				<fileContains contains="name"/>
			</allValid>
			<anyValid>
				<fileContains contains="123"/>
			</anyValid>
			<not>
				<anyValid>
					<fileContains contains="123"/>
				</anyValid>
			</not>
		</fileNameFilter>
		<attributeMapping>
			<moduleInstanceId>system</moduleInstanceId>
			<timeStepId>SETS360</timeStepId>
			<qualifiersId>q1</qualifiersId>
			<gridId>locationForGrid</gridId>
			<ensembleId>era5</ensembleId>
			<areaId>myAreaId</areaId>
			<sourceId>mySourceId</sourceId>
		</attributeMapping>
	</externalCFCompliantNetCDFStorage>
</externalStorage>

Reference an external THREDDS server

It is possible to include an externally hosted THREDDS server into your OpenArchive. Instead of an data folder you should then define the URL of the the THREDS catalogue and URL of the THREDS file server.

Below an example.

externalCFCompliantNetCDFStorage>
</externalStorage>

Data folder and file name filter

Usually a data folder for the external netcdf storage contains a lot of different data. It is usually inconvenient or even impossible to assign all the data to a single netcdf storage.

If for example a data folder contains scalar data and grid data then the data cannot be assigned to a single external storage because a single external storage always contains only scalar or only grid data.

It is therefore common to assign only a part of the data to a single netcdf storage. To define which netcdf file should be assigned to a specific netcdf storage you can use data folder filters and file name filters

They can be used to define which folders and which files belong to the netcdf storage. Below an example.

Code Block

language	xml

<?xml version="1.0" encoding="UTF-8"?>
<externalStorage xsi:schemaLocation="http://www.wldelft.nl/fews/archive http://fews.wldelft.nl/schemas//version1.0/archive-schemas/externalStorage.xsd" xmlns="http://www.wldelft.nl/fews/archive" xmlns:xsi

Code Block

<?xml version="1.0" encoding="UTF-8"?>
<externalStorage xsi:schemaLocation="http://www.wldelftw3.nlorg/fews/archive http://fews.wldelft.nl/schemas//version1.0/archive-schemas/externalStorage.xsd" xmlns="http://www.wldelft.nl/fews/archive" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
	<externalCFCompliantNetCDFStorage id="observedscalarstorage">
		<thredsCatalogueURL>https://fews-ut-ar-202201.avi.deltares.nl/thredds/catalog/data/netcdf-storage/observedscalarstorage/catalog.xml</thredsCatalogueURL>
		<threddsFileServerURL>https://fews-ut-ar-202201.avi.deltares.nl/thredds/fileServer/</threddsFileServerURL>
		<timeSeriesType>observed</timeSeriesType>
		<attributeMapping>
			<moduleInstanceId>attributeForModuleInstance</moduleInstanceId>
			<areaId>areaId</areaId>
			<sourceId>sourceId</sourceId>
		</attributeMapping>
	</externalCFCompliantNetCDFStorage>
</externalStorage>

layerLocationPrefix

For z-layer or sigma layer time series a location id will be generated for each layer by using the defined prefix. If a prefix layer is defined then the layer with id 3 will have location id layer3.

If this option is not defined then each layer will have location id equal to the layer index.

attribute

...

2001/XMLSchema-instance">
	<externalCFCompliantNetCDFStorage id="Maps1D_Rivieren">
		<dataFolder>D:\FEWS\archive\matroos\maps1d</dataFolder>
		<timeSeriesType>externalForecast</timeSeriesType>
		<dataFolderFilter>
			<allValid>
				<dataFolderContains contains="areaId"/>
			</allValid>
			<anyValid>
				<dataFolderContains contains="abc"/>
				<dataFolderContains contains="def"/>
			</anyValid>
			<not>
				<anyValid>
					<dataFolderContains contains="123"/>
				</anyValid>
			</not>
		</dataFolderFilter>
		<fileNameFilter>
			<allValid>
				<fileContains contains="name"/>
			</allValid>
			<anyValid>
				<fileContains contains="123"/>
			</anyValid>
			<not>
				<anyValid>
					<fileContains contains="123"/>
				</anyValid>
			</not>
		</fileNameFilter>
		<attributeMapping>
			<moduleInstanceId>system</moduleInstanceId>
			<timeStepId>SETS360</timeStepId>
			<qualifiersId>q1</qualifiersId>
			<gridId>locationForGrid</gridId>
			<ensembleId>era5</ensembleId>
			<areaId>myAreaId</areaId>
			<sourceId>mySourceId</sourceId>
		</attributeMapping>
	</externalCFCompliantNetCDFStorage>
</externalStorage>

Reference an external THREDDS server

It is possible to include an externally hosted THREDDS server into your OpenArchive. Instead of an data folder you should then define the URL of the the THREDS catalogue and URL of the THREDS file server.

Below an example.

Code Block

<?xml version="1.0" encoding="UTF-8"?>
<externalStorage xsi:schemaLocation="http://www.wldelft.nl/fews/archive http://fews.wldelft.nl/schemas//version1.0/archive-schemas/externalStorage.xsd" xmlns="http://www.wldelft.nl/fews/archive" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
	<externalCFCompliantNetCDFStorage id="observedscalarstorage">
		<thredsCatalogueURL>https://fews-ut-ar-202201.avi.deltares.nl/thredds/catalog/data/netcdf-storage/observedscalarstorage/catalog.xml</thredsCatalogueURL>
		<threddsFileServerURL>https://fews-ut-ar-202201.avi.deltares.nl/thredds/fileServer/</threddsFileServerURL>
		<timeSeriesType>observed</timeSeriesType>
		<attributeMapping>
			<moduleInstanceId>attributeForModuleInstance</moduleInstanceId>
			<areaId>areaId</areaId>
			<sourceId>sourceId</sourceId>
		</attributeMapping>
	</externalCFCompliantNetCDFStorage>
</externalStorage>

layerLocationPrefix

For z-layer or sigma layer time series a location id will be generated for each layer by using the defined prefix. If a prefix layer is defined then the layer with id 3 will have location id layer3.

If this option is not defined then each layer will have location id equal to the layer index.

attribute

With this option it is possible to add attribute key-value pairs to a external netcdf storage. They will be added to the attributes defined in the netcdf file.

Simulated netcdf storage

Simulations which are stored in the Open Archive consists of several parts:

time series (netcdf)
reports
states
modifiers (xml)
what-ifs (xml)

In the Open Archive all these parts are stored in sub folders of a single directory.

It is possible to store the time series in the netcdf storage and the other parts in the Open Archive . This can be configured in the archive export for simulated data in FEWS by using the tag <netcdfStorageExport>

To make these time series in these netcdf files available you need to configure a simulatedNetcdfForecastingNetcdfStorage in your ExternalStorages.xml. Each exported netcdf file will contain a task run id. The task run id will be used to match

the netcdf-files to the simulations in the Open Archive. When you export multiple simulations to the Open Archive during a single run in FEWS then it is not possible anymore to known to which simulation a netcdf-file belongs because there will be multiple simulations in the archive with the same task run id. For this situation the matchingAttributeId should be used. In this case the netcdf files will also be matched by using the values of the defined matching attributes.

Code Block

<?xml version="1.0" encoding="UTF-8"?>
<externalStorage xsi:schemaLocation="http://www.wldelft.nl/fews/archive http://fews.wldelft.nl/schemas//version1.0/archive-schemas/externalStorage.xsd" xmlns="http://www.wldelft.nl/fews/archive" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
	<simulatedForecastingNetcdfStorage id="simulated">
		<dataFolder>D:\FEWS\archive\matroos\maps1d</dataFolder>
		<matchingAttributeId>attributeA</matchingAttributeId>
		<matchingAttributeId>attributeB</matchingAttributeId>
	</simulatedForecastingNetcdfStorage>
</externalStorage>

Archives.xml

Besides defining the external netcdf storages at the archive itself they also need to defined in FEWS by using the Archives.xml.

...

Page tree

Versions Compared

Old Version 7

New Version 8

Key

Timeseries mapping

Timeseries mapping

Area id and source id

Grid id

Area id and source id

Data folder and file name filter

Grid id

Reference an external THREDDS server

Data folder and file name filter

layerLocationPrefix

attribute

Reference an external THREDDS server

layerLocationPrefix

attribute

Simulated netcdf storage

Archives.xml