Export module

Configuration

The export module can export timeseries for use in other systems. The configuration of the module is split into three sections:

• General: Specify file name, data type etc...
• metadata: Export specific settings
• timeseriesSets: actual data to export

In the sections below the different elements of the configuration are described

General

description

An optional description

exportTypeStandard

This type specifies which writer should be used to write the file. The type must be one from the enumeration. Presently (2007/02) only bfg and pi are included in this list.

exportType

This type specifies which writer should be used to write the file. It may be any string as long as this type is supported by the TimeSeriesExport module. The list of supported types is given here.

folder

Folder (directory) in which to store the exported files.

exportFileName

This elements describes how to construct the filename(s) of the exported file(s).

If only the name element is given a fixed name is used for each export. The prefix and suffix elements describe how to create a filename prefix and/or suffix. The temporaryPrefix is used to generate a prefix for the temporary file as it is being written. After that the file is renamed.

validate

Optional element. Only applicable if the data are exported to the xml-file. This option activates the validation of the exported file against a XML schema.

idmapId

Id of IdMap to be used for parameterId and locationId mapping

unitConversionsId

Id of UnitConversions to be used for unit mapping

flagConversionsId

Id of flagConversions to be used for flag mapping

exportMissingValue/exportMissingValueString

Missing value definition for this time series. Either a string or a number. Defaults to NaN if not defined.

omitMissingValues

If set to true records with missing values are not exported

precision

Available since 2018.02. Optional element to set the number of decimals all values should be displayed with. If set, additional zeros will be appended and/or values will be rounded when necessary. 

It is possible to configure a valueResolution for parameters via the parameters.xml. The configured precision for time series with parameters which have a value resolution should never exceed the maximum number of decimals needed to display values with this resolution. If the precision does exceed this, a warning will be given and the configured precision will be ignored.

The precision can never exceed 8 decimals due to limitations on the resolution with which values can be stored in the FEWS database (floating point errors). 

exportTimeZone

TimeZone in which to export the data. Can either be a string (timeZoneName) or an offset (timeZoneOffset).

convertDatum

Convert (vertical) datum to local datum during export. The conversion will be done for all parameters which use datum (as configured in Parameters.xml) The local datum is defined in the z element in the locations.xml file.

geoDatum

Convert the geographical coordinate system (horizontal datum and projection) to specified geoDatum during export. Not all serializers support this parameter so please check the documentation for a particular serializer to see if it is supported.

ensembleMemberFormat

Available since 2019.02. Can either have value 'name' or  'index'.  If 'name' is configured, the ensemble member Id is written. Otherwise the ensemble member index is written.

forecastSelectionPeriod

If configured all forecasts with a forecast time within the configured period will be exported.

When also configuring a <timeZeroFormattingString> in the <prefix> of the <exportFileName>, each forecast will be exported to a separate file to easily differentiate between the different forecasts.

When no <timeZeroFormattingString> is configured in the <exportFileName> all forecasts will be exported to the same file.

Example configuration:

<export>
	<general>
		<exportType>SomeValidExportType</exportType>
		<folder>MyExportFolder</folder>
		<exportFileName>
			<name>_MyExportedFile.txt</name>
			<prefix>
				<timeZeroFormattingString>yyyyMMddHHmm</timeZeroFormattingString>
			</prefix>
		</exportFileName>
		<idMapId>MyIdMap</idMapId>
		<forecastSelectionPeriod start="-2" end="0" unit="day"/>
	</general>
	...
</export>

exportManualChanges

If used, only manual changed to the data will be exported. Unless the manualDBChangeViewPeriod is used, the relativeViewPeriod from the associated timeSeriesSet is used.

columnSeparator and decimalSeparator

Since 2016.01 (so far only implemented for GeneralCsv export type) it is possible to choose from multiple column separators: comma ","  or semi-colon ";" or pipe "|" or tab "&#009;" or space "&#x20;"

When specifying a column separator it is compulsory to also specify the decimal separator as comma ","  or point "."

For an example see generalCsv export type.

properties

Here properties for specific serializers can be configured. For example for the NetCDFSerializers the following properties will be taken into account:

<properties>
   <bool key="includeComments" value="true"/>
   <bool key="includeFlags" value="true"/>
   <bool key="includeTSProperties" value="true"/>
   <bool key="tryCompactingNetCDFData" value="true"/>
   <string key="netCDFWriteFormat" value="netcdf4"/>
   <int key="netCDF4DeflateLevel" value="6"/>
</properties>

includecomments

Export comment for each time step to NetCDF, default false

includeFlags 

Export flag for each time step to NetCDF, default false

includeTSProperties

Export time series properties for each time step to NetCDF, default false

tryCompactingNetCDFData 

Depending on the difference between the minimum and maximum and the value resolution of a netcdf variable, try to use smaller sized integer variables like short or byte to compact the data. A scale factor and offset will be used to fit the data in the smaller sized variable and will added to the netCDF variable as attributes. This kind of compression will keep the precision of the value resolution. All standard netCDF viewers will take these attributes into account automatically, but other tools and especially scripts might not. This property will be false by default and only works for scalar and grid data.

netCDFWriteFormat

With this property the netcdf format can be set to netcdf4, default it will be netcdf3. Netcdf4 is needed to write compressed netcdf files which can result in 2 to 100 times smaller files.

netCDF4DeflateLevel

This property only works with netcdf4.

With this property the deflate level for writing compressed netcdf files can be set from 0 to 9. 0 meaning no compression and 9 maximum compression. Default will be 5, this level gives best compression without losing too much time when reading or writing. 

metadata

(Meta data export has only been implemented for a limited set of export types. Currrently the NetCDF, LILA and HHRR types export meta data)

Optional metadata that is written in the exported file. The options netcdfMapDPhase and alertMapDPhase are deprecated (do not use these). For the other options it is possible to use the following tags:
%TIME_ZERO% the T0 of this time series export run.
%CURRENT_TIME% the current time.
%MODULE_INSTANCE_ID% the id of this module instance.
%MODULE_INSTANCE_NAME% the name of this module instance.
%MODULE_INSTANCE_DESCRIPTION% the configured description of this module instance.
%WORKFLOW_ID% the id of the workflow in which this export runs.
%WORKFLOW_NAME% the name of the workflow in which this export runs.
%WORKFLOW_DESCRIPTION% the configured description of the workflow in which this export runs.
%USER_ID% the id of the user by which this export run is executed
%EXTERNAL_ANALYSIS_TIME% - external analysis time. When configuring it, it needs two parameters: the first is the external forecast is, the second should be the time format. Neither the id not the time format should contain "," (a comma). The two arguments should be separated bz a comma.
        Example: %EXTERNAL_ANALYSIS_TIME(thisIsTheId),(yyyy/MM/dd HH:mm:ss z)%

%COLD_STATE_START_TIME(yyyy/MM/dd HH:mm:ss z)% - the cold state start time. If data is unavailable it will be filled as "Unknown".

Configuration example of metadata:

<metadata>
	<title>title</title>
	<institution>  institution  </institution>
	<source>source</source>
	<history>Exported at time zero = %TIME_ZERO(yyyy/MM/dd HH:mm:ss z)% in module instance %MODULE_INSTANCE_ID% as part of workflow %WORKFLOW_NAME% by user %USER_ID%.</history>
	<references>references</references>
	<comment>The actual time of writing was %CURRENT_TIME(yyyy-MM-dd HH:mm:ss z)%</comment>
	<summary>A summary of the data</summary>
	<keyword>keyword1</keyword>
	<keyword>  keyword  with  lots  of  spaces  </keyword>
	<keyword>keyword 3</keyword>
	<customAttributes>
		<string key="emptyAttribute" value="    "/>
		<int key="  custom2  " value="123456"/>
		<string key="custom_3" value="This is a custom attribute with 'quotes' in it."/>
		<string key="    " value="attribute with empty key specified is not written"/>
		<float key="just_another_float" value="3.5"/>
		<bool key="truth" value="true"/>
	</customAttributes>
</metadata>

title

A short description of the dataset. Its value will be used by THREDDS opendap servers as the name of the dataset. It therefore should be human readable and reasonable to display in a list of such names.

institution

Specifies where the original data was produced.

source

The method of production of the original data. If it was model-generated, source should name the model and its version, as specifically as could be useful. If it is observational, source should characterize it (e.g. "surface observation" or "radiosonde").

history

Provides an audit trail for modifications to the original data. It should contain a separate line for each modification with each line including a timestamp, user name, modification name, and modification arguments. Its value will be used by THREDDS opendap servers as a history-type documentation. It is recommended that each line begins with a timestamp indicating the date and time of day at which the modification was performed.

references

Published or web-based references that describe the data or methods used to produce it.

comment

Miscellaneous information about the data or methods used to produce it.

summary

The "summary" attribute gives a longer description of the dataset. In many discovery systems, the title and the summary will be displayed in the results list from a search. It should therefore capture the essence of the dataset it describes. For instance, include information on the type of data contained in the dataset, how the data was created (e.g. instrument X or model X, run Y), the creator of the dataset, the project for which the data was created, the geospatial coverage of the data, and the temporal coverage of the data.

keyword

Optional one or more key words or phrases that are relevant to the dataset. The values in this list may be taken from a controlled list of keywords (e.g. the AGU Index list or the GCMD Science Keywords).

customAttributes

If you want to add an attribute that is not predefined in the schema, then you can add it as a custom attribute here.

timeseriesSet

Define the timeseriesset to be exported. Please note that not all exports support all timeseriestypes (e.g. csv only supports scalar type).

filterId

Since 2018.01 it is possible to configure a filter id that refers to a filter from Filters.xml in the RegionConfigFiles.

    </general>
   <filterId>AllQualifiersFilter</filterId>
</export>

This way time series can be exported based on all options present in a filter like location, parameter and qualifier constraints:

<filter id="AllQualifiersFilter">
   <timeSeries>
      <moduleInstanceId>ExportRunMultipleTimeSeries</moduleInstanceId>
   </timeSeries>
   <relativeViewPeriod unit="day" start="-7" end="0"/>
   <locationConstraints>
      <idContains contains="12965"/>
   </locationConstraints>
   <parameterConstraints>
      <idContains contains="H.m"/>
   </parameterConstraints>
</filter>