20 Tabular Config Files Display

Overview

The Tabular Config Files Display can show a list of configuration files (csv and dbf) that are used to configure the selected locations, parameters and qualifiers. The configuration files can be displayed using a table viewer with sorting, filtering and column hiding functionality. This display is also accessible using the F12 debug menu. There have been several improvements to this display since the first introduction in 2015:

• Since 2019.02 the display is also connected to the forecast tree.
• Since 2020.02 the Tabular Config Files Display is connected to shapefiles that are displayed in the Map Display or the Spatial Display.
• Since 2024.01 it is also possible to edit CSV config files with the Tabular Config Files Display. In this way, it is a Metadata Manager Display.

Configuration

To use the TabularConfigFilesDisplay viewer, configure it as  an <explorerTask> in Explorer.xml. For example:

		<explorerTask name="Tabular ConfigFiles Display">
			<iconFile>csvfile.png</iconFile>
			<mnemonic>L</mnemonic>
			<predefinedDisplay>tabular config files display</predefinedDisplay>	
			<toolbarTask>true</toolbarTask>
			<menubarTask>true</menubarTask>
			<accelerator>ctrl L</accelerator>
			<toolWindow>false</toolWindow>
			<loadAtStartup>true</loadAtStartup>
		</explorerTask>	

When the editing functionality is used, an additional display config file (TabularConfigFilesDisplay.xml) can be added to the  DisplayConfigFiles configuration folder. The configuration of this files is explained in the editing section below.

Display functionality, link to Data Viewer (Filters)

When the Tabular Config Files Display is started, there will be a toolbar at the top of the display, and the two panels will probably be empty as is shown in the example below 

Top panel (list view)

The top panel lists meta data files in the \MapLayers folder that are used in the Delft-FEWS application. 

• Only .csv or .dbf files that are used in the configuration can be displayed.
  If locations are configured in *.xml, *.Json or database, these file will not be shown.
• Files are sorted alphabetically.
• The meta data files are not grouped in sub-folders, this information is ignored. 
• Content of the first file is shown by default in the bottom panel. Selecting another file will display its content. 

The display is linked to the locations list in the Data Viewer (Filters.xml). 

• After a location in the locations list is selected, the meta data files that are used by this location will be listed. 
• If parameters and/or qualifiers are also configured in csv files, the display will be linked to parameters and qualifiers in a similar fashion.

Bottom panel (table view)

The bottom panel shows the content of a meta data file selected in the top panel, by default the content of the first file in the list is shown.

Light blue columns on the left are "virtual columns", which show the Location ID and Name, not the content of a meta data file column.

• LOCATION_NAME : visible for all location attribute files 
• LOCTION_ID: visible for files that are used as main files
• PARAMETER_NAME: visible for all parameter attribute file 

Sorting of column(s)

The table with metadata can be sorted by clicking on a column header, rotating through the following options:

• sort alphabetically ascending
• sort alphabetically descending 
• remove sort 

When multiple column headers have been sorted on, indices indicate which is the order in which the columns are sorted. 
In the right mouse popup menu the Undo sort option quickly deselects all selected sort columns.

Filtering in columns

Tables can be filtered by double clicking on a cell entry.

• Only records with exactly the same entry for the selected column will be displayed.
• The selected entry will be highlighted (blue).
• Double clicking another entry in a different column will expand the filter. 
• To undo filtering, double click a highlighted cell again. 
• In the right mouse popup menu the Remove all filters option quickly remove the set filters.

Hiding of columns

Columns can be hidden.

• In the right mouse popup menu the Set column visibility options opens a list of all columns.
• This allows a user to select or deselect columns that should be visible (at least one column has to remain visible).
• These settings are stored in the user settings and will be applied when Delft-FEWS is restarted.

Toolbar

The toolbar contains 6 buttons with the following functionality

Link to Forecast Tree (Topology)

The Tabular Config File Display can also be linked to the forecast tree (Topology.xml, since 2019.02).
When locations are configured in the Topology.xml configuration with the tabularLocationId element, the related tabular config files for those locations can be displayed. 

<nodes id="NodesWithLocations" name="Nodes with Location Ids">
	<node id="NodeWithLocationId" name="Node with location">
		<tabularLocationId>GLB</tabularLocationId>
	</node>
...

Since 2021.01 it is no longer possible to use the <locationId> element was for this linking. Please update your configuration to use <tabularLocationId> instead.

Link to Layer panel in Map or Spatial display (shapefiles)

Attributes from map layer shape files configured in the Map or Spatial Display, can be viewed with the Tabular Config Files display. No additional configuration is required.

• There are two ways select a shapefile attribute:
  • In the Map Display or Spatial Display, open the layer panel and select a layer in the top panel, and a layer attribute in the bottom panel.
  • In the Map display, enable the tooltip icon and select a layer attribute:
    • When hovering over the layers on the map or Grid, the layer attributes will be highlighted.
    • When they are highlighted, using a left mouse click to select the map element, this will show the corresponding entry in the Tabular Config Files Display.
• The Tabular Config Files Display will show the associated Shape Layer DBF file and its entries.
  • The selected layer attribute will highlight the corresponding row in the table.
  • When selecting a row in the table, the corresponding layer attribute will be highlighted on the Map or Spatial Display.

Configure Metadata Column Headers

By default the column headers of the meta data table use the column name in the CSV or DBF file. 
They can be overruled in LocationSets.xml with two name elements:

• <columnDisplayName> for standard location elements (like id, name, x, y) 
• <name> for location attribute names (like HYFS_STATIONID in the example below)
  Note: Do not use the same <name> twice within a single meta data file. This will throw a Config ERROR when Delft-FEWS is started.

<csvFile>
	<file>AUS_stations</file>
	<readOnly>true</readOnly>
	<geoDatum>WGS 1984</geoDatum>
	<id columnDisplayName="HyFS Station ID">H%bom_stn_num#6%</id>
	<name columnDisplayName="HyFS Station Name">%bom_stn_name% - H%bom_stn_num#6%</name>
	<x columnDisplayName="Longitude">%long_dec_deg%</x>
	<y columnDisplayName="Latitude">%lat_dec_deg%</y>
	<attribute id="HYFS_STATIONID" name="HyFS Station ID">
		<description>HYFS station location identifier</description>
		<text>H%bom_stn_num#6%</text>
	</attribute>
.....

Configure Metadata Rules

Meta data rules to check the meta data <text> for errors can be configured in LocationSets.xml. 
There are rules available to check the text value on format, against enumerationValues and against a regularExpression.

The Meta Data Rules are checked by Delft-FEWS when the meta data configuration is loaded,
Errors will be logged when meta data is not according to the configured rules.
In The Tabular Config File Display, the rules are used to find and flag errors. 

format

In the attribute element, a format can be added to the <text> element. The example below forces the attribute to contain 6 tokens. 

• If the meta data file contains a value that breaks with this rule, a leading zero is added to the attribute.
  In the example if the meta data contains the value "12345", Delft-FEWS will change this to "012345".

<attribute id="HYFS_SENSLOCID" name="BoM Sensor Location Id">
	<description>Associated HYFS station location identifier</description>
	<text>%BOM_STATIONID#6%</text>
	<required>true</required>
</attribute>

<enumerationValue>

In the attribute element, you can add <enumerationValue> elements.

• The attribute <text> value will be checked to match one of the enumeration values. 
• In the Tabular Config Files editor, a drop-down element will allow the user to select one of the enumerations.

<attribute id="ACCEPTANCE">
	<description>Only accepted sensors are included in the location set</description>
	<text>%ACCEPTANCE%</text>
	<enumerationValue>ACCEPTED</enumerationValue>
	<enumerationValue>REJECTED</enumerationValue>
	<enumerationValue>DISABLED</enumerationValue>
	<required>true</required>
</attribute>

<regularExpression>

When an attribute can contain multiple values, a regularExpression can be used. 
The example below allows for example an attribute value "NSW;VIC".
See regular expression function examples on the internet for more examples. 

<attribute id="REGION" name="Region">
	<description>Regional office(s) responsible for station. Enumeration: WA, NT, SA, QLD, NSW, VIC, TAS. Reflects time zone information</description>
	<text>%REGION%</text>
	<regularExpression>[WA|NT|SA|QLD|NSW|VIC|TAS|;]*</regularExpression>
	<required>true</required>
</attribute>

Edit Metadata

You can edit meta data from CSV meta data files (since 2024.01).

For this, a TabularConfigFilesDisplay.xml configuration file is needed in the DisplayConfigFiles folder (see example below):

• editPermission: this is the only required element. It restricts the permission to edit meta data files to specific user group(s).

• With this display file added, the toolbar of the Tabular Config Files display is extended with meta data edit functionality.

<?xml version="1.0" encoding="UTF-8"?>
<tabularConfigFilesDisplay xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/tabularConfigFilesDisplay.xsd">
	<editPermission>Configuration</editPermission>
</tabularConfigFilesDisplay>

Toolbar (extended)

The toolbar contains 11 additional buttons with the following functionality:

Edit process

The meta data edit process is split in several editing and commit steps. This allows the user to first test metadata changes, before updating the Delft-FEWS configuration.

• When metadata values are edited, the changes are stored in the \temp\editedMapLayers\ folder. All files that are edited will be temporarily stored in this folder.
  In the Tabular Config Files Display GUI, all files that are edited will be shown in bold with (draft) as post-fix.
• After pressing the Refresh Config button, the meta data files in this \temp folder will be used by the Delft-FEWS application. 
  The user can then check the changes made (and revert if needed).
• After pressing the Commit changes to config dir button, the edited files from the temp folder will overwrite the original files in the config folder. 
  At this point is not possible to revert changes anymore or check the original values.

 The updated meta data files will not only contain the updated meta data values.
Two additional columns (with blue background) are added with information on the user that made the changes and when the changes were committed.

• LAST_EDIT_USER
• LAST_EDIT_TIME

Add Custom Activities (toolbar buttons)

The Tabular Config File Display also allows for adding additional task buttons to the toolbar.
These task buttons can start executables or scripts to check configuration consistency, link to GIT or SVN or run Python or Powershell scripts.
They allow a user to update other parts of the configuration, like XML files, by using (Python) scripts that create or update Delft-FEWS XML configuration files based on the meta data.

In the example below, a taskButton is added that starts several Python scripts through Powershell. 

<?xml version="1.0" encoding="UTF-8"?>
<tabularConfigFilesDisplay xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/tabularConfigFilesDisplay.xsd">
	<editPermission>Configuration</editPermission>
	<taskButton name="Update Config and Metadata for Public Web Products">
		<description>Update Config and Metadata for Public Web Products</description>
		<permission>Configuration</permission>
		<iconFile>update_config.svg</iconFile>
		<workDir>%REGION_HOME%\Modules\HYMM_python_scripts</workDir>
		<executable>$WINPOWERSHELL$</executable>
		<arguments>
			<argument>$PYTHON$ create_metadata.py %REGION_HOME%\Config\MapLayerFiles\Australia;</argument>
			<argument>$PYTHON$ create_maps.py %REGION_HOME% %REGION_HOME%\Modules\HYMM_python_scripts;</argument>
			<argument>$PYTHON$ create_products.py %REGION_HOME% %REGION_HOME%\Modules\HYMM_python_scripts;</argument>
		</arguments>
	</taskButton>
</tabularConfigFilesDisplay>

Global.properties
WINPOWERSHELL=C://Windows/System32/WindowsPowerShell/v1.0/powershell.exe
PYTHON=%REGION_HOME%/Modules/Python3/python.exe

Automated completion of attribute files

Often, locations will be defined in a single main location file, while the meta data is spread over several attributeFiles.
To make it easier to add new locations (and their required meta data), you can configure LocationSets.xml to automatically complete attributeFiles (since 2023.01).

Auto-complete based on parent file

• <requireAllIdsInParentFile>: when true, a new location added to the location file will automatically be added to the attributeFile

In the example below there is a main meta data file (AUS_sensors.csv) listing all locations and some key attributes, with an additional location attribute file (AUS_sensors_acceptance.csv).
Since requireAllIdsInParentFile = True, a new location row will automatically be added to AUS_sensors_acceptance.csv (the attributeFile) when a new location is added to AUS_sensors.csv (the main csvFile).
This way Delft-FEWS will make sure that the AUS_sensors_acceptance.csv file contains the same records as in the master AUS_sensors.csv file.

Auto-complete based on locationSet

Sometimes there's not such a one-on-one relation between a location file and attribute file, but instead you want the attribute file to contain all locations in a locationSet.

• <requireAllIdsInLocationSet>: when true, a new location added to the locationSet will automatically be added to the attributeFile

The example below shows how the locationSet from before, which is based on a location file (AUS_sensors.csv), is filtered using a constraint based on an attribute (AUS_sensors_acceptance.csv).
For the remaining attributeFiles, we want to ensure they contain all locations in the (filtered/constraint) locationSet, not all locations listed in the original locationFile.

Since requireAllIdsInLocationSet= true, a new location row will automatically be added to AUS_sensor_idmap.csv (the attributeFile) when a new location is added to the locationSet (e.g. by changing the attribute to satisfy the constraint).

Note: When you configure the <required> element for an attribute, Delft-FEWS will log an ERROR if they are not filled. Using this option can help you identify missing meta data more easily.

<constraints>
    <attributeTextEquals id="ACCEPTANCE" equals="ACCEPTED"/>
</constraints>