Overview

The Tabular Config Files Display can show a list of configuration files (csv and dbf) that are used to configure the selected locations, parameters and qualifiers. The configuration files can be displayed using a table viewer with sorting, filtering and column hiding functionality. This display is also accessible using the F12 debug menu. There have been several improvements to this display since the first introduction in 2015:

  • Since 2019.02 the display is also connected to the forecast tree.
  • Since 2020.02 the Tabular Config Files Display is connected to shapefiles that are displayed in the Map Display or the Spatial Display.
  • Since 2024.01 it is also possible to edit CSV config files with the Tabular Config Files Display.

Configuration

To use the   TabularConfigFilesDisplay   viewer, configure it as  an <explorerTask> in Explorer.xml. For example:

		<explorerTask name="Tabular ConfigFiles Display">
			<iconFile>csvfile.png</iconFile>
			<mnemonic>L</mnemonic>
			<predefinedDisplay>tabular config files display</predefinedDisplay>	
			<toolbarTask>true</toolbarTask>
			<menubarTask>true</menubarTask>
			<accelerator>ctrl L</accelerator>
			<toolWindow>false</toolWindow>
			<loadAtStartup>true</loadAtStartup>
		</explorerTask>

When the editing functionality is used, an additional display config file (TabularConfigFilesDisplay.xml) can be added to the  DisplayConfigFiles configuration folder. The configuration of this files is explained in the editing section below.

Display functionality

When the Tabular Config Files Display is started, there will be a toolbar at the top of the display, and the two panels will probably be empty as is shown in the example below 

The top panel will list the meta data files that are used in the Delft-FEWS application, the bottom panel will show the content of a selected meta data file.  Only meta data files in .csv or .dbf format that are used in the configuration can be displayed. If the locations are configured in *.xml, *.Json or database, the file will not be shown in the top panel of the display.

The display is linked to the locations list in the Data Viewer (Filters). When a location in the locations list is selected, the meta data files that are used by this location will be listed. The content of the first configuration file will be shown by default in the bottom panel. Selecting another file will display its content.

The list of meta data files is sorted alphabetically. A list is made of all meta data files in the \MapLayers folder that are used by the Delft-FEWS application. The meta data files are not grouped on sub-folders, this information is ignored. 

The table with meta data  can show virtual columns, these are on the left in light blue. For all location attribute files, the linked location Name column is visible. For all meta data files that are used as main files, the location ID is also shown as virtual column. These virtual columns show the Location ID and Name, not the content of a meta data file column.

The toolbar contains 6 buttons with the following functionality

Lock: It is possible to lock the tabular config file display. When locked, selecting another location in the locations list, won't change the selected location row in the display.

List all Files: When this button is pressed, the file list will populate all meta data files used in the configuration

Show modifiers: It's possible to show modified values in the tabular config files, these are the location attributes where modifiers have been applied. All values that have been changed by modifiers will get a blue background color. A tooltip will display the unmodified value. 

Show virtual columns:  Location ID and Location Name are based on meta data in the CSV/dbf files. These columns, together with special edit columns can be made visible or hidden in the bottom panel

Copy: Copy the content of selected cells to the clipboard

Export to CSV: The selected meta data file can be saved as CSV file. This is especially relevant when modified values need to be exported to a CSV file.

Sorting

The table with meta data can be sorted by clicking on a column header. Clicking one extra time on the same column header toggles the direction or deselects the sort column header. When multiple column headers have been sorted on, tiny indices indicate which is the order in which the columns are sorted. In the right mouse popup menu the Undo sort option quickly deselects all selected sort columns.

Filtering

Tables can be filtered by double clicking on a cell entry. Only records with exactly the same entry for the selected column will be displayed. The selected entry will be highlighted (blue). Double clicking another entry will expand the filter. To undo the filtering the highlighted column has to be double clicked again. In the right mouse popup menu the Remove all filters option quickly remove the set filters.

Column hiding

In case not all columns should be visible, right clicking on a value will show the pop-up menu with the "Set column visibility" menu item. This allows a user to select or deselect columns that should be visible (at least one column has to remain visible). These settings will be stored in the user settings and will remain available after FEWS was restarted.

Link to the Forecast Tree

Since 2019.02 the Tabular Config File Display can also be linked to the forecast tree. When locations are configured in the Topology.xml configuration with the tabularLocationId element, the related tabular config files for those locations can be displayed. 

Topology.xml with tabularLocationId
<nodes id="NodesWithLocations" name="Nodes with Location Ids">
	<node id="NodeWithLocationId" name="Node with location">
		<tabularLocationId>GLB</tabularLocationId>
	</node>
...

In FEWS versions before 2021.01 the locationId element was used for this linking, this is not possible anymore. The tabularLocationId element need to be used instead.

Link to Shape Files

Attributes from map layer shape files that have been configured in the Map or Spatial Display, can be viewed with the Tabular Config Files display. No additional configuration is required for this functionality.

In the layer panel of the Map Display or Spatial Display a layer attribute can be selected. The Tabular Config Files Display will show the associated Shape Layer DBF file and its entries. The selected layer attribute will highlight the corresponding row in the table. When selecting a row in the table, the corresponding layer attribute will be highlighted on the Map or Spatial Display.

It is also possible to select a layer attribute on the Map display. This requires the tooltip icon to be enabled. When hovering over the layers on the map or Grid, the layer attributes will be highlighted. When they are highlighted, using a left mouse click to select the map element, this will show the corresponding entry in the Tabular Config Files Display.

Configure Meta Data Column Headers

The column headers of the meta data table use by default the column name in the CSV or DBF file. It is possible to overrule the column headers with configurable names. These names are configured in the LocationSets.xml file. There are two name elements that can be used for this.

  • For standard location elements (like id, name, x, y) use the columnDisplayName element.
  • When the meta data is used as a location attribute, use the name element.


LocationSets.xml with name elements
<csvFile>
	<file>AUS_stations</file>
	<readOnly>true</readOnly>
	<geoDatum>WGS 1984</geoDatum>
	<id columnDisplayName="HyFS Station ID">H%bom_stn_num#6%</id>
	<name columnDisplayName="HyFS Station Name">%bom_stn_name% - H%bom_stn_num#6%</name>
	<x columnDisplayName="Longitude">%long_dec_deg%</x>
	<y columnDisplayName="Latitude">%lat_dec_deg%</y>
	<attribute id="HYFS_STATIONID" name="HyFS Station ID">
		<description>HYFS station location identifier</description>
		<text>H%bom_stn_num#6%</text>
	</attribute>
.....

When using the name element for location attributes, make sure not to use the same attribute name for attributes from the same meta data file, this will throw a Config ERROR when Delft-FEWS starts.

Configure Meta Data Rules

It is possible to configure meta data rules in the LocationSets.xml file. These rules are used to check the meta data on Errors.

One of the rules is included in the attribute element, a format can be added to the <text> element (see below). The example forces the attribute will always contain 6 numbers. When the meta data file contains a value "12345", FEWS will change this to "012345". A leading zero is added to the attribute.

LocationSets.xml with name elements
<attribute id="HYFS_SENSLOCID" name="BoM Sensor Location Id">
	<description>Associated HYFS station location identifier</description>
	<text>%BOM_STATIONID#6%</text>
	<required>true</required>
</attribute>


It is also possible to add an enumeration. The attribute will then be checked on one of the enumeration values. In the Tabular Config Files editor, a drop-down element will be used where the user can select one of the enumerations.

LocationSets.xml with name elements
<attribute id="ACCEPTANCE">
	<description>Only accepted sensors are included in the location set</description>
	<text>%ACCEPTANCE%</text>
	<enumerationValue>ACCEPTED</enumerationValue>
	<enumerationValue>REJECTED</enumerationValue>
	<enumerationValue>DISABLED</enumerationValue>
	<required>true</required>
</attribute>


When an attribute can contain multiple values, a regularExpression can be used. The example below allows for example an attribute value "NSW;VIC". See regular expression function examples on the internet for more examples. 

LocationSets.xml with name elements
<attribute id="REGION" name="Region">
	<description>Regional office(s) responsible for station. Enumeration: WA, NT, SA, QLD, NSW, VIC, TAS. Reflects time zone information</description>
	<text>%REGION%</text>
	<regularExpression>[WA|NT|SA|QLD|NSW|VIC|TAS|;]*</regularExpression>
	<required>true</required>
</attribute>

The Meta Data Rules are used by Delft-FEWS when the meta data configuration is loaded, Errors will be logged when meta data is not according to the configured rules. In The Tabular Config File Display, the rules are used to find and flag errors. 

Edit Meta Data

Since Delft-FEWS version 2024.01 there is functionality added to edit meta data from CSV meta data files. To activate this functionality a TabularConfigFilesDisplay.xml configuration file is needed in the DisplayConfigFiles config folder.

An example of such a file is shown below.

TabularConfigFilesDisplay.xml
<?xml version="1.0" encoding="UTF-8"?>
<tabularConfigFilesDisplay xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/tabularConfigFilesDisplay.xsd">
	<editPermission>Configuration</editPermission>
</tabularConfigFilesDisplay>

There is only one element required in this configuration file, this is the permission for the user group that is allowed to edit meta data files. When this display file is added, the toolbar of the Tabular Config Files display is extended with meta data edit functionality.

 

The toolbar contains 11 additional buttons with the following functionality:

Only Show Errors: When Errors are found in the meta data, enabling this button will only show the meta data files and rows that have errors. Very much linked to the configurable meta data rules.

Only Show Edits: When this button is pressed, the non-committed edits will be filtered on. Only the meta data files and rows that have edits will be shown.

Edit Mode: With the correct permissions it is possible to edit meta data. This button activates the edit mode.

Show Originals:  Wen data is edited and not yet committed, the original values can be shown.

Revert: Edited values that are not yet committed can be reverted. The user will be asked to revert the selected row, the selected meta data file, or all meta data changes.

Refresh Config: When edits are made, the Delft-FEWS system can be updated to use this edited configuration

Commit changes to config dir: The edited meta data files will be copied from temp folder to the config folder.

Delete Row: Delete the selected row from the meta data table

Add Row: Add a new row to the selected meta data table

Cut: Cut the selected meta data table cells to the Clipboard

Paste: Paste the clipboard to the selected meta data table cells

The meta data edit process is split in several editing and commit steps. This allows the user to first test meta data changes, before updating the Delft-FEWS configuration.

  • When meta data values are edited, the changes are stored in the \temp\editedMapLayers\ folder. All files that are edited will be temporarily stored in this folder. In the Tabular Config Files Display GUI, all files that are edited will be shown in bold with (draft) as post-fix.
  • After pressing the Refresh button, the meta data files in this \temp folder will be used by the Delft-FEWS application. The user can then check the changes made.
  • After pressing the Commit changes to config dir button, the edited files from the temp folder will overwrite the original files in the config folder.  It is then not possible to revert changes anymore or check the original values.

 The updated meta data files will not only contain the updated meta data values, two additional columns are added with information on the user that made the changes and when the changes were committed.

Add Custom Activities

The Tabular Config File Display also allows for adding additional task buttons to the toolbar. The task buttons can start executables or scripts to check configuration consistency, link to GIT or SVN or run Python or Powershell scripts.

In the example below a taskbutton is added that starts several Python scripts through Powershell. 

Task Button
<?xml version="1.0" encoding="UTF-8"?>
<tabularConfigFilesDisplay xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/tabularConfigFilesDisplay.xsd">
	<editPermission>Configuration</editPermission>
	<taskButton name="Update Config and Metadata for Public Web Products">
		<description>Update Config and Metadata for Public Web Products</description>
		<permission>Configuration</permission>
		<iconFile>update_config.svg</iconFile>
		<workDir>%REGION_HOME%\Modules\HYMM_python_scripts</workDir>
		<executable>$WINPOWERSHELL$</executable>
		<arguments>
			<argument>$PYTHON$ create_metadata.py %REGION_HOME%\Config\MapLayerFiles\Australia;</argument>
			<argument>$PYTHON$ create_maps.py %REGION_HOME% %REGION_HOME%\Modules\HYMM_python_scripts;</argument>
			<argument>$PYTHON$ create_products.py %REGION_HOME% %REGION_HOME%\Modules\HYMM_python_scripts;</argument>
		</arguments>
	</taskButton>
</tabularConfigFilesDisplay>

Global.properties
WINPOWERSHELL=C://Windows/System32/WindowsPowerShell/v1.0/powershell.exe
PYTHON=%REGION_HOME%/Modules/Python3/python.exe

The Task Buttons allow a user to update other parts of the configuration, like XML files, by using (Python) scripts that create or update Delft-FEWS XML configuration files based on the meta data.

Automated completion of CSV files and attributes

In many Delft-FEWS configurations the meta data is split in several meta data files. In this case there is a main meta data CSV/DBF file and several location attribute files. 

In the example below a location attribute file is used to accept or reject a sensor (location) from the main location CSV file. If the element requireAllIdsInParentFile = true, then a new location row will automatically be added to the AUS_sensors_acceptnace.csv file when a new location is added to the AUS_sensors.csv file. This makes it easier to add new locations, FEWS will make sure that the AUS_sensors_acceptance.csv file contains the same records as in the master AUS_sensors.csv file.

 


It is also possible to have a similar functionality in related location attribute files. In the example below a locationSet is created from all accepted locations (sensor example above).  When a new location is added to the locationSet, the new location will be added to other location Attribute files where the element requireAllIdsInLocationSet = true. In this case there is not a on-one relation between 2 CSV files as in the above example, but the locations that are included in a locationSet is important. 

It is also important to set the required element to true for all location attributes that are required. When these are not filled, FEWS will log an ERROR.


  • No labels

Disclaimer