Within FEWS, and thus NGMS, locations are essential as they are the placeholder to all model inputs and output.
The locations concept and the application within NGMS is described in a seperate page.
One of the most time consuming and error prone parts of the system is setting up those locations and ensuring that the Ids of observation data and model data, as defined in the WEL and STR files, match the licences and or gauging stations. To assist in this process, a standard data template has been produced for specyfying the locations, as well as an locations-generator spreadsheet which automates the process of sorting, IdMapping and XML-file generation.
Note on tooltips
Locations on the Explorer-map have tooltips showing details on licences, coordinates etc. Within a tooltip, a line-feed is established by useing '\n' (backward slash n).
The sorting macro combines tooltips of children by combining strings which end with Row: , Column: , Layer: \.
Notes on OSreference
OSreferences will provide the unique key to identify locations. The sorting macro will add layer details where needed, but the configurator has to ensure that two different boreholes/gauges etc. have two differents OSreferences.
Go to: How to populate OSreference coordinates from kilometer-coordinates
Step 5.1 Check table structure of all worksheets in Data spreadsheet handed over
Go to Step 5.2. Grids_loc
The starting condition of this step is a populated DataTemplate, which contains all relevant information for the model.
When running the locations-generator macro, it will pick up this spreadsheet, conduct a variety of checks and sorts all locations in the appropriate order. For this purpose, the Generator uses hard-coded references to named worksheets and specific columns within a worksheet. To prevent failure in this processing, the following checks the need to be conducted on the table structure of the populated DataTemplate.
Notes on Calculation mode in Excel
Please pay attention to the calculation mode specified in Excel. This mode should be set to Automatic. If this is not set to automatic it could result in a wrong interpretation of abstraction for child wells.
The comparison is done against the Empty_DataTemplate spreadsheet.
- worksheet Model_info
- ensure that the geological layers record are filled, as the generator checks this list to assign the observation boreholes to model layers
- calculate the simulation length in days by subtracting end-date - start-date (B65-B64) and setting the cell format to general. this simulation length needs to be used in the runMF-spreadsheet to specify the export length
- worksheet GWabs_org
- ensure that no more than 4 runs are listed (column A-D), as the IdMapping can currently handle 4 runs only
Note: If more than 4 runs are to be processed, use a second spreadsheet and fill column A-D with the line numbers of the wells in those runs. Please ensure that both spreadsheets contain the same locations. - ensure that LicenceNo starts in column F (i.e. C6)
- ensure that no more than 4 runs are listed (column A-D), as the IdMapping can currently handle 4 runs only
- worksheet SWlat_org
- See above: ensure that no more than 4 runs are listed (column A-D), as the IdMapping can currently handle 4 runs only
Note: If more than 4 runs are to be processed, use a second spreadsheet and fill column A-D with the line numbers of the wells in those runs. Please ensure that both spreadsheets contain the same locations. - ensure that ConsentNo/LicenceNo starts in column I (i.e. C9).
- See above: ensure that no more than 4 runs are listed (column A-D), as the IdMapping can currently handle 4 runs only
- worksheet Branches_org
- check if reachNr is in column C (C3); chainage is in column D (C4), OSRef is in column J (C10)
- if not, copy the worksheet from the Empty_DataTemplate and populate again
or - add column D and column J from the Empty_DataTemplate.xls
- worksheet GWabs_IDmap_hist; GWabs_IDmap_nat; GWabs_IDmap_recact; GWabs_IDmap_fullic
- if not available, copy all 4 from Empty_DataTemplate
- worksheet SWlat_IDmap_hist; SWlat_IDmap_nat; SWlat_IDmap_recact; SWlat_IDmap_fullic
- if not available, copy all 4 from Empty_DataTemplate
The DataTemplate contains one macro, called 'CheckData' which checks if the essential data is in place. While the details will be addressed in the various steps below, most errors are caused if abovementioned steps are not properly conducted or if the OSreference column is empty. An OSreference column with empty or 'na' is not acceptable as this column is the key for creating a unique locationId
Step 5.2 Create and populate worksheets for Grids: Grids_loc and Grids_locSets
Go to #Step 5.3. GW management units
The worksheets Grids_loc and Grids_locSets are manually prepared. They specify the grid geometry and associated location sets as used in the data processing.
Step 5.3 Create and populate worksheet for GW management units: GWunits_loc and GWunits_locSets
Go to #Step 5.4. GW abstractions
The worksheets GWunits_loc and GWunits_locSets are manually prepared. They specify the predefined GWMUs (groundwater management units) and associated location sets as used in the data processing.
Step 5.4 Populate and verify worksheet for groundwater abstractions: GWabs_org
Go to #Step 5.5. Observation boreholes
The worksheet GWabs_org holds location information on the groundwater abstractions as defined in the model runs (WEL file). The worksheet GWabs_org is populated by the data provider. The information is used to generate the Locations.xml holding the XY-position of the abstraction as well as the tooltip shown on the Explorer.map. Multi-layer abstractions or multi-licence abstractions will be sorted by the macro and assigned a 'parent' location shown on the Explorer-map. Various grouping methods will be applied to assign the abstractions to different location sets. The macro will use this worksheet in combination with the GWabs_IDmap-worksheets to define the IdMapping between the location defined in the spreadsheet and the time series as held in the WEL file.
Notes on the use of columns:
- The sorting macro contains hard-coded column references. It is therefore essential that column E is empty and column F holds the LicenceNo. One may not list more than 4 WEL files in the first 4 columns. If more WEL-files need to be accounted for, is recommended to create a second spreadsheet (see details #More than four runs provided).
- The XY-position of the NGR will be adopted as the XY-coordinate of the point location for this borehole. Taking this position over the actual NGR-coordinate ensures that the interpolation from grid cell to OBH - using the closest distance method - will be in line with the way the model has been set up.
- The NGR-coordinates are typically displayed in the tooltip.
- The OSreference will be used for the unique identifier of each abstraction. It is essential that this OSreference is based on the NGR coordinate.
Please conduct the following verification and preparation steps:
- Remove the example rows (if any) such that the row 40 is the first one containing the actual data
- Populate column I (OSreference) based on NGR (Easting, Northing) of borehole (column K and L).
- Check that column A,B,C,D are populated in correspondence to the provided runs (ModuleDataSets)
- Check that the number of locations/licences as listed in the spreadsheet corresponds (at least) to the number of wells in each WEL-file
- Define concatenate functions for cell B16, B17, B18, B19, B21, B22. The actual contents depends on data availability.
- Verify that the column(s) as used for the concatenate functions are properly populated with appropriate information (i.e. preferably no empty cells, 'unknowns, 'na' etc.). If necessary apply an 'IF-construction on the concatenate function. This verification is especially critical for columns being used in location names.
If the table accounts for more than 4 WEL files (i.e. column E and possibly F are used to hold line numbers as well), one should do the following steps:
- Create a copy of the spreadsheet
- Remove the additional columns (E possibly F) from the 'original' spreadsheet and ensure that concatenate functions are still valid
- In the spreadsheet-copy, move contents of additional columns (E possibly F) to column A and B. Empty column C and D if needed.
Ensure that concatenate functions are still valid
Step 5.5 Populate and verify worksheet for observation boreholes: OBH_org
Go to #Step 5.6. Gauges
The worksheet OBH_org holds location information on the observation boreholes which are used to present observation data model results in graphical form. The worksheet OBH_org is populated by the data provider. The information is used to generate the Locations.xml holding the XY-position of the observation borehole as well as the tooltip shown on the Explorer.map. Multi-layer observation boreholes will be sorted by the macro and assigned a 'parent' location shown on the Explorer-map. Various grouping methods will be applied to assign the observation boreholes to different location sets.
Notes on the use of columns:
- The XY-position of the grid cell will be adopted as the XY-coordinate of the point location for this borehole. Taking this position over the actual NGR-coordinate will ensures that the interpolation from gridcell to OBH - using the closest distance method - will be in line with the way that the model has been set up.
- The NGR-coordinates are typically displayed in the tooltip.
- The OSreference will be used for the unique identifier of each borehole. It is recommended, but not required, to convert the gridcell position to OSreference. In any case, please document properly what coordinate has been used for this purpose.
- The sorting macro contains hard-coded column references. It is therefore essential that SitceCode starts at column C. Note the difference with column usage for the GW abstraction points.
Please conduct the following verification and preparation steps:
- Remove the example rows (if any) such that the row 36 is the first one containing the actual data
- Populate column D (OSreference) based on XY-coordinate of the associated grid cell (column H and I).
- Define concatenate functions for cell B14, B15, B16, B17, B19, B20. The actual contents depends on data availability.
- Verify that the column(s) as used for the concatenate functions are properly populated with appropriate information (i.e. preferably no empty cells, 'unknowns, 'na' etc.). If necessary apply an 'IF-construction on the concatenate function. This verification is especially critical for columns being used in location names.
- Verify that observation locations do not refer to layer numbers which are not supported by the model. Data mapping is done on a model layer basis. Hence, an observation in layer 0 will not receive information if the model contains layer 1-3.
Step 5.6 Populate and verify worksheet for gauging stations: Gauges_org
Go to #Step 5.7. SW laterals
The worksheet Gauges_org holds location information on the stream and river gauging stations which are used to present observation data and model results in graphical form. The worksheet Gauges_org is populated by the data provider. The information is used to generate the Locations.xml holding the XY-position of the gauging stations as well as the tooltip shown on the Explorer.map. Various grouping methods will be applied to assign the gauging stations to different location sets.
Notes on the use of columns:
- The XY-position of the grid cell will be adopted as the XY-coordinate of the point location for this gauge. Taking this position over the actual NGR-coordinate will ensures that the interpolation from gridcell to gauge - using the closest distance method - will be in line with the way that the model has been set up.
- The NGR-coordinates are typically displayed in the tooltip.
- The OSreference will be used for the unique identifier of each gauge. It is recommended, but not required, to convert the gridcell position to OSreference. In any case, please document properly what coordinate has been used for this purpose.
- The sorting macro contains hard-coded column references. It is therefore essential that SiteCode starts at column A. Note the difference with column usage for the GW abstraction points.
Please conduct the following verification and preparation steps:
- Remove the example rows (if any) such that the row 35 is the first one containing the actual data
- Populate column B (OSreference) based on XY-coordinate of the associated grid cell(column F and G).
- Define concatenate functions for cell B12, B13, B14, B15, B17, B18. The actual contents depends on data availability.
- Verify that the column(s) as used for the location name are properly populated with appropriate information (i.e. preferably no 'unknowns, 'na' etc.)
- Verify that observations do not refer to layer numbers which are not supported by the model. Data mapping is done on a model layer basis. Hence, gaugeboards in layer 0 do not receive information if the model contains layer 1-3).
Step 5.7 Populate and verify worksheet for SW laterals (discharges and abstractions): SWlat_org
Go to #Step 5.8. Branches
The worksheet SWlat_org holds location information on the surface water abstractions and discharges as defined in the model runs. The worksheet SWlat_org is populated by the data provider. The information is used to generate the Locations.xml holding the XY-position of the SW abstractions and discharges as well as the tooltip shown on the Explorer.map. Multiple abstractions/discharges one grid cell will be sorted by the macro and assigned a 'parent' location shown on the Explorer-map. If the STR-file is the only source of time series information, the parent will be used to hold the time series information. The information detail may be improved if seperate time series are available for import and mapping to individual abstractions/discharges. Various grouping methods will be applied to assign the abstractions to different location sets. The macro will use this worksheet in combation with the SWlat_IDmap-worksheets to define the IdMapping between the locations defined in the spreadsheet and the time series as held in the STR file.
Notes on the use of columns:
- The NGR-coordinate will be adopted as the XY-coordinate of the point location for the abstraction or discharge. This position will ensure unique identification of each consent.
- The OSreference will be used for the unique identifier of each consent. It is required to convert the NGR-coordinate position to OSreference.
- The sorting macro contains hard-coded column references. It is therefore essential that OS reference is at column L. Note the difference with column usage for the GW abstraction points.
Please conduct the following verification and preparation steps:
- Remove the example rows (if any) such that the row 37 is the first one containing the actual data
- Populate column L (OSreference) based on the NGR-coordinate of the discharge or abstraction point (column N and O).
- Define concatenate functions for cell B14, B15, B16, B17, B19, B20. The actual contents depends on data availability.
- Verify that the column(s) as used for the concatenate functions are properly populated with appropriate information (i.e. preferably no empty cells, 'unknowns, 'na' etc.). If necessary apply an 'IF-construction on the concatenate function. This verification is especially critical for columns being used in location names.
Step 5.8 Populate and verify worksheet for river branches: Branches_org
Go to #Step 5.9. IdMaps for GW abstraction
The worksheet Branches_org holds location information on the branches shown as an accretion profile, including the sequence of stream cells within a branch. The worksheet Branches_org is populated by the data provider. The information is used to generate the Branches.xml as well as the Locations.xml. Branches should be defined from source to outlet, to ensure that the accretion profile builds up from left to right.
Notes on the use of columns:
- The Stream/rivername is normally used as unique identifier. Please do not use a slash
in the name. - A new branch is identified by a changing name.
Please conduct the following verification and preparation steps:
- Remove the example rows (if any) such that the row 12 is the first one containing the actual data
- Ensure that you use the correct table format: column C should say reachNr and column J should be OSreference
- Populate column J (OSreference) based on the XY-coordinate of the associated streamcell (column H and I).
- Check that the number of features is limited to keep a clear display
- Ensure that each branch is sorted from source (chainage = 0 m) to outlet (chainage= ???? m). Chainage should be in meters
Step 5.9 Populate and verify worksheets with IdMap data for GW abstractions: GWabs_IDMap (4 worksheets)
Go to #Step 5.10. IdMaps for SW laterals
If step 5.1 is done correctly, 4 worksheets have been added: GWabs_IDmap_hist; GWabs_IDmap_nat; GWabs_IDmap_recact; GWabs_IDmap_fullic
The worksheets need to be populated with results from step 4.
- Go to the XML-files as generated in step 4.8
- For each scenario-type conduct the following steps:
- copy the individual locations (id, x,y,z) as listed in XXX_GW_abstractions_Locations.xml
- paste into column H,I,J,K (starting at row 3)
- copy the parent locations (id, x,y,z) as listed in XXX_GW_abstractions_Locations_MultiLayer.xml
- paste into column B,C,D,E (starting at row 3)
If you do have an XML-editor which can only copy-paste the table with XML-structure, it might be wise to create a macro in your text editor which converts the table into a comma seperated table which can be loaded into Excel.
Step 5.10 Populate and verify worksheets for SW laterals: SWlat_IDMap (4 worksheets)
Go to Step 5.11. Generate Location XML-files
If step 5.1 is done correctly, 4 worksheets have been added: SWlat_IDmap_hist; SWlat_IDmap_nat; SWlat_IDmap_recact; SWlat_IDmap_fullic
The worksheets need to be populated with results from step 4.
- Go to the XML-files as generated in step 4.8
- For each scenario-type conduct the following steps:
- copy the individual locations (id, x,y,z) as listed in XXX_Stream_Cell_Locations.xml
- paste into column H,I,J,K (starting at row 3)
If you do have an XML-editor which can only copy-paste the table with XML-structure, it might be wise to create a macro in your text editor which converts the table into a comma seperated table which can be loaded into Excel.
TO DO Step Add 'new wells'
Step 5.11 Generate Location files
Go to #Step 5.12. Update Grids.xml file
If everything is ready, you can run the macros. Note that the macro version 5 assumes that you use FEWS-software build 20300 or higher.
- To run the macro, conduct the following steps:*
- Open the spreadsheet NGMS-locations_Generator_v5.00.xls.
- Activate the data-spreadsheet and run the macro 'SortLocationsAddParents' held in the NGMS_Locations_generator-spreadsheet.
- The macro asks what to items to process. Normally you would like to process all. Since file saving to DBF4 contains some layout issues, the macro can do this automatically. Note that it shows a message at the time of saving if the file already exists. Dependent on the model size it can take up to 2 hours (e.g. YNN).
- Check the results
- Are the newly generated worksheets properly filled ?
- Did the concatenate functions give proper results (especially for parents) ?
- Is the IDMapping OK ?
- If anything is incorrect:
- Fix the error in the _org worksheet
- Remove the associated worksheets generated by the macro.
- Save again under the same name (possibly with a version number)
- Run the macro 'SortLocationsAddParents' again and pick only the relevant location types.
- Check again.
If everything seems OK one can continue with the generation of the XML-files.
To generate the XML-files, conduct the following steps:
- Run macro 'create_XMLfiles'. (NB at the end of the Sort-macro, it also asks if this next step should be conducted autiomatically).
- The macro creates XML-files in the directory of the data-spreadsheet.
- The macro overwrites existing files
- You can skip location types to be generated (e.g. if you want to update a branches file), but be aware:
If you skip any of the location types listed, your Locations.xml file might not be complete.
- Check the contents of the files. It might be wise to validate with an XML-validation tool.
- Copy the files which are OK to your configuration:
- Any DBF-file to \Config\MapLayerfiles
- Locations.XML, LocationSets.xml, Branches.xml, Polygons.xml --> region-root\Config\RegionConfigFiles
- IdMap files --> \Config\IdMapfiles
- Leave the Grids.xml file to be updated in the next step
Step 5.12 Update Grids.XML file
The Grids.xml file needs to be update with some data from step 4.
The Grids-file, generated in the previous step is not complete yet. if properly setup in step 5.2 it contains all regular grid definitions as well as the irregular grid-definition (placeholder) for the StreamCells_sparse_grid, i.e. the vector of streamcells. To be valid, such irregular grid needs cell centre coordinates for each grid cell. These coordinates have been generated by the ModuleAdapter in step 4.8 as file XXX_StreamCell_Locations_Grid.xml.
To complete the Grids.xml file, conduct the following steps (assuming that all scenario-types share the same streamcells):
- Go to the XML-files as generated in step 4.8
- Copy the individual locations in XML-structure (cell centre) as listed in XXX_Stream_Cell_Locations_Grid.xml
- Paste into the associated irregular grids section of the Grids.xml file.
- Update the number of rows to 1
- Update the number of columns to the total number of streamcells
- Save the file and copy to \Config\RegionConfigFiles
