Versions Compared

Old Version 12

changes.mady.by.user Marc van Dijk

Saved on

compared with

New Version Current

changes.mady.by.user Erik Pelgrim

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Migration of unmigrated content due to installation of a new plugin

...

Imports time series data from Diver loggers. The files have a sort of Windows ini file format with file extension (*.mon). The format of the MON ini files is not well defined. Many programs interpret the structure differently and have various names for the ini file sections and parameters.

  • Wiki MarkupSections: Section declarations start with '\[' and end with '\]'; i.e. '\[Logger settings\]' or '\[Instrument info\]'.
  • Parameters or item: this is the content of a section with an '=' sign between the key and the value; i.e. "location = abc"

The Date format used is: "yyyy/MM/dd HH:mm:ss"

Configuration (Example)

A complete import module configuration consists of an ID Mapping file and a Import Module Instance file.

ModuleConfigFiles

The files can contain a flexible number of channels, where a channel will contain information on one parameter. Example files with 2 and 3 channels will be shown, when multiple channels/parameters need to be imported a <timeSeriesSet> needs to be configured for each channel/parameter. To illustrate this an example config for an import with 3 channels is given.

Configuration (Example)

A complete import module configuration consists of an ID Mapping file and a Import Module Instance file.

ModuleConfigFiles

The following example of an Import Module Instance will import the time series as equidistant The following example of an Import Module Instance will import the time series as equidistant series for timezone GMT+1 with a time step of 1 hour. Many times the MON files do not save the data at rounded hourly tims, therefore a tolerance has been added to map the imported data to correct hourly interval time series.

Code Block
xml
xml
titleImportMON.xml
<?xml version="1.0" encoding="UTF-8"?>
<timeSeriesImportRun ......">
  <import>
    <general>
      <importType>DIVERMON</importType>
      <folder>$IMPORT_FOLDER_MON$</folder>
      <failedFolder>$IMPORT_FAILED_FOLDER_MON$</failedFolder>
      <backupFolder>$IMPORT_BACKUP_FOLDER_MON$</backupFolder>
      <idMapId>IdImportMON</idMapId>
      <unitConversionsId>ImportUnitConversions</unitConversionsId>
      <importTimeZone>
        <timeZoneOffset>+01:00</timeZoneOffset>
      </importTimeZone>
      <dataFeedId>MON</dataFeedId>
    </general>
    <tolerance locationSetId="ImportMON_H.meting.cm_uur" parameterId="H.meting.cm" timeUnit="minute" unitCount="30"/>
    <timeSeriesSet>
      <moduleInstanceId>ImportMON</moduleInstanceId>
      <valueType>scalar</valueType>
      <parameterId>H.meting.cm</parameterId>
      <locationSetId>ImportMON_H.meting.cm_uur</locationSetId>
      <timeSeriesType>external historical</timeSeriesType>
      <timeStep unit="hour" multiplier="1"/>
      <readWriteMode>add originals</readWriteMode>
      <synchLevel>1</synchLevel>
    </timeSeriesSet>
  </import>
</timeSeriesImportRun>

IdMapFiles

ID mapping fefines mappings between Diver MON and FEWS parameters and locations. Remember that ID mapping is case sensitive.

...


<idMap version="1.1"  ..............>
    <map internalParameter="H.meting.cm" internalLocation="Dorperdijk_beneden" externalParameter="niveau" externalLocation="Dorperdijk beneden"/>
</idMap>

?xml version="1.0" encoding="UTF-8"?>
<timeSeriesImportRun ......">
  <import>
    <general>
      <importType>DIVERMON</importType>
      <folder>$IMPORT_FOLDER_MON$</folder>
      <failedFolder>$IMPORT_FAILED_FOLDER_MON$</failedFolder>
      <backupFolder>$IMPORT_BACKUP_FOLDER_MON$</backupFolder>
      <idMapId>IdImportMON</idMapId>
      <unitConversionsId>ImportUnitConversions</unitConversionsId>
      <importTimeZone>
        <timeZoneOffset>+01:00</timeZoneOffset>
      </importTimeZone>
      <dataFeedId>MON</dataFeedId>
    </general>
    <tolerance locationSetId="ImportMON_H.meting.cm_uur" parameterId="H.meting.cm" timeUnit="minute" unitCount="30"/>
    <timeSeriesSet>
      <moduleInstanceId>ImportMON</moduleInstanceId>
      <valueType>scalar</valueType>
      <parameterId>H.meting.cm</parameterId>
      <locationSetId>ImportMON_H.meting.cm_uur</locationSetId>
      <timeSeriesType>external historical</timeSeriesType>
      <timeStep unit="hour" multiplier="1"/>
      <readWriteMode>add originals</readWriteMode>
      <synchLevel>1</synchLevel>
    </timeSeriesSet>
  </import>
</timeSeriesImportRun>

Here is an example config file for an import file with three channels:

Code Block
xml
xml
titleImportMON_3channels.xml

<?xml version="1.0" encoding="UTF-8"?>
<timeSeriesImportRun ......">
  <import>
		<general>
			<importType>DiverMon</importType>
			<folder>$IMPORT_FOLDER$/DiverMon</folder>
			<idMapId>IdMap</idMapId>
			<missingValue>-999.0</missingValue>
			<expiryTime unit="day" multiplier="366"/>
		</general>
		<timeSeriesSet>
			<moduleInstanceId>ImportMON</moduleInstanceId>
			<valueType>scalar</valueType>
			<parameterId>X.yy.zz</parameterId>
			<locationId>Delft</locationId>
			<timeSeriesType>external historical</timeSeriesType>
			<timeStep unit="minute" multiplier="10"/>
			<readWriteMode>add originals</readWriteMode>
		</timeSeriesSet>
		<timeSeriesSet>
			<moduleInstanceId>ImportMON</moduleInstanceId>
			<valueType>scalar</valueType>
			<parameterId>A.bb.cc</parameterId>
			<locationId>Delft</locationId>
			<timeSeriesType>external historical</timeSeriesType>
			<timeStep unit="minute" multiplier="10"/>
			<readWriteMode>add originals</readWriteMode>
		</timeSeriesSet>
		<timeSeriesSet>
			<moduleInstanceId>ImportMON</moduleInstanceId>
			<valueType>scalar</valueType>
			<parameterId>D.ee.ff</parameterId>
			<locationId>Delft</locationId>
			<timeSeriesType>external historical</timeSeriesType>
			<timeStep unit="minute" multiplier="10"/>
			<readWriteMode>add originals</readWriteMode>
		</timeSeriesSet>
	</import>
</timeSeriesImportRun>

IdMapFiles

ID mapping defines mappings between Diver MON and FEWS parameters and locations. Remember that ID mapping is case sensitive.

Code Block
xml
xml
titlesample of IdImportMON.xml

<idMap version="1.1"  ..............>
    <parameter internal="G.hh.ii" external="LEVEL"/>
	<parameter internal="J.kk.ll" external="TEMPERATURE"/>

    <parameter internal="X.yy.zz" external="PCH"/>
	<parameter internal="A.bb.cc" external="HG"/>
	<parameter internal="D.ee.ff" external="2: SPEC.COND."/>

	<location internal="Utrecht" external="13PB1"/>
	<location internal="Delft" external="ro954"/>
</idMap>

Example file

There is a wide range of MON file types, here are examples of files with 2 and 3 channels respectively.

Code Block
xml
xml
titlesample of 13PB175646_03_16_0708_23_07.mon with 2 channels

Data file for DataLogger.
==============================================================================
COMPANY    : Your Company Name
LICENSE    :
DATE       : 08/24/07
TIME       : 11:20:44
FILENAME   : H:\Grondwater\DLG-Mariapeel NIEUW\24-8-2007\13PB175646_03_16_0708_23_07.mon
CREATED BY : EnviroMon version 1.9
==========================    BEGINNING OF DATA     ==========================
[Instrument info]
  Instrument type         =Diver =3
  Instrument state        =Stopped =2
  Serial number           =.N00-75646  210.
  Instrument number       =13PB1
                          =0.00000000000000E-0000
  Location                =13PB1
  Sample Rate             =00 01:00:00 0
  Sample Mode             =T
  Channel                 =2
[Channel 1]
  Identification          =LEVEL
  Reference               =0.0       cm
  Range                   =500.0     cm
  Master Level            =0.0       cm
  Altitude                =0         m
[Channel 2]
  Identification          =TEMPERATURE
  Reference               =-20.00    deg C
  Range                   =100.00    deg C
  Master Level            =100.00    deg C
  Altitude                =-20.00    deg C
[Instrument info from data header]
  Serial number           =.N00-75646  210.
  Instrument number       =13PB1
  Location                =13PB1
  Sample Rate             =00 01:00:00 0
  Sample Mode             =T
  Tb                      =22:47:10 16/03/07
  Te                      =22:47:10 23/08/07
[Channel 1 from data header]
  Identification          =LEVEL
  Reference               =0.0       cm
  Range                   =500.0     cm
  Master Level            =0.0       cm
  Altitude                =0         m
[Channel 2 from data header]
  Identification          =TEMPERATURE
  Reference               =-20.00    deg C
  Range                   =100.00    deg C
  Master Level            =100.00    deg C
  Altitude                =-20.00    deg C
[Data]
3841
2007/03/16 10:47:22.0        93.2       11.61
2007/03/16 11:47:22.0       209.7        8.96
2007/03/16 12:47:22.0       209.4        8.93
2007/03/16 13:47:22.0       208.9        8.94
2007/03/16 14:47:22.0       208.5        8.95
2007/03/16 15:47:22.0       207.9        8.95
2007/03/16 16:47:22.0       207.7        8.96

Example file

...

 Code Block
xml
xml
titlesample of 13PB175646ro954_03_16_0708_23_07.mon131022072229_M0026.MON with 3 channels
Data file for DataLogger.
==============================================================================
COMPANY    : Your Company Name
LICENSE    : 
DATE    : <Company  name>
COMP.STATUS: 08/24/07
TIMEDo
DATE       : 11:20:44
FILENAME   : H:\Grondwater\DLG-Mariapeel NIEUW\24-8-2007\13PB175646_03_16_0708_23_07.mon
CREATED BY : EnviroMon version 1.9
==========================    BEGINNING OF DATA     =22/10/2013
TIME       : 07:22:29
FILENAME   : D:\diver offive\vistrap\MON\ro954_131022072229_M0026.MON
CREATED BY : SWS Diver-Office 5.0.25.0
=========================
[Instrument info]
  Instrument type         =Diver =3
  Instrument state        =Stopped =2
  Serial number BEGINNING OF DATA        =.N00-75646  210.==========================
[Logger settings]
  Instrument numbertype       =13PB1     =CTD-Diver=17
  Status        
          =Started =0
  Serial  number           =0.00000000000000E-0000N10-M0026  317.
  LocationInstrument number       =
        =13PB1
  Sample Rate             =00 01:00:00 =0
  SampleLocation Mode             =T   =ro954
  Channel     Sample period            =2M10
[Channel 1]
  Identification  Sample method           =LEVELT
  Number of channels      =3
[Channel 1]
  Identification  
  Reference      =PRESSURE
  Reference level      =0.0    =400   cm    cm
  Range                   =500.05750      cm    
  Master Levellevel            =0.05750       cm    CMH2O
  Altitude                =0         m     m
[Channel 2]
  Identification          =TEMPERATURE         
  Reference   level            =-20.00    deg C °C
  Range                   =100.00    deg C °C


[Channel 3]
  MasterIdentification Level            =100.00    deg C 
  Altitude     =2: SPEC.COND.
  Reference level         =0.000  =-20.00   mS/cm
 deg CRange 
[Instrument info from data header]
  Serial number           =120.N00-75646  210.000   mS/cm


[Series settings]
  InstrumentSerial number       =13PB1    =.N10-M0026  317.
  Instrument number       =
  Location                =13PB1ro954
  Sample Rate  period           =00 0100:0010:00 0
  Sample Modemethod             =T   
  TbStart date /   time                 =22:47:10 16/03/07
  Te             =00:00:08 12/09/13
  End date / time         =2200:4720:1006 2322/0810/0713
[Channel 1 from data header]
  Identification          =LEVEL               PRESSURE
  Reference      level         =0.0400       cm    
  Range                   =500.05750      cm    
  Master Level   Master level        =0.0    =5750   cm    CMH2O
  Altitude                =0         m     
[Channel 2 from data header]
  Identification          =TEMPERATURE
  Reference level         =-20.00    °C
  Range    Reference               =-20100.00    °C


[Channel 3 degfrom Cdata header]
  Identification    Range      =2: SPEC.COND.
  Reference level         =1000.00000    deg C mS/cm
  MasterRange Level            =100.00    deg C 
=120.000  Altitude     mS/cm


[Data]
5751
2013/09/12 08:00:00.0         1332   =-20.00    deg18.18 C 
[Data]      
3841
2007/03/16 10:47:22.0 0.824
2013/09/12 08:10:00.0       93.2       11.61
2007/03/16 11:47:22.0  1326       20918.718        80.96828
20072013/0309/1612 1208:4720:2200.0         209.41328       18.17        80.93824
20072013/0309/1612 1308:47:22.030:00.0         1329       20818.916        80.94828
20072013/0309/1612 1408:4740:2200.0        208.5 1329       8.95
2007/03/16 15:47:22.018.17         207.90.824
2013/09/12 08:50:00.0        8.95
2007/03/16 16:47:22.0 1328       20718.721        80.96
 
Accepted Mon file Sections and Parameters
The Mon Import module in Delft-FEWS does not parse all data in the MON file. The important sections and parameters are the following:
824
2013/09/12 09:00:00.0         1329       18.23       0.820
 
Accepted Mon file Sections and Parameters
The Mon Import module in Delft-FEWS does not parse all data in the MON file. The important sections and parameters are the following:
  • [Logger settings] or [Instrument info]; Read location Id from this section 
    • Location or Locatie ; location Id
  • [Channel X] or [Kanaal X] ; Not used
  • [Series settings] or [Instrument info from data header] ; Not used
  • [Channel X from data header] or [Kanaal X from data header]; Read Parameter 
  •  Wiki Markup
    \[Logger settings\] or \[Instrument info\]; Read location Id from this section
    • Location or Locatie ; location Id
  •  Wiki Markup
    \[Channel 1\] or \[Kanaal 1\] ; Not used
  •  Wiki Markup
    \[Channel 2\] or \[Kanaal 2\] ; Not used
  •  Wiki Markup
    \[Series settings\] or \[Instrument info from data header\] ; Not used
  •  Wiki Markup
    \[Channel 1 from data header\] or \[Kanaal 1 from data header\]; Read Parameter Id
    • Identification or Identificatie = Parameter Id
  •  Wiki Markup\[Channel 2 from data header\] or \[Kanaal 2 from data header\]; Read Parameter  Id 
    • Identification or Identificatie = Parameter Id
  •  Wiki Markup\[Data\];   Data   values   with   the   different   channels   in   columns.   The   datavalues   may   have   a   "."or   a   ","as   decimal   seperator,   both   options   are   accepter   by   the   import   function.
When the MON file is not in the correct format a warning message is returned. Known problems are missing location ID's or parameter ID's in the MON files.
Java source code
DiverMonTimeSeriesParser.java