Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Migration of unmigrated content due to installation of a new plugin

Overview

Imports time series data from Diver loggers. The files have a sort of Windows ini file format with file extension

...

(*.mon).

...

The

...

format

...

of

...

the

...

MON

...

ini

...

files

...

is

...

not

...

well

...

defined.

...

Many

...

programs

...

interpret

...

the

...

structure

...

differently

...

and

...

have

...

various

...

names

...

for

...

the

...

ini

...

file

...

sections

...

and

...

parameters.

...

  • Sections:

...

  • Section

...

  • declarations

...

  • start with

...

  • '

...

  • ['

...

  • and

...

  • end with

...

  • '

...

  • ]';

...

  • i.e.

...

  • '[Logger

...

  • settings]'

...

  • or

...

  • '[Instrument

...

  • info]'.

...

  • Parameters

...

  • or

...

  • item:

...

  • this

...

  • is

...

  • the

...

  • content

...

  • of

...

  • a

...

  • section

...

  • with

...

  • an

...

  • '='

...

  • sign

...

  • between

...

  • the

...

  • key

...

  • and

...

  • the

...

  • value;

...

  • i.e.

...

  • "location

...

  • =

...

  • abc"

...

The

...

Date

...

format

...

used

...

is:

...

"yyyy/MM/dd

...

HH:mm:ss"

The files can contain a flexible number of channels, where a channel will contain information on one parameter. Example files with 2 and 3 channels will be shown, when multiple channels/parameters need to be imported a <timeSeriesSet> needs to be configured for each channel/parameter. To illustrate this an example config for an import with 3 channels is given.

Configuration (Example)

A complete import module configuration consists of an ID Mapping file and a Import Module Instance file.

ModuleConfigFiles

The following example of an Import Module Instance will import the time series as equidistant series for timezone GMT+1 with a time step of 1 hour. Many times the MON files do not save the data at rounded hourly tims, therefore a tolerance has been added to map the imported data to correct hourly interval time series.

Code Block
xml
xml
titleImportMON.xml


h2. Configuration (Example)

A complete import module configuration consists of an ID Mapping file and a Import Module Instance file.

h3. ModuleConfigFiles

The following example of an Import Module Instance will import the time series as equidistant series for timezone GMT+1 with a time step of 1 hour. Many times the MON files do not save the data at rounded hourly tims, therefore a tolerance has been added to map the imported data to correct hourly interval time series.
{code:xml|title=ImportMON.xml}
<?xml version="1.0" encoding="UTF-8"?>
<timeSeriesImportRun ......">
  <import>
    <general>
      <importType>DIVERMON</importType>
      <folder>$IMPORT_FOLDER_MON$</folder>
      <failedFolder>$IMPORT_FAILED_FOLDER_MON$</failedFolder>
      <backupFolder>$IMPORT_BACKUP_FOLDER_MON$</backupFolder>
      <idMapId>IdImportMON</idMapId>
      <unitConversionsId>ImportUnitConversions</unitConversionsId>
      <importTimeZone>
        <timeZoneOffset>+01:00</timeZoneOffset>
      </importTimeZone>
      <dataFeedId>MON</dataFeedId>
    </general>
    <tolerance locationSetId="ImportMON_H.meting.cm_uur" parameterId="H.meting.cm" timeUnit="minute" unitCount="30"/>
    <timeSeriesSet>
      <moduleInstanceId>ImportMON</moduleInstanceId>
      <valueType>scalar</valueType>
      <parameterId>H.meting.cm</parameterId>
      <locationSetId>ImportMON_H.meting.cm_uur</locationSetId>
      <timeSeriesType>external historical</timeSeriesType>
      <timeStep unit="hour" multiplier="1"/>
      <readWriteMode>add originals</readWriteMode>
      <synchLevel>1</synchLevel>
    </timeSeriesSet>
  </import>
</timeSeriesImportRun>
{code}

h3. IdMapFiles
ID mapping fefines mappings between Diver MON and FEWS parameters and locations. Remember that ID mapping is case sensitive.
{code:xml|title=sample of IdImportMON.xml}
<idMap version="1.1"  ..............>
    <map internalParameter="H.meting.cm" internalLocation="Dorperdijk_beneden" externalParameter="niveau" externalLocation="Dorperdijk beneden"/>
</idMap>
{code}

h3. Example file
There is a wide range of MON file types, this is just one example.
{code:xml|title=sample of 13PB175646_03_16_0708_23_07.mon}
Data file for DataLogger.
==============================================================================
COMPANY    : Your Company Name
LICENSE    : 
DATE       : 08/24/07
TIME       : 11:20:44
FILENAME   : H:\Grondwater\DLG-Mariapeel NIEUW\24-8-2007\

Here is an example config file for an import file with three channels:

Code Block
xml
xml
titleImportMON_3channels.xml

<?xml version="1.0" encoding="UTF-8"?>
<timeSeriesImportRun ......">
  <import>
		<general>
			<importType>DiverMon</importType>
			<folder>$IMPORT_FOLDER$/DiverMon</folder>
			<idMapId>IdMap</idMapId>
			<missingValue>-999.0</missingValue>
			<expiryTime unit="day" multiplier="366"/>
		</general>
		<timeSeriesSet>
			<moduleInstanceId>ImportMON</moduleInstanceId>
			<valueType>scalar</valueType>
			<parameterId>X.yy.zz</parameterId>
			<locationId>Delft</locationId>
			<timeSeriesType>external historical</timeSeriesType>
			<timeStep unit="minute" multiplier="10"/>
			<readWriteMode>add originals</readWriteMode>
		</timeSeriesSet>
		<timeSeriesSet>
			<moduleInstanceId>ImportMON</moduleInstanceId>
			<valueType>scalar</valueType>
			<parameterId>A.bb.cc</parameterId>
			<locationId>Delft</locationId>
			<timeSeriesType>external historical</timeSeriesType>
			<timeStep unit="minute" multiplier="10"/>
			<readWriteMode>add originals</readWriteMode>
		</timeSeriesSet>
		<timeSeriesSet>
			<moduleInstanceId>ImportMON</moduleInstanceId>
			<valueType>scalar</valueType>
			<parameterId>D.ee.ff</parameterId>
			<locationId>Delft</locationId>
			<timeSeriesType>external historical</timeSeriesType>
			<timeStep unit="minute" multiplier="10"/>
			<readWriteMode>add originals</readWriteMode>
		</timeSeriesSet>
	</import>
</timeSeriesImportRun>

IdMapFiles

ID mapping defines mappings between Diver MON and FEWS parameters and locations. Remember that ID mapping is case sensitive.

Code Block
xml
xml
titlesample of IdImportMON.xml

<idMap version="1.1"  ..............>
    <parameter internal="G.hh.ii" external="LEVEL"/>
	<parameter internal="J.kk.ll" external="TEMPERATURE"/>

    <parameter internal="X.yy.zz" external="PCH"/>
	<parameter internal="A.bb.cc" external="HG"/>
	<parameter internal="D.ee.ff" external="2: SPEC.COND."/>

	<location internal="Utrecht" external="13PB1"/>
	<location internal="Delft" external="ro954"/>
</idMap>

Example file

There is a wide range of MON file types, here are examples of files with 2 and 3 channels respectively.

Code Block
xml
xml
titlesample of 13PB175646_03_16_0708_23_07.mon with 2 channels
CREATEDData BYfile : EnviroMon version 1.9
for DataLogger.
===========================    BEGINNING OF DATA     ===================================================
[InstrumentCOMPANY info]
  Instrument type: Your Company Name
LICENSE      =Diver =3
:
DATE  Instrument state    : 08/24/07
TIME   =Stopped =2
  Serial number : 11:20:44
FILENAME   : H:\Grondwater\DLG-Mariapeel NIEUW\24-8-2007\13PB175646_03_16_0708_23_07.mon
CREATED BY : EnviroMon version =.N00-75646  210.
  Instrument number       =13PB11.9
==========================    BEGINNING OF DATA         
       ==========================
[Instrument info]
  Instrument type         =Diver =3
  Instrument state      =0.00000000000000E-0000
  Location=Stopped =2
  Serial  number           =13PB1
.N00-75646  Sample210.
 Rate Instrument number       =13PB1
     =00 01:00:00 0
  Sample Mode             =T   =0.00000000000000E-0000
  Channel Location                =2
[Channel 1]13PB1
  IdentificationSample Rate         =LEVEL    =00 01:00:00 0
  Sample Mode      
  Reference     =T
  Channel        =0.0       cm  =2
[Channel 1]
  Identification      
  Range  =LEVEL
  Reference               =5000.0       cm
  Range   
   Master Level            =0500.0       cm
  Master Level 
  Altitude         =0.0       =0cm
  Altitude       m     
[Channel 2]
  Identification =0         =TEMPERATUREm
[Channel 2]
  Identification          =TEMPERATURE
  Reference               =-20.00    deg C 
  Range                   =100.00    deg C 
  Master Level            =100.00    deg C 
  Altitude                =-20.00    deg C 
[Instrument info from data header]
  Serial number           =.N00-75646  210.
  Instrument number       =13PB1               
  Location                =13PB1
  Sample Rate             =00 01:00:00 0
  Sample Mode             =T   
  Tb                      =22:47:10 16/03/07
  Te                      =22:47:10 23/08/07
[Channel 1 from data header]
  Identification          =LEVEL               
  Reference               =0.0       cm    
  Range                   =500.0     cm    
  Master Level            =0.0       cm
  Altitude  
  Altitude                =0         m     
[Channel 2 from data header]
  Identification          =TEMPERATURE         
  Reference               =-20.00    deg C 
  Range                   =100.00    deg C 
  Master Level            =100.00    deg C 
  Altitude                =-20.00    deg C 
[Data]      
3841
2007/03/16 10:47:22.0        93.2       11.61
2007/03/16 11:47:22.0       209.7        8.96
2007/03/16 12:47:22.0       209.4        8.93
2007/03/16 13:47:22.0       208.9        8.94
2007/03/16 14:47:22.0       208.5        8.95
2007/03/16 15:47:22.0       207.9        8.95
2007/03/16 16:47:22.0       207.7        8.96
 {code}

h2. Accepted Mon file Sections and Parameters
The Mon Import module in Delft-FEWS does not parse all data in the MON file. The important sections and parameters are the following:

* [Logger settings] or [Instrument info]; Read location Id from this section
** Location or Locatie ; location Id 
* [Channel 1] or [Kanaal 1] ; Not used
* [Channel 2] or [Kanaal 2] ; Not used
* [Series settings] or [Instrument info from data header] ; Not used
* [Channel 1 from data header] or [Kanaal 1 from data header]; Read Parameter Id
** Identification or Identificatie = Parameter Id
* [Channel 2 from data header] or [Kanaal 2 from data header]; Read Parameter Id
** Identification or Identificatie = Parameter Id
* [Data]	Data values with the different channels in columns

When the MON file is not in the correct format a warning message is returned.


{checklist:name=Oviparious|parent=Animals|excerpt-heading=Classification|label=oviparous|checklabels=fish, amphibians, reptiles, birds|mutuallyexclusive=true}

Code Block
xml
xml
titlesample of ro954_131022072229_M0026.MON with 3 channels

Data file for DataLogger.
==============================================================================
COMPANY    : <Company name>
COMP.STATUS: Do
DATE       : 22/10/2013
TIME       : 07:22:29
FILENAME   : D:\diver offive\vistrap\MON\ro954_131022072229_M0026.MON
CREATED BY : SWS Diver-Office 5.0.25.0
==========================    BEGINNING OF DATA     ==========================
[Logger settings]
  Instrument type         =CTD-Diver=17
  Status                  =Started =0
  Serial number           =.N10-M0026  317.
  Instrument number       =
                          =0
  Location                =ro954
  Sample period           =M10
  Sample method           =T
  Number of channels      =3
[Channel 1]
  Identification          =PRESSURE
  Reference level         =400       cm
  Range                   =5750      cm
  Master level            =5750      CMH2O
  Altitude                =0         m
[Channel 2]
  Identification          =TEMPERATURE
  Reference level         =-20.00    °C
  Range                   =100.00    °C


[Channel 3]
  Identification          =2: SPEC.COND.
  Reference level         =0.000     mS/cm
  Range                   =120.000   mS/cm


[Series settings]
  Serial number           =.N10-M0026  317.
  Instrument number       =
  Location                =ro954
  Sample period           =00 00:10:00 0
  Sample method           =T
  Start date / time       =00:00:08 12/09/13
  End date / time         =00:20:06 22/10/13
[Channel 1 from data header]
  Identification          =PRESSURE
  Reference level         =400       cm
  Range                   =5750      cm
  Master level            =5750      CMH2O
  Altitude                =0         m
[Channel 2 from data header]
  Identification          =TEMPERATURE
  Reference level         =-20.00    °C
  Range                   =100.00    °C


[Channel 3 from data header]
  Identification          =2: SPEC.COND.
  Reference level         =0.000     mS/cm
  Range                   =120.000   mS/cm


[Data]
5751
2013/09/12 08:00:00.0         1332       18.18       0.824
2013/09/12 08:10:00.0         1326       18.18       0.828
2013/09/12 08:20:00.0         1328       18.17       0.824
2013/09/12 08:30:00.0         1329       18.16       0.828
2013/09/12 08:40:00.0         1329       18.17       0.824
2013/09/12 08:50:00.0         1328       18.21       0.824
2013/09/12 09:00:00.0         1329       18.23       0.820
 

Accepted Mon file Sections and Parameters

The Mon Import module in Delft-FEWS does not parse all data in the MON file. The important sections and parameters are the following:

  • [Logger settings] or [Instrument info]; Read location Id from this section
    • Location or Locatie ; location Id
  • [Channel X] or [Kanaal X] ; Not used
  • [Series settings] or [Instrument info from data header] ; Not used
  • [Channel X from data header] or [Kanaal X from data header]; Read Parameter Id
    • Identification or Identificatie = Parameter Id
  • [Data]; Data values with the different channels in columns. The datavalues may have a "."or a ","as decimal seperator, both options are accepter by the import function.

When the MON file is not in the correct format a warning message is returned. Known problems are missing location ID's or parameter ID's in the MON files.

Java source code

DiverMonTimeSeriesParser.java