When the data file is available through an url, you can download the file first, before you run your normal import module. The process involves 3 simple steps. A (simple) example is given for data available through https://apps.des.qld.gov.au/data-sets/storm-tides/

Note: the importmodule of Delft-FEWS is able to directly read from ftp or http(s) anyhow.

.bat (or .sh) file with the curl command

Create a bat file (eg. download_with_curl.bat) with the following content: 

curl "%1" %2 %3

Note 1: The arguments will be provided by Delft-FEWS in the next step.

Note 2: You can create a ModuleDataSetFile from this .bat file, so that you can export it in the next step. This is not strictly necessary, but is a nice way to include the .bat file in the Config files.

GA module to run the bat file

<generalAdapterRun xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/generalAdapterRun.xsd">
    <general>
        <rootDir>$IMPORT_FOLDER$/scalar/opendataportal</rootDir>
        <workDir>%ROOT_DIR%</workDir>
        <exportDir>%ROOT_DIR%/</exportDir>
        <exportDataSetDir>%ROOT_DIR%</exportDataSetDir>
        <importDir>%ROOT_DIR%/</importDir>
        <dumpFileDir>$GA_DUMPFILEDIR$</dumpFileDir>
        <dumpDir>%ROOT_DIR%</dumpDir>
        <diagnosticFile>%ROOT_DIR%/piDiagnostic.xml</diagnosticFile>
        <missVal>-99</missVal>
        <convertDatum>true</convertDatum>
    </general>
    <activities>
        <startUpActivities>
            <purgeActivity>
                <filter>%ROOT_DIR%/*.*</filter>
            </purgeActivity>
        </startUpActivities>
        <exportActivities>
        	<exportDataSetActivity>
				<moduleInstanceId>download_with_curl</moduleInstanceId>
			</exportDataSetActivity>
        </exportActivities>
        <executeActivities>
            <executeActivity>
                <description>download tide data</description>
                <command>
                    <executable>download_with_curl.bat</executable>
                </command>
                <arguments>
                    <argument>https://apps.des.qld.gov.au/data-sets/storm-tides/stdtide-7dayopdata.csv</argument>
                    <argument>--output</argument>
                    <argument>stdtide-7dayopdata.csv</argument>
                </arguments>
                <timeOut>3600000</timeOut>
                <ignoreDiagnostics>true</ignoreDiagnostics>
            </executeActivity>
        </executeActivities>
    </activities>
</generalAdapterRun>

Import module for the actual import

<timeSeriesImportRun xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/timeSeriesImportRun.xsd">
    <import>
        <general>
            <importType>generalCSV</importType>
            <folder>$IMPORT_FOLDER$/scalar/opendataportal</folder>
            <fileNamePatternFilter>stdtide-7dayopdata*.csv</fileNamePatternFilter>
            <failedFolder>$IMPORT_FAILED_FOLDER$/opendataportal/</failedFolder>
            <deleteImportedFiles>false</deleteImportedFiles>
            <table>
                <locationColumn name="Site"/>
                <dateTimeColumn name="DateTime" pattern="yyyy-MM-dd'T'HH:mm"/>
                <valueColumn name="Water Level" parameterId="H.obs" unit="m"/>
                <valueColumn name="Prediction" parameterId="H.fc" unit="m"/>
                <valueColumn name="Residual" parameterId="H.res" unit="m"/>
            </table>
            <logWarningsForUnmappableTimeSeries>true</logWarningsForUnmappableTimeSeries>
            <idMapId>IdImportOpenDataPortal</idMapId>
            <unitConversionsId>ImportUnitConversions</unitConversionsId>
            <missingValue>-99</missingValue>
            <importTimeZone>
                <timeZoneName>AET</timeZoneName>
            </importTimeZone>
            <dataFeedId>OpenDataPortal</dataFeedId>
            <expiryTime unit="day" multiplier="7"/>
            <skipFirstLinesCount>1</skipFirstLinesCount>
        </general>
        <timeSeriesSet>
            <moduleInstanceId>ImportObservations</moduleInstanceId>
            <valueType>scalar</valueType>
            <parameterId>H.obs</parameterId>
            <locationSetId>stations_tide_opendataportal</locationSetId>
            <timeSeriesType>external historical</timeSeriesType>
            <timeStep unit="nonequidistant"/>
            <readWriteMode>add originals</readWriteMode>
        </timeSeriesSet>
        <timeSeriesSet>
            <moduleInstanceId>ImportObservations</moduleInstanceId>
            <valueType>scalar</valueType>
            <parameterId>H.fc</parameterId>
            <locationSetId>stations_tide_opendataportal</locationSetId>
            <timeSeriesType>external historical</timeSeriesType>
            <timeStep unit="nonequidistant"/>
            <readWriteMode>add originals</readWriteMode>
        </timeSeriesSet>
        <timeSeriesSet>
            <moduleInstanceId>ImportObservations</moduleInstanceId>
            <valueType>scalar</valueType>
            <parameterId>H.res</parameterId>
            <locationSetId>stations_tide_opendataportal</locationSetId>
            <timeSeriesType>external historical</timeSeriesType>
            <timeStep unit="nonequidistant"/>
            <readWriteMode>add originals</readWriteMode>
        </timeSeriesSet>
    </import>
</timeSeriesImportRun>
  • No labels

Disclaimer