When the data file is available through an url, you can download the file first, before you run your normal import module. The process involves 3 simple steps. A (simple) example is given for data available through https://apps.des.qld.gov.au/data-sets/storm-tides/
Note: the importmodule of Delft-FEWS is able to directly read from ftp or http(s) anyhow.
.bat (or .sh) file with the curl command
...
GA module to run the bat file
| Code Block | ||
|---|---|---|
| ||
generalAdapterRun<generalAdapterRun xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/generalAdapterRun.xsd"> <general> <rootDir>$IMPORT_FOLDER$/scalar/opendataportal</rootDir> <workDir>%ROOT_DIR%</workDir> <exportDir>%ROOT_DIR%/</exportDir> <exportDataSetDir>%ROOT_DIR%</exportDataSetDir> <importDir>%ROOT_DIR%/</importDir> <dumpFileDir>$GA_DUMPFILEDIR$</dumpFileDir> <dumpDir>%ROOT_DIR%</dumpDir> <diagnosticFile>%ROOT_DIR%/piDiagnostic.xml</diagnosticFile> <missVal>-99</missVal> <convertDatum>true</convertDatum> </general> <activities> <startUpActivities> <purgeActivity> <filter>%ROOT_DIR%/*.*</filter> </purgeActivity> </startUpActivities> <exportActivities> <exportDataSetActivity> <moduleInstanceId>download_with_curl</moduleInstanceId> </exportDataSetActivity> </exportActivities> <executeActivities> <executeActivity> <description>download tide data</description> <command> <executable>download_with_curl.bat</executable> </command> <arguments> <argument>https://apps.des.qld.gov.au/data-sets/storm-tides/stdtide-7dayopdata.csv</argument> <argument>--output</argument> <argument>stdtide-7dayopdata.csv</argument> </arguments> <timeOut>3600000</timeOut> <ignoreDiagnostics>true</ignoreDiagnostics> </executeActivity> </executeActivities> </activities> </generalAdapterRun> |
Import module for the actual import
| Code Block | ||
|---|---|---|
| ||
<timeSeriesImportRun xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/timeSeriesImportRun.xsd">
<import>
<general>
<importType>generalCSV</importType>
<folder>$IMPORT_FOLDER$/scalar/opendataportal</folder>
<fileNamePatternFilter>stdtide-7dayopdata*.csv</fileNamePatternFilter>
<failedFolder>$IMPORT_FAILED_FOLDER$/opendataportal/</failedFolder>
<deleteImportedFiles>false</deleteImportedFiles>
<table>
<locationColumn name="Site"/>
<dateTimeColumn name="DateTime" pattern="yyyy-MM-dd'T'HH:mm"/>
<valueColumn name="Water Level" parameterId="H.obs" unit="m"/>
<valueColumn name="Prediction" parameterId="H.fc" unit="m"/>
<valueColumn name="Residual" parameterId="H.res" unit="m"/>
</table>
<logWarningsForUnmappableTimeSeries>true</logWarningsForUnmappableTimeSeries>
<idMapId>IdImportOpenDataPortal</idMapId>
<unitConversionsId>ImportUnitConversions</unitConversionsId>
<missingValue>-99</missingValue>
<importTimeZone>
<timeZoneName>AET</timeZoneName>
</importTimeZone>
<dataFeedId>OpenDataPortal</dataFeedId>
<expiryTime unit="day" multiplier="7"/>
<skipFirstLinesCount>1</skipFirstLinesCount>
</general>
<timeSeriesSet>
<moduleInstanceId>ImportObservations</moduleInstanceId>
<valueType>scalar</valueType>
<parameterId>H.obs</parameterId>
<locationSetId>stations_tide_opendataportal</locationSetId>
<timeSeriesType>external historical</timeSeriesType>
<timeStep unit="nonequidistant"/>
<readWriteMode>add originals</readWriteMode>
</timeSeriesSet>
<timeSeriesSet>
<moduleInstanceId>ImportObservations</moduleInstanceId>
<valueType>scalar</valueType>
<parameterId>H.fc</parameterId>
<locationSetId>stations_tide_opendataportal</locationSetId>
<timeSeriesType>external historical</timeSeriesType>
<timeStep unit="nonequidistant"/>
<readWriteMode>add originals</readWriteMode>
</timeSeriesSet>
<timeSeriesSet>
<moduleInstanceId>ImportObservations</moduleInstanceId>
<valueType>scalar</valueType>
<parameterId>H.res</parameterId>
<locationSetId>stations_tide_opendataportal</locationSetId>
<timeSeriesType>external historical</timeSeriesType>
<timeStep unit="nonequidistant"/>
<readWriteMode>add originals</readWriteMode>
</timeSeriesSet>
</import>
</timeSeriesImportRun> |
...