When the data file is available through an url, you can download the file first, before you run your normal import module. The process involves 3 simple steps. A (simple) example is given for data available through https://apps.des.qld.gov.au/data-sets/storm-tides/
.bat (or .sh) file with the curl command
Create a bat file (eg. download_with_curl.bat) with the following content:
curl "%1" %2 %3
Note 1: The arguments will be provided by Delft-FEWS in the next step.
Note 2: You can create a ModuleDataSetFile from this .bat file, so that you can export it in the next step. This is not strictly necessary, but is a nice way to include the .bat file in the Config files.
GA module to run the bat file
fdsa
Code Block |
---|
generalAdapterRun xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/generalAdapterRun.xsd">
<general>
<rootDir>$IMPORT_FOLDER$/scalar/opendataportal</rootDir>
<workDir>%ROOT_DIR%</workDir>
<exportDir>%ROOT_DIR%/</exportDir>
<exportDataSetDir>%ROOT_DIR%</exportDataSetDir>
<importDir>%ROOT_DIR%/</importDir>
<dumpFileDir>$GA_DUMPFILEDIR$</dumpFileDir>
<dumpDir>%ROOT_DIR%</dumpDir>
<diagnosticFile>%ROOT_DIR%/piDiagnostic.xml</diagnosticFile>
<missVal>-99</missVal>
<convertDatum>true</convertDatum>
</general>
<activities>
<startUpActivities>
<purgeActivity>
<filter>%ROOT_DIR%/*.*</filter>
</purgeActivity>
</startUpActivities>
<exportActivities>
<exportDataSetActivity>
<moduleInstanceId>download_with_curl</moduleInstanceId>
</exportDataSetActivity>
</exportActivities>
<executeActivities>
<executeActivity>
<description>download tide data</description>
<command>
<executable>download_with_curl.bat</executable>
</command>
<arguments>
<argument>https://apps.des.qld.gov.au/data-sets/storm-tides/stdtide-7dayopdata.csv</argument>
<argument>--output</argument>
<argument>stdtide-7dayopdata.csv</argument>
</arguments>
<timeOut>3600000</timeOut>
<ignoreDiagnostics>true</ignoreDiagnostics>
</executeActivity>
</executeActivities>
</activities>
</generalAdapterRun> |
Import module for the actual import
Code Block |
---|
<timeSeriesImportRun xmlns="http://www.wldelft.nl/fews" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.wldelft.nl/fews http://fews.wldelft.nl/schemas/version1.0/timeSeriesImportRun.xsd">
<import>
<general>
<importType>generalCSV</importType>
<folder>$IMPORT_FOLDER$/scalar/opendataportal</folder>
<fileNamePatternFilter>stdtide-7dayopdata*.csv</fileNamePatternFilter>
<failedFolder>$IMPORT_FAILED_FOLDER$/opendataportal/</failedFolder>
<deleteImportedFiles>false</deleteImportedFiles>
<table>
<locationColumn name="Site"/>
<dateTimeColumn name="DateTime" pattern="yyyy-MM-dd'T'HH:mm"/>
<valueColumn name="Water Level" parameterId="H.obs" unit="m"/>
<valueColumn name="Prediction" parameterId="H.fc" unit="m"/>
<valueColumn name="Residual" parameterId="H.res" unit="m"/>
</table>
<logWarningsForUnmappableTimeSeries>true</logWarningsForUnmappableTimeSeries>
<idMapId>IdImportOpenDataPortal</idMapId>
<unitConversionsId>ImportUnitConversions</unitConversionsId>
<missingValue>-99</missingValue>
<importTimeZone>
<timeZoneName>AET</timeZoneName>
</importTimeZone>
<dataFeedId>OpenDataPortal</dataFeedId>
<expiryTime unit="day" multiplier="7"/>
<skipFirstLinesCount>1</skipFirstLinesCount>
</general>
<timeSeriesSet>
<moduleInstanceId>ImportObservations</moduleInstanceId>
<valueType>scalar</valueType>
<parameterId>H.obs</parameterId>
<locationSetId>stations_tide_opendataportal</locationSetId>
<timeSeriesType>external historical</timeSeriesType>
<timeStep unit="nonequidistant"/>
<readWriteMode>add originals</readWriteMode>
</timeSeriesSet>
<timeSeriesSet>
<moduleInstanceId>ImportObservations</moduleInstanceId>
<valueType>scalar</valueType>
<parameterId>H.fc</parameterId>
<locationSetId>stations_tide_opendataportal</locationSetId>
<timeSeriesType>external historical</timeSeriesType>
<timeStep unit="nonequidistant"/>
<readWriteMode>add originals</readWriteMode>
</timeSeriesSet>
<timeSeriesSet>
<moduleInstanceId>ImportObservations</moduleInstanceId>
<valueType>scalar</valueType>
<parameterId>H.res</parameterId>
<locationSetId>stations_tide_opendataportal</locationSetId>
<timeSeriesType>external historical</timeSeriesType>
<timeStep unit="nonequidistant"/>
<readWriteMode>add originals</readWriteMode>
</timeSeriesSet>
</import>
</timeSeriesImportRun> |
...