BIL BIP BSQ Parser

Overview

The BIL, BIP and BSQ raster data formats were originally developed by USGS for CCT tape formats to store Landsat satellite data from 1972 on. These are simple uncompressed binary files representing a raster of one or more image bands, accompanied by an ASCII header file (*.hdr) with image information. The main file extension depends on the format, or storage sequence:

*.BIL : Bands Interleaved by Line

*.BIP : Bands Interleved by Pixel

*.BSQ : Bands Sequential

File Format

The file format was documented by ESRI online. (PDF version) and supports integer (1- to 32- bit) pixel values in signed or unsigned format, in either little- or big endian storage order. The 1- and 4- bit integer pixel formats have not been implemented (yet) as they seem less useful for FEWS, but the rest of the documentation is fully implemented.

Note: It is assumed that your data is using unsigned integers, unless the keyword "pixeltype" is used and its value is"signedint".

BIL files in FEWS

In the Public Interface (PI interface) of FEWS, an implementation of the BIL format is also provided, as documented here.

The PI implementation of BIL files however includes several adaptations to specific FEWS purposes (i.e. using floating point data, extension for time series using blocks) which causes this FEWS implementation to not fully comply with the official specifications of the BIL format and lack support for BIP and BSQ format files. Changing this implementation would break backward compatibility so it was decided to create a new parser, which also properly supports the BIL, BIP and BSQ format.

This new implementation is fully compliant with the ESRI technical specifications and can be configured with the new "BILBIPBSQ" identifier, whereas the existing PI implementation of FEWS BIL files uses the identifier "BIL".

Using NoData for missing values

In the header files (*.HDR) which normally come with a BIL / BIP / BSQ file, a NoData field can be used to define the numerical value which has been assigned to pixels for which there is no valid data.  This field is not mentioned in the ESRI documentation but it's usage is very common, as follows:

	NODATA -999

The result if this example is identical to defining <missingValue>-999</missingValue> in the import configuration.

Using NBLOCKS for time series

For many applications multiple BIL, BIP or BSQ files could be available, representing a time series of observations (daily, monthly, weekly, etc.)

In order to be able to import such a time series of Images, the binary image files can be appended into a single file in chronological order where the original files become so-called "blocks" in the combined image file. In this case an extra field NBLOCKS is added to the header file to indicate the number of time steps in the combined image and an extra (*.tim) file is added to provide a date/time stamp for each block.

Appending the binary image files (each having the same format and dimensions) to create a time series can be done using (for example) a DOS copy command in binary append mode, as follows:

	copy /b step1.bil+step2.bil+step3.bil+step4.bil 4steps.bil

A header file for the resulting time series can be made by copying and renaming any of the input header files (*.hdr) and adding an extra line to define the number of file blocks:

	NBLOCKS		4

If the header contains a definition of NBLOCKS larger than 1, an extra file (*.tim) is required to define a timestamp per block in the format YYYYMMDDHHMM:

200001011200
200001011300
200001011400
200001011500

Note: The way the NBLOCKS parameter is added to the header file (*.hdr) and an extra file with time step definitions (*.tim) is added is practically the same as how these are used by the PI-BIL parser, but one should realize that the order in which BANDS (parameters) and BLOCKS (time steps) are stored is not the same, so the files are not compatible unless either NBANDS or NBLOCKS equals 1.

Configuration

	<location id="SATH_AFRICA" name="SATH_AFRICA">
		<shortName>SATH_AFRICA</shortName>
		<x>0</x>
		<y>0</y>
	</location>

	<regular locationId="SATH_AFRICA">
		<description>SATH Rainfall files</description>
		<rows>702</rows>
		<columns>1002</columns>
		<geoDatum>WGS 1984</geoDatum>
		<firstCellCenter>
			<x>-13</x>
			<y>25</y>
			<z>0</z>
		</firstCellCenter>
		<xCellSize>.03</xCellSize>
		<yCellSize>.03</yCellSize>
	</regular>

<idMap >
   <parameter internal="P.obs" external = "0"/>
   <parameter internal="T.obs" external = "1"/>
</idMap>

<timeSeriesImportRun>
   <import>
      <general>
         <importType>BILBIPBSQ</importType>
         <folder>$IMPORT_FOLDER$\BSQ</folder>
         <fileNameObservationDateTimePattern>'NI_'yyyyMMdd'.BSQ'</fileNameObservationDateTimePattern>         
         <idMapId>IdImportBSQ</idMapId>
         <unitConversionsId>ImportUnitConversions</unitConversionsId>
         <importTimeZone>
            <timeZoneOffset>+00:00</timeZoneOffset>
         </importTimeZone>
         <geoDatum>WGS 1984</geoDatum>
         <dataFeedId>SATH_AFRICA</dataFeedId>
      </general>
      <timeSeriesSet>
         <moduleInstanceId>ImportBSQ</moduleInstanceId>
         <valueType>grid</valueType>
         <parameterId>P.obs</parameterId>
         <locationId>SATH_AFRICA</locationId>
         <timeSeriesType>external historical</timeSeriesType>
         <timeStep unit="day"/>
         <readWriteMode>add originals</readWriteMode>
         <expiryTime unit="day" multiplier="30"/>
      </timeSeriesSet>
         <timeSeriesSet>
         <moduleInstanceId>ImportBSQ</moduleInstanceId>
         <valueType>grid</valueType>
         <parameterId>T.obs</parameterId>
         <locationId>SATH_AFRICA</locationId>
         <timeSeriesType>external historical</timeSeriesType>
         <timeStep unit="day"/>
         <readWriteMode>add originals</readWriteMode>
         <expiryTime unit="day" multiplier="30"/>
      </timeSeriesSet>
   </import>
</timeSeriesImportRun>