Introduction

This page describes the purpose and progress of the development of the image classification toolbox in the OpenEarthTools repository. The toolbox is developed in context of the NEMO and NatureCoast projects by Bas Hoonhout and Max Radermacher respectively. It will provide a fully automated and generic tool for classification arbitrary coastal images.

Applications

Applications of classification of coastal images are numerous. From classified images beach and vegetation surface areas can be extracted as is the instantaneous waterline. We can determine the intertidal area or the number of pixels related to beach goers and subsequently estimate the number of people at the beach.

Methodology

The intention is to develop a generic multi-step classification algorithm for coastal images. Each classification task will start with a coarse classification in pre-defined classes, like sky, sea, beach, dune and objects. More task-specific classification algorithms may subsequently classify the pixels in more detail, for example the number of people at the beach or the area of vegetation in the dunes.

Segmentation

First we segmentate the images in clusters of pixels: superpixels. Each superpixel is classified.

Classification

A single-step classification is done using a trained Conditional Random Field (CRF). A CRF is a type of probabilistic graphical model that computes the most likely classification of an image given data. Data consists of manually classified images.

Training dataset

Several training datasets with manually classified images are constructed:

  • Argus images from Dutch camera stations
  • Argus images from camera stations world-wide
  • Coastal images from arbitrary sources (including Flickr and Google)

Tools

A tool is developed to manually classify images. It is a webservice that is hosted at flamingo.tudelft.nl.

Progress

38%

Research

  1. handler

    Add LAB features

    Priority MEDIUM
    m.radermacher@tudelft.nl
    N/A
  2. handler

    Add relative location feature (Gould et al., 2008)

    Priority LOW
    m.radermacher@tudelft.nl
    12-12-2013
  3. handler

    Normalize features that depend on image resolution (lengths or areas)

    Priority HIGH
    m.radermacher@tudelft.nl
    07-01-2014
  4. handler

    Calibrate regularization parameter

    Priority MEDIUM
    hoonhout
    07-01-2014
  5. handler

    Add Gabor features

    Priority MEDIUM
    baart_f
    27-01-2014
  6. handler

    Investigate possibility to use SVD

    Priority LOW
    m.radermacher@tudelft.nl
    27-01-2014
  7. handler

    Large scale Gabor feature to detect the global orientation of the image

    Priority LOW
    hoonhout
    27-01-2014
  8. handler

    Reconsider segmentation algorithm

    Priority MEDIUM
    m.radermacher@tudelft.nl
    27-01-2014
  9. handler

    Compute learning curve

    Priority MEDIUM
    m.radermacher@tudelft.nl
    27-01-2014
  10. handler

    Analyze confusion matrix

    Priority MEDIUM
    m.radermacher@tudelft.nl
    27-01-2014
  11. handler

    Annotate 50 extra images

    Priority MEDIUM
    m.radermacher@tudelft.nl
    27-01-2014
  12. handler

    Optimize feature-set: 1-by-1 dropping of worst features

    Priority MEDIUM
    m.radermacher@tudelft.nl
    02-04-2014
  13. handler

    Check correlation of prediction errors and low certainty of class prediction

    Priority MEDIUM
    m.radermacher@tudelft.nl
    02-04-2014
50%

Tool

  1. handler

    Add Re-feature function

    Priority MEDIUM
    m.radermacher@tudelft.nl
    N/A
  2. handler

    Add convergence check to train function

    Priority MEDIUM
    hoonhout
    N/A
  3. handler

    Fix bug where fills of superpixels are shifted with respect to the class list (possibly related to missing superpixel bug)

    Priority LOW
    hoonhout
    N/A
  4. handler

    Fix bug where segmentation doesn't give a regular grid (n_segments != nx * ny)

    Priority LOW
    hoonhout
    N/A
  5. handler

    Fix error handling from subprocesses

    Priority MEDIUM
    hoonhout
    N/A
  6. handler

    Fix hotstart training

    Priority MEDIUM
    hoonhout
    N/A
  7. handler

    Fix reading original category list from trained model file when hotstart training

    Priority MEDIUM
    hoonhout
    N/A
  8. handler

    Make drawSuperpixelRaster save for missing superpixels (d= javascript error)

    Priority LOW
    hoonhout
    N/A
  9. handler

    Split data in train and test dataset and compute scores (when enough data is available)

    Priority HIGH
    m.radermacher@tudelft.nl
    N/A
  10. handler

    Include matrix features in training

    Priority HIGH
    m.radermacher@tudelft.nl
    12-12-2013
  11. handler

    Find out whether pairwise potentials can be made dependent on features (function of xi, xj, yi, yj for superpixels i and j)

    Priority LOW
    m.radermacher@tudelft.nl
    07-01-2014
  12. handler

    Annotate first batch of Argus images Dutch coast (50 - 100 pictures)

    Priority HIGH
    m.radermacher@tudelft.nl
    07-01-2014
  13. handler

    Temporarily switch off pairwise potentials withing gridCRF (essentially logistic regression)

    Priority HIGH
    hoonhout
    07-01-2014
  14. handler

    Cut top and bottom rows from images to omit labels

    Priority HIGH
    m.radermacher@tudelft.nl
    07-01-2014
  15. handler

    Feature normalization (mu = 0, sigma = 1) based on first batch of annotated images

    Priority HIGH
    m.radermacher@tudelft.nl
    07-01-2014
  16. handler

    Calibrate segmentation (explore options between 400 and 1000 superpixels and optimize compactness parameter)

    Priority HIGH
    m.radermacher@tudelft.nl
    07-01-2014
  17. handler

    Add bias term to argument of loss function (S) based on a priori class probabilities

    Priority MEDIUM
    m.radermacher@tudelft.nl
    07-01-2014
  18. handler

    Check why not all features are computed for all channels

    Priority HIGH
    m.radermacher@tudelft.nl
    15-01-2014
  19. handler

    Solve problems with identical feature values for different channels

    Priority HIGH
    m.radermacher@tudelft.nl
    16-01-2014
  20. handler

    Remove features with nan values

    Priority HIGH
    m.radermacher@tudelft.nl
    22-01-2014
  21. handler

    Create function for 1-by-1 dropping of worst features

    Priority HIGH
    m.radermacher@tudelft.nl
    02-04-2014
  22. handler

    Implement class probabilities instead of single class prediction

    Priority HIGH
    m.radermacher@tudelft.nl
    02-04-2014
100%

Dissemination

  1. handler

    Create presention on image classification

    Priority MEDIUM
    m.radermacher@tudelft.nl
    N/A
  2. handler

    Prepare training set for image classification meeting

    Priority MEDIUM
    m.radermacher@tudelft.nl
    N/A
  3. handler

    Collect relevant literature

    Priority MEDIUM
    m.radermacher@tudelft.nl
    N/A
  4. handler

    Visualize CRF created using pystruct toolbox, including pairwise potentials

    Priority MEDIUM
    m.radermacher@tudelft.nl
    N/A

Relevant documents

  File Modified
PDF File ICS2014_Hoonhout_v2.pdf ICS 2014 paper on application of the classification technique 12-12-2013 by Bas Hoonhout
Microsoft Powerpoint Presentation SuperpixelClassification.pptx Presentation on the classification methodology 12-12-2013 by Bas Hoonhout

Relevant literature

  1. Achanta, R., Shaji, A., Smith, K., Lucchi, A., Fua, P. and Susstrunk, S., 2012. SLIC superpixels compared to state-of-the-art superpixel methods. IEEE Transactions on Pattern Analysis and Machine Intelligence.
  2. Gould, S., Rodgers, J., Cohen, D., Elidan, G. and Koller, D., 2008. Multi-class segmentation with relative location prior. International Journal of Computer Vision.
  3. Koller, D. and Friedman, N., 2009. Probabilistic Graphical Models. The MIT Press.
  4. Mueller, A. and Behnke, S., 2013. Learning a loopy model for semantic segmentation exactly. Technical Report.
  5. Nowozin, S. and Lampert, C.H., 2010. Structured learning and prediction in computer vision. Foundations and Trends in Computer Graphics and Vision.
  • No labels

Disclaimer