Ching L. Teo, Cornelia Fermüller, Yiannis Aloimonos
The problem of border ownership assignment is to determine whether regions adjacent to the boundaries belong to the foreground (FG) or background (BG). The neurological processes of border ownership assignment are well documented  and it plays a central role in visual Gestalt, specifically for the task of figure-ground organization .
We present a fast approach for border ownership assignment in real images. By combining local patch-based features sensitive to border ownership with a Structured Random Forest  classifier, we are able to predict, given a new test image, both boundaries and ownership in one single efficient inference step: ~0.1s for a 320x240 image. Our predictions are also state-of-the-art in terms of accuracy compared to [4,5], that use a slower CRF inference step over boundaries that were separately detected.
Extremal edges (EE)  are strong local border ownership cues that are characterized by local changes in grayscale intensities along the boundaries. By analysing intensity patterns within aligned patches using Principal Component Analysis (PCA) (I), we determined that PC2 (boxed) exhibits the distinctive signature of EE. In practice we use the projections of the top 5 PCs as spectral features (II).
Border ownership is also captured by more global Gestalt (grouping) cues. By extending the mid-level image "torque" operator  to other Gestalt patterns besides closure (I), which were observed in the macaques , we capture in these Gestalt-like features longer range responses (II).
SRF for border ownership assignment
In addition to the spectral and Gestalt-like features, we extract Histograms of Gradients (HoG) to capture local shape: convexity and concavity (A). We pair these patch-based features with their an orientation coded annotation (we use 8 discrete orientations) to train a set of 16 decision decision trees in the SRF. The training learns thresholds within the split nodes that associate features with their ownership structures at the leaf nodes (B). During inference, given a test feature, we average the responses from all trees in the SRF to get the final boundary + ownership predictions (C).
8 example results (counter-clockwise from top left): Input RGB, groundtruth ownership (red: FG, blue:BG), SRF predictions (blue: boundaries, red: FG, yellow: BG).
"Real-time" boundary (red) and ownership (FG: green, BG: blue) predictions on cluttered indoor scenes. Note that the predictions are from a SRF trained over the NYU-Depth V2 dataset.
An early version of this work was accepted at the 15th Annual meeting of the Vision Science Society (VSS), St. Pete Beach, Florida, 2015: Abstract, poster
 H. Zhou, H. S. Friedman, and R. Von Der Heydt. Coding of border ownership in monkey visual cortex. The Journal of Neuroscience, 20(17):6594–6611, 2000.
 J. Wagemans, J. H. Elder, M. Kubovy, S. E. Palmer, M. A. Peterson, M. Singh, and R. von der Heydt. A century of gestalt psychology in visual perception:I. perceptual grouping and figure–ground organization. Psychological bulletin, 138(6):1172, 2012.
 P. Kontschieder, S. R. Bulo, H. Bischof, and M. Pelillo. Structured class-labels in random forests for semantic image labelling. In Proc. IEEE International Conference on Computer Vision, pages 2190–2197, 2011.
 X. Ren, C. C. Fowlkes, and J. Malik. Figure/ground assignment in natural images. In Proc. European Conf. on Computer Vision, pages 614–627. 2006.
 I. Leichter and M. Lindenbaum. Boundary ownership by lifting to 2.1 d. In Proc. IEEE International Conference on Computer Vision, pages 9–16, 2009.
 T. Ghose and S. E. Palmer. Extremal edges versus other principles of figure-ground organization. Journal of Vision, 10(8):3, 2010.
 M. Nishigaki, C. Fermüller, and D. DeMenthon. The image torque operator: A new tool for mid-level vision. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, pages 502–509, 2012
 J. L. Gallant, C. E. Connor, S. Rakshit, J. W. Lewis, and D. C. Van Essen. Neural responses to polar, hyperbolic, and
cartesian gratings in area v4 of the macaque monkey. Journal of Neurophysiology, 76(4):2718–2739, 1996
This work was funded by the support of the European Union under the Cognitive Systems program (project POETICON++), the National Science Foundation under INSPIRE grant SMA 1248056, and by DARPA through U.S. Army grant W911NF-14-1-0384.