The Approach

Our approach combines the processes of smoothing, segmentation, 3D motion and structure estimation. New constraints have been developed which are defined directly on the image derivatives, leading to a geometrical and statistical estimation problem. The main idea is based on the interaction between 3D motion and shape, which allows us to estimate the 3D motion while at the same time segmenting the scene. If we use a wrong 3D motion estimate to compute depth, then we obtain a distorted version of the depth function. The distortion, however, is such that the worse the motion estimate, the more likely we are to obtain depth estimates that vary locally more than the correct ones. Local variability of depth is due either to the existence of a discontinuity or to a wrong 3D motion estimate; by exploiting the statistics of the raw image measurements (derivatives) these two cases can be differentiated. Clearly, at the end of the process a good estimate of correspondence can also be made.

Since at the beginning of the process correspondence or flow is not available, we cannot utilize the epipolar constraint that has been traditionally used. Instead, we utilize the positive depth constraint and geometric constraints arising from understanding the distortion function, as explained before. More specifically, if Z is the actual scene model and \hat{Z}> what is computed on the
basis of a wrong 3D motion, then <IMG SRC=, with D a distortion function depending on the errors in the 3D transformation and image measurements. Understanding this function provides the insight that human visual space is a non-Euclidean space; further, it explains a number of illusions and predicts others (see the section on biology). At the same time, this understanding gives rise to algorithms for 3D motion estimation, motion segmentation and scene reconstruction from video sequences, producing results not obtainable by correspondence-based approaches. Theoretical results also demonstrate that existing approaches are special cases of our approach; that is, our approach is provably better than the state-of-the-art, correspondence-based schemes.

Return to the front page

Revised 1999/04/15
Send questions about these Web pages to