A Semi-Automatic Method for Resolving Occlusions in Augmented Reality

Constructing the 3D occluding boundary

Lets ignore the uncertainty on the viewpoints for the moment. Let C1 and C2 be the 2D boundaries of an occluding object outlined in two consecutive key views. We build the 3D occluding boundary using stereo triangulation : Let m1 be a point on C1. Its corresponding point m2 in the other view is the intersection of the epipolar line with C2. As usual, the order constraint is used to solve the ambiguity of the correspondence problem.

Taking into account the uncertainty on the viewpoints while constructing the 3D occluding boundary

Since we have curve correspondences, the point m2 actually depends on the viewpoints in the two key views. To take into account the uncertainty on these viewpoints, we resort to an exhaustive approach, and consider the extremal viewpoints, that are the vertices of the 6-dimensionnal indifference ellipsoid. Let  (resp ) the extremal viewpoints in the two key views. Let m1 be a point on C1. Given an extremal viewpoint p1, we can compute the 12 possible reconstructions of m1 with the 12 extremal views in key view 2:


 




Using the 12 extremal viewpoints in key view 1, we then obtain 12^2 reconstructions of m1. The convex hull of these 144 points is a good approximation of the 3D reconstruction error of m1.

Taking into account the uncertainty on the viewpoint while reprojecting the 3D occluding boundary

To estimate the 2D uncertainty on the projected boundary C, we have to consider the 3D reconstruction error and the uncertainty on the consider viewpoint. We again resort to an exhaustive method : for each point m of C1, the 12^2 extremal reconstructions are projected onto the current frame using the 12 extremal viewpoints of this frame. We define the uncertainty on the predicted occluding boundary to m as the convex hull of these 12^3 image points. This area is noted  in the following.


Real case : the point m1, the associated epipolar lines and the reprojections.