Disparity
Compensated Interpolation:
The objects in
the two key views appear shifted with respect to each other. This shift is
termed as disparity. Objects closer to the camera plane appear to be shifted
more than the objects farther away. Also different objects can move in
different directions (for example, in the Ballet sequence, the ballerina and
the trainer shift in different directions in some of the views). The idea
behind disparity compensated interpolation (DCI) is to generate a novel view by
interpolating the disparity of every object for any view-point which lies along
the line joining the two camera centers for the two key views. The proposed
algorithm does not actually need to segment the scene into objects, but works
on a block-by-block basis. It is thus oblivious to scene complexity. We assume,
however, that the amount of disparity between the two key views, 
and
, is not too large and hence, occlusion related problems are
very limited.
Figure 2: The novel view is divided into blocks. Every block is filled with content obtained from a block-matching algorithm.
As shown in
Figure 2, the novel view is divided into blocks. The goal is to find a
disparity vector 
for each
block,
, and then fill it up using the corresponding blocks from andas follows:
.
The disparity
vector is obtained by minimizing the sum of squared differences
(SSD) between the candidate blocks from andas follows:
When the ratio 
is 0.5, we generate the novel view corresponding to the
view-point which is exactly half-way through between the two camera centers. If
it is less than 0.5 then our view-point is closer to the first camera and if it
is more than 0.5 then our view-point is closer to the second camera. The search
for the vector can be
limited to a reasonable range by guessing the maximum shift.
(For a more graphic explanation of the algorithm, please refer to the ppt.)
Note that this algorithm is similar to motion-compensated interpolation [1].
Design parameters
in block-matching:
- Block-size: A small
block-size leads to some spurious matches and hence fails to interpolate
disparity of objects. A large block-size, on the other hand, fails to
adapt to the object boundaries and hence associates the same disparity
with the object and some part of the background. For the Ballet sequence,
in 256x192 resolution, we found 16x16 to be a
good choice. This block-size produces some spurious matches but the novel
views are acceptable.
- Image resolution: Since large spatial
resolution might entail prohibitive computational complexity for the
block-matching, we might run the algorithm on a smaller spatial
resolution. The disparity vectors thus obtained can be scaled up before
applying to larger blocks in the large spatial resolution. There is some
inaccuracy introduced due to working in the smaller spatial resolution.
- Color channels: We decided to use the
luminance component in full resolution and the two chrominance components
in sub-sampled resolution to calculate the SSD for a single block-match as
shown above. Different parts of the background might have the same
chrominance, but different luminance due to position of lights, e.g.
different areas of the wall.
References:
[1] O. Ojo and G. de Haan, "Robust motion-compensated video upconversion," IEEE Transactions on Consumer Electronics, vol. 43, no. 4, pp. 1045-1056, November 1997.