Results:
For the cartoon example, we have 2 key views
and we produce 7 intermediate novel views to enable smooth view-point
traversal. These 7 intermediate novel views are obtained by setting the ratio
respectively. Note that since this is a synthetic sequence, there are no cues related to depth and hence no 3-D perception. It serves as a toy example to demonstrate the working of the algorithm. The results for a real-world sequence, called Ballet, are presented next.
For the Ballet sequence, we have 8 key views
and we produce 7 intermediate novel views, one novel view between every 2
successive key views. Spatial resolution 512x384 entails prohibitive computational
complexity for the block-matching algorithm and hence we run the algorithm on
spatial resolution 256x192. We also generate novel views at resolution 512x384
by scaling the disparity vectors obtained for the smaller spatial resolution.
Ballet (256x192, 10 fps) video
Ballet (512x384, 10 fps) video
Future Directions:
We see some spurious block-matches in some
novel views. This problem can be avoided by enforcing some continuity constraints
on the neighboring disparity vector values. For example, if all neighboring
blocks have very similar disparity vector values and the current block has a
very different one, then this vector is perhaps spurious.
-
If
we can carry out some kind of object segmentation, then it would help enforce
continuity even better.
- Or
if we had depth values for every pixel in every key view, for example obtained
from a range scanner, then this would help pin-point spurious disparity vector
values quite well. This option, of course, requires additional side-information
about depth values.
It would help if we come up with a way to adapt
block-sizes. Smaller block-sizes would be preferred near object boundaries.
Larger block-sizes should be used otherwise to avoid spurious matches.
Conclusions:
An IBR technique for rendering a novel view was
presented. This technique obviates the need for complex geometry. From the
viewer's perspective, novel views are critical for smooth traversal of
view-point. The proposed algorithm produces some block artifacts in the novel
views but since these views are displayed interleaved with the key views, this
does not spoil viewing experience too much. In general, the more the number of
novel views the better it is for smooth view-point traversal, provided the
quality of the novel views is not too bad. If we have many novel views, we can
always increase the display frame-rate to get to the final changed view-point
in good time. This amount of time is flexible, since we are rendering an unnatural
phenomenon anyway, viz. the dynamic scene never freezes in reality for us to
change our view-point.
Appendix:
MATLAB source code:
create_avi_file512_compare.m (Creates Ballet
results AVI file in resolution 512x384.)
create_avi_file_compare.m (Creates Ballet
results AVI file in resolution 256x192.)
create_cartoon_avi_file_compare.m (Creates
Cartoon results AVI file.)
create_dat_file_compare.m (Creates Cartoon
results YUV file.)
create_yuv_file_compare.m (Creates Ballet
results YUV file in resolution 256x192.)
get_novel_view.m (Estimates disparity vectors,
and produces novel view spaced at given location on the line joining two camera
centers.)
get_novel_view_noest.m (Produces novel view by
using already estimated disparity vectors.)
get_pred_block_qpel.m (Extracts a block of
given size, from a location pointed to by given disparity vectors, relative to
a given block location from the original grid. The disparity vectors can be
quarter pixel accurate. Any block involving sub-pixel locations is obtained by
using bilinear interpolation.)
pad_images.m (Pads images by desired number of
pixels.)
read_ith_GOP_keyframe.m (Reads the required
frame from input file.)
script_novel_views.m (Script file to generate
required number of novel views.)
script_optimized_views.m (Script file to
generate required number of novel views from already estimated disparity
vectors.)
unpad_images.m (Remove the padded pixels from
the images to yield original spatial resolution.)