Transmitting compactly represented geometry of a dynamic
scene from a sender can enable a multitude of 3D imaging
functionalities at a receiver, such as synthesis of virtual images
from freely chosen viewpoints via depth-image-based
rendering (DIBR).While depth maps can now be readily captured
using inexpensive depth sensors, they are often corrupted
by non-negligible acquisition noise. In this paper, we
derive 3D surfaces of a dynamic scene from noise-corrupted
depth maps in a rate-distortion (RD) optimal manner. Specifically,
unlike previous work that finds the most likely (e.g.,
maximum likelihood) 3D surface from noisy observations
regardless of representation size, we judiciously search for
the best fitting (i.e., minimum distortion) 3D surface subject
to a bitrate constraint. Our RD-optimal solution reduces
to the maximum likelihood solution as the rate constraint is
loosened. Using the MVC codec for compression of multiview
depth video and MPEG free viewpoint test sequences
as input, experimental results show that RD-optimized 3D reconstructions
computed by our algorithm outperform unprocessed
depth maps by up to 2.42dB in PSNR of synthesized
virtual views at the decoder for the same bitrate.