THE FUTURE IS NOT FRAMED

Date

May 13, 2005

Speaker

Benjamin Watson and David Luebke

Affiliation

University of Virginia & Northwestern University

Overview

The ultimate display will not show images. To drive the display of the future, we must abandon our traditional concepts of pixels, and of images as grids of coherent pixels, and of imagery as a sequence of images.

So what is this ultimate display? One thing is obvious: the display of the future will have incredibly high resolution. A typical monitor today has 100 dpi—-far below a satisfactory printer. Several technologies offer the prospect of much higher resolutions; even today you can buy a 300 dpi e-book. Accounting for hyperacuity, one can make the argument that a “perfect” desktop-sized monitor would require about 6000 dpi—-call it 11 gigapixels. Even if we don’t seek a perfect monitor, we do want large displays. The very walls of our offices should be active display surfaces, addressable to a resolution comparable to or better than current monitors.

It’s not just spatial resolution, either. We need higher temporal resolution: hardcore gamers already use single buffering to reduce delays. The human factors literature justifies this: even 15 ms of delay can harm task performance. Exotic technologies (holographic, autostereoscopic…) just increase the spatial, temporal, and directional resolution required.

Suppose we settle for 1 gigapixel displays that can refresh at 240 Hz—-roughly 4000x typical display bandwidths today. Recomputing and refreshing every pixel every time is a Bad Idea, for power and thermal reasons if nothing else.

We will present an alternative: discard the frame. Send the display streams of samples (location+color) instead of sequences of images. Build hardware into the display to buffer and reconstruct images from these samples. Exploit temporal coherence: send samples less often where imagery is changing slowly. Exploit spatial coherence: send fewer samples where imagery is low-frequency. Without the rigid sampling patterns of framed renderers,sampling and reconstruction can adapt with very fine granularity to spatio-temporal image change. Sampling uses closed-loop feedback to guide sampling toward edges or motion in the image. A temporally deep buffer stores all the samples created over a short time interval for use in reconstruction. Reconstruction responds both to sampling density and spatio-temporal color gradients. We argue that this will reduce bandwidth requirements by 1-2 orders of magnitude, and show results from our preliminary experiments.

NOTE: this talk is an update to work we presented at MSR last year (July 20, 2004). We will briefly reiterate the basic approach, describe our improved algorithm, implementation, and results, and discuss some interesting implications for displays, cameras, and rendering hardware. This work is currently under review for the 2005 Eurographics Symposium on Rendering.

Speakers

Benjamin Watson and David Luebke

Benjamin Watson is an Assistant Professor at the Computer Science department of Northwestern University. There he leads the Realism Lab, which strives to bring realistic complexity to interactive display. His research interests address understanding, generating, displaying, and interacting with realism, and therefore include topics such as measuring visual similarity, procedural modeling of human artifacts, temporally adaptive rendering, simplification, visualization and 3D interaction. His work has been applied to digital entertainment and training, marketing and financial intelligence, medical therapy and assessment, and education. Watson earned his Ph.D. at Georgia Tech’s GVU Center, co-chaired the Graphics Interface 2001 conference, and chaired the IEEE Virtual Reality 2004 conference. He is coauthor with David Luebke of “Level of Detail for 3D Graphics”, published by Morgan Kaufman.

David Luebke is an Assistant Professor in the Department of Computer Science at the University of Virginia. He earned his Ph.D. in Computer Science at the University of North Carolina under Frederick P. Brooks, Jr., and earned his Bachelors degree in Chemistry at the Colorado College. Professor Luebke’s principal research interest is interactive computer graphics, particularly the problem of acquiring and rendering very complex real-world scenes at interactive rates. Specific projects include polygonal level of detail (LOD), temperature-aware graphics architecture, scientific computation on graphics hardware, advanced reflectance and illumination models for real-time rendering, and image-based acquisition of real-world environments. Another project was the Virtual Monticello museum exhibit, which ran for over 3 months and helped attract over 110,000 visitors as a centerpiece of the major exhibition “Jefferson’s America and Napoleon’s France” at the New Orleans Museum of Art. Luebke is also lead author of the book “Level of Detail for 3D Graphics”.