The goals of this project are to develop a neural network language that
- is easy to use and understand
- can be compiled to very efficient code
- allows derivatives of any order
- makes it easy for the end user to keep track of complex tensor expressions
In particular, the derivatives of matrix expressions in deep neural networks result in complex tensor expressions. These can be difficult for the average programmer to understand and manipulate and also can be difficult to fully optimize. As a consequence many existing neural network libraries hide complex sequences of operations in larger functions, e.g., combining a backward gradient computation with some aspect of the optimization algorithm. This makes it more difficult for end users to quickly and easily test out novel optimization strategies.
The first version of the neural network language formed the basis of CNTK, the Computational Network Tool Kit. This language only allowed for gradient computation, and certain aspects of the gradient computation were unnecessarily entwined with some aspects of the optimization. The new language is being designed to expose every aspect of the computation to the end user so nothing will be hidden in black box code.
Eye tracker dependencies have been removed and replaced with function stubs to make it easy to modify to work with your eye tracker. For good results you will need a high frame rate, low latency eye tracker, and a high frame rate monitor. We used a 300Hz eye tracker with less than 10ms latency and a 120Hz monitor with a special low latency gaming mode. Lower performance devices could (probably will) cause visible foveation artifacts.
Download: Foveated 3D software
Paper: Foveated 3D Graphics, SIGGRAPH Asia 2012
All lenses have optical aberrations which reduce image sharpness. These aberrations can be reduced by deconvolving an image using the lens point spread function (PSF). However, fully measuring a PSF is laborious. Alternatively, one can simulate the PSF if the lens model is known. However, due to manufacturing tolerances lenses differ subtly from their models, so often a simulated PSF is a poor match to measured data. We present an algorithm that uses a PSF measurement at a single depth to calibrate the nominal lens model to the measured PSF. The fitted model can then be used to compute the PSF for any desired setting of lens parameters for any scene depth, without additional measurements or calibration. The fitted model gives deconvolution results comparable to measurement but is much more compact and require hundreds of times fewer calibration images.
Paper: Image Enhancement using Calibrated Lens Simulations, IEEE ECCV 2012.
Derivatives arise frequently in graphics applications. To meet this need we have added symbolic differentiation as a built in language feature in the HLSL shading language, available in the June 2010 DirectX SDK release (a much improved version is in the Windows 8 SDK preview). The symbolic derivative is computed at compile time so it is available in all types of shaders (geometry, pixel, vertex, etc.). The algorithm for computing the symbolic derivative is simple and has reasonable compilation and run time overhead. The latest preview implementation is described in this paper, Symbolic Differentiation On The GPU, which also has several detailed example HLSL programs. This is joint work with Mark Finch and John Rapp. Tutorials on this work were presented at GDC 2011 by Kev Gee (“Direct 3D 11: Symbolic Derivatives and HLSL”) and at GameFest 2011 by Mark Finch: Symbolic Differentiation in HLSL.
Download: Examples of HLSL Symbolic-Differentiation Programs. The source code includes an interactive editor for generating procedural geometric models, and several DirectX examples showing how to use the symbolic differentiation feature to generate the geometry and texture detail at runtime.
Video: Procedural texture for HLSL symbolic differentiation. These show procedural surfaces and textures defined using symbolic differentiation.
In a modern data center the cost of power, for computers and air conditioning, can be more than the cost of the computer hardware. Modern computers have a variety of power states with different power vs. response time tradeoffs: off, sleep, hibernate, etc. With thousands of computers in a typical data center it is challenging to determine what power state each computer should be in at any moment in order to minimize power while maximizing responsiveness. I developed an algorithm which breaks the problem into two pieces: predicting future demand and determining power state transitions to minimize power while meeting demand in the best way. Any prediction scheme can be used but in our first implementation we used simple linear prediction. The optimal power state transitions are computed with linear programming. In the general case this is an integer, rather than a linear, programming problem, but a novel representation of the system allows linear programming to be used, while guaranteeing integer results. This makes the algorithm very fast even for data centers with tens of thousands of computers. Our evaluation on three very different data center workloads shows that the energy savings are close to optimal, saving 96%-99.5% of the maximum possible.
Paper: Managing Cost, Performance, and Reliability Tradeoffs for Energy-Aware Server Provisioning, IEEE INFOCOM 2011
The download includes the D* executable and full source code for the following: basic D* programming examples, Lagrangian dynamics, and a basic interactive geometric modeling tool. This is unsupported code but if you find bugs or have feature requests send email to email@example.com.
Video: Symbolic Lagrangian Mechanics
There is a book that describes the dynamics algorithms, as well as simple procedural modeling techniques. You will probably want the book if you are downloading the code, since it has more detailed documentation of the software than is available in the download. You can get the book here: Symbolic Dynamics and Geometry: Using D* in Graphics and Game Programming
D* generates symbolic derivatives which can be thousands of times faster than those generated by Mathematica or automatic differentiation. Functions with densely interconnected expression graphs, which arise in applications such as dynamics, spacetime optimization (also known as the optimal control problem), and PRT, can be difficult to efficiently differentiate using existing symbolic or automatic differentiation techniques. The D* algorithm computes efficient symbolic derivatives for these functions by symbolically executing the expression graph at compile time to eliminate common subexpressions and by exploiting the special nature of the graph that represents the derivative of a function. This graph has a sum of products form; the new algorithm computes a factorization of this derivative graph along with an efficient grouping of product terms into subexpressions. For the test suite problems D* generates symbolic derivatives which are up to 4.6ÃƒÆ’Ã¢â‚¬â€10^3 times faster than those computed by the symbolic math program Mathematica and up to 2.2ÃƒÆ’Ã¢â‚¬â€10^5 times faster than the non-symbolic automatic differentiation program CppAD. In some cases, the D* derivatives rival the best manually derived solutions.
Paper: Symbolic Differentiation. This paper is a revised and expanded version of the paper presented at SIGGRAPH 2007. I will be updating it as I find the time. (most recently updated 11/1/2007)
Generative CSG models, while having the desirable characteristics of compactness and resolution independence, have never been used for real time rendering because no algorithms existed which could both maintain their compact representation and render them efficiently at run time. The key difficulty in doing this was finding a compact, exact representation of the implicit curve of intersection that arises from CSG operations. The primary contribution of this paper is a new algorithm for finding a piecewise parametric representation for this intersection curve. The parametric representation is compact and exact to the limits of precision of floating point arithmetic. Arbitrary points on the intersection curve can be efficiently evaluated at run time which allows triangulation density to be adapted dynamically. Using this representation, we have made complex procedural objects that have a memory footprint of just 7-11 KBytes, which render at approximately 20 million triangles/ sec. on an NVidia 6800 GPU.
Paper: Exact Procedural CSG Modeling for Real Time Graphics
Video: Procedural Geometry for Real Time Graphics
Paper: Making Faces, SIGGRAPH 98 with Cindy Grimm, Henrique Malvar, Daniel Wood, and Frederic Pighin
Video: Making Faces. An actress reads from a script designed to provide maximal phonetic coverage (we were planning to do automatic lip sync as a follow-on research project but never got around to it). Hence the funky monologue.
Real-time, Photo-realistic, Physically Based Fine-scale Human Skin Structure Rendering
A. Haro, B. Guenter, and I. Essa, Proceedings 12th Eurographics Workshop on Rendering, London, England, June 2001
Modeling and Rendering for Realistic Facial Animation
Stephen R. Marschner, Brian Guenter, and Sashi Raghupathy, 11th Eurographics Rendering Workshop, 2000
Lossless Compression of Computer-Generated Animation Frames (published in Transactions on Graphics, October 97) with Hee Cheol Yun, and Russel M. Mersereau
ACM Transactions on Graphics, v.16, no. 4, October 1997, pp. 359-396
Paper: Quadrature Prefiltering for High Quality Antialiasing, Transactions on Graphics, October 96 with Jack Tumblin
This is a movie made by my graduate students in 1993 when I was an assistant professor at Georgia Tech. I hadn’t looked at this video for more than 15 years. It holds up remarkably well and I’m still impressed by the incredible job they did. My contribution was negligible even though I’m listed as a producer on the credits — the students deserve all the credit. Unfortunately, we made some sort of scaling error when we rendered the frames so the entire film came out much darker than we intended (the producer is supposed to make sure that things like this don’t happen). When it was shown at the electronic theater the projector was also very dim so it was almost impossible to see what was going on. A few bat-like people who could see in the dark complimented us on the film but my students were still terribly disappointed that after all their hard work no one had really seen their film. Now you can see it in all its original glory, and by using the brightness and contrast controls in Windows media player you can largely eliminate the darkness problem.
Video: Deus ex Machina