Models as Code: Differentiable Programming with Zygote

  • Viral Shah & Elliot Saba | Julia Computing

Scientific computing is increasingly incorporating the advancements in machine learning and the ability to work with large amounts of data. At the same time, machine learning models are becoming increasingly sophisticated and exhibit many features often seen in scientific computing, stressing the capabilities of machine learning frameworks. Just as the disciplines of scientific computing and machine learning have shared common underlying infrastructure in the form of numerical linear algebra, we now have the opportunity to further share new computational infrastructure, and thus ideas, in the form of Differentiable Programming. We demo Zygote, a Differentiable Programming system that is able to take gradients of general program structures. We implement this system in the Julia programming language. Our system supports almost all language constructs (control flow, recursion, mutation, etc.) and compiles high-performance code without requiring any user intervention or refactoring to stage computations. This enables an expressive programming model for deep learning, but more importantly, it enables us to incorporate a large ecosystem of libraries in our models in a straightforward way. We showcase our approach to automatic differentiation, including its support for advanced techniques such as mixed-mode, complex and checkpointed differentiation, and present several examples of differentiating programs.

[SLIDES]

Speaker Details

Viral Shah, CEO
Co-Creator of the Julia Programming Language and Co-Founder of Julia Computing

Elliot Saba, Sr. Research Engineer
Elliot Saba is a senior research engineer at Julia Computing, where he design next-generation tools for the Julia programming language. He received his Ph.D. from the University of Washington in Electrical Engineering in 2018, specializing in Digital Signal Processing and Machine Learning. Elliot received the Julia community prize at JuliaCon 2018, recognizing his open source contributions to the Julia project. When he is not building fundamental infrastructure for the Julia language, he enjoys dreaming up ever more complex strategies to automate watering his houseplants.

Series: Microsoft Research Talks