Microsoft Research Blog

On infinitely wide neural networks that exhibit feature learning 

July 22, 2021 | Edward Hu and Greg Yang
In the pursuit of learning about fundamentals of the natural world, scientists have had success with coming at discoveries from both a bottom-up and top-down approach. Neuroscience is a great example of the former. Spanish anatomist Santiago Ramón y Cajal discovered the neuron in the…
While the NNGP and NTK limits essentially only considers the neural network initialization, the feature learning limit incorporates the entire training trajectory. A Neural network is represented by a stack of vertical shapes: an inverted trapezoid, a square, and a triangle. On the left side of the shape, A blue arrow moves upward and represents the first forward pass. The NNGP limit can be thought of as the limit of this first forward pass. On the right side of the shape, a green arrow moves downward and represents the first backward pass. The NTK limit can be thought of as the limit for this first backward pass. In contrast, the feature learning limit takes into account the many cycles of forward and backward passes that take place during the entire training process. These cycles are represented by many repetitions of blue upward arrow and green downward arrows to the right of the neural network. An orange box encloses all of these cycles. On top of the box is the annotation “SGD Training Progress” with an arrow to the right. An arrow comes out from the bottom of the box pointing to a textbox that says “Feature Learning Limit, This Work.”

Recent Posts

  1. While the NNGP and NTK limits essentially only considers the neural network initialization, the feature learning limit incorporates the entire training trajectory. A Neural network is represented by a stack of vertical shapes: an inverted trapezoid, a square, and a triangle. On the left side of the shape, A blue arrow moves upward and represents the first forward pass. The NNGP limit can be thought of as the limit of this first forward pass. On the right side of the shape, a green arrow moves downward and represents the first backward pass. The NTK limit can be thought of as the limit for this first backward pass. In contrast, the feature learning limit takes into account the many cycles of forward and backward passes that take place during the entire training process. These cycles are represented by many repetitions of blue upward arrow and green downward arrows to the right of the neural network. An orange box encloses all of these cycles. On top of the box is the annotation “SGD Training Progress” with an arrow to the right. An arrow comes out from the bottom of the box pointing to a textbox that says “Feature Learning Limit, This Work.”

    On infinitely wide neural networks that exhibit feature learning 

    July 22, 2021 | Edward Hu and Greg Yang

    In the pursuit of learning about fundamentals of the natural world, scientists have had success with coming at discoveries from both a bottom-up and top-down approach. Neuroscience is a great example of the former. Spanish anatomist Santiago Ramón y Cajal discovered the neuron in the…

  2. Micah Stampley, Lisa Nakamura posing for a photo

    Lecture series aims to help spur dialogue around race and technology 

    July 21, 2021

    In November, NYU media professor Charlton McIlwain (opens in new tab) joined fellow scholars Safiya Noble, Ruha Benjamin, and André Brock for a virtual discussion on anti-Blackness and technology hosted by the University of California Santa Barbara. The conversation was an engaging one, and McIlwain…

  3. Traditional cellular network infrastructure compared to cellular network infrastructure in the Microsoft cloud Two graphics. The first depicts traditional cellular network infrastructure, beginning with cell towers receiving data and transferring it to physical buildings--local hubs, then central exchanges and finally data centers. The second depicts cloudified cellular network infrastructure, with cell towers transmitting data to telco edges and Microsoft edges. This is also labelled "RAN in the cloud". The data then flows to the Microsoft cloud, including core network and OSS/BSS as a service.

    Project Arno: How Microsoft Research created the technology and industry momentum for Azure to empower telecom operators in the cloud 

    July 19, 2021 | Yongguang Zhang and Bozidar Radunovic

    Editor’s note: In recent years, telecommunications operators have faced a growing challenge to meet surging global demand for immersive online services and collaboration tools. Upgrading their proprietary networks to prepare for 5G and beyond would require major capital expenditures, even as competition was driving down…

  9. Bosque programming language code sample

    Creating the Future of Software Development 

    May 26, 2021

    “If you think about the world that we’re in now, software plays a key role in our lives – in everything we do. And at the core of solving big problems and fundamental challenges is the ability to use software to attack the problem and…

  11. A close-up of a man wearing casual clothing, he has his smartphone in his hand and he is using an assistive mobile app for people with vision disabilities to assist him.

    CHI 2021: Redefining accessibility to build more inclusive technologies 

    May 21, 2021

    Accessibility and inclusion represent a growing space in the technology landscape, and how research and development are being used to empower people across abilities is expanding in exciting ways. Instead of treating disabilities as conditions in need of solutions—as has been the case over the…

Explore More

  • Events & conferences

    Events & conferences 

    Meet our community of researchers, learn about exciting research topics, and grow your network

  • Podcasts

    Podcasts 

    Ongoing conversations at the cutting edge of research

  • Microsoft Research Forum

    Microsoft Research Forum 

    Join us for a continuous exchange of ideas about research in the era of general AI