{"id":1146334,"date":"2025-08-12T13:09:21","date_gmt":"2025-08-12T20:09:21","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?p=1146334"},"modified":"2025-09-09T10:06:33","modified_gmt":"2025-09-09T17:06:33","slug":"dion-the-distributed-orthonormal-update-revolution-is-here","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/dion-the-distributed-orthonormal-update-revolution-is-here\/","title":{"rendered":"Dion: the distributed orthonormal update revolution is here"},"content":{"rendered":"\n
\"Three<\/figure>\n\n\n\n

Training AI models requires choosing an optimizer and for nearly a decade, Adam( (opens in new tab)<\/span><\/a>–W) (opens in new tab)<\/span><\/a> has been the optimizer of choice. Given that durability and success, it was fair to doubt that any further improvement was possible. And yet, last December, a new optimizer called Muon (opens in new tab)<\/span><\/a> showed serious promise by powering a nanoGPT speedrun (opens in new tab)<\/span><\/a>. This proved out, with multiple AI labs (e.g., Kimi-AI (opens in new tab)<\/span><\/a> and Essential-AI (opens in new tab)<\/span><\/a>) reporting 2x scale improvements and the release of the 1T parameter Kimi K2 (opens in new tab)<\/span><\/a> model. Restated: you can train a model to similar performance with half as many GPUs.<\/p>\n\n\n\n

There\u2019s one fly in the ointment: Muon requires large matrix multiplications in the optimizer, which requires heavy communication in large models at the scale where FSDP (opens in new tab)<\/span><\/a> and TP (opens in new tab)<\/span><\/a> parallelization becomes desirable. Going back to the inspiration for Muon, (opens in new tab)<\/span><\/a> the key idea is an orthonormal update, which sparked the search for more scalable alternative linear algebras realizing the same goal. That\u2019s exactly what Dion<\/a> is. We have open-sourced this new optimizer to enable anyone to train large models more efficiently at scale.  <\/p>\n\n\n\n

What\u2019s an orthonormal update?<\/h2>\n\n\n\n
\"Illustration
Figure1. Illustration of matrix parameters<\/figcaption><\/figure>\n\n\n\n

At the core of Transformers, a set of input activations is multiplied by a learned weight matrix to produce a new set of output activations. When the weight matrix is updated during training, the resulting change in the output activations generally depends on the direction of the input activations. As a result, the learning rate must be chosen conservatively to accommodate the input direction that induces the largest change. Orthonormalized updates alter this behavior by (approximately) making the change in output activations invariant to the direction of the input. This is achieved by enforcing orthonormality (opens in new tab)<\/span><\/a> on the update matrix, thereby equalizing its effect across all input directions.<\/p>\n\n\n\n

What is Dion?<\/h2>\n\n\n\n

While Muon has shown strong empirical results, scaling it to very large models poses challenges. As reported by Essential AI (opens in new tab)<\/span><\/a>, applying Muon to large architectures like LLaMA-3 becomes compute-bound<\/em>\u2014and potentially communication-bound<\/em>\u2014due to the cost of the Newton\u2013Schulz orthonormalization steps (opens in new tab)<\/span><\/a>.<\/p>\n\n\n\n

\"Pseudocode
Figure 2. Pseudocode of the centralized version of Dion<\/figcaption><\/figure>\n\n\n\n

This is where Dion<\/strong> enters. At a high level, Dion introduces a new axis for scalability: the rank<\/strong>. Specifically, for a given rank r, Dion orthonormalizes only the top r of the singular vector space, reducing communication and compute overhead while preserving performance. Empirically, we observe that the necessary rank for good performance grows much more slowly than the number of parameters in larger models.<\/p>\n\n\n\n

\n\t
\n\t\t
\n\t\t\t\t\t\tTool<\/span>\n\t\t\tDion optimizer<\/span> <\/span><\/a>\t\t\t\t\t<\/div>\n\t<\/article>\n<\/div>\n\n\n\n

Dion implements orthonormalization using amortized power iteration<\/em> (opens in new tab)<\/span><\/a>. <\/em>Power iteration typically pulls out the largest singular value by repeated matrix multiplication. By amortizing this process over optimization steps\u2014applied to the slowly-evolving momentum matrix\u2014we reduce the cost to just two matrix multiplications per step. Incorporating a QR decomposition allows us to extract an approximate orthonormal basis spanning the top singular directions, rather than just the leading one. This amortized power iteration is fully compatible with standard distributed training techniques such as FSDP<\/strong> and tensor parallelism<\/strong>. Here, we show a simple centralized version, but the technique works for more complex forms of parallelization as presented in the paper. In other words, we can orthogonalize a matrix without ever seeing a full row or column of it<\/em>. <\/p>\n\n\n\n

Low-rank approximation would ordinarily introduce error, but Dion overcomes this through an error feedback mechanism. This keeps the residual of low rank approximation in the momentum matrix so that any systematic gradient structure not initially captured accumulates to eventually be applied in a future update.<\/p>\n\n\n\n\t

\n\t\t\n\n\t\t

\n\t\tSpotlight: Microsoft research newsletter<\/span>\n\t<\/p>\n\t\n\t

\n\t\t\t\t\t\t
\n\t\t\t\t\n\t\t\t\t\t\"\"\n\t\t\t\t<\/a>\n\t\t\t<\/div>\n\t\t\t\n\t\t\t
\n\n\t\t\t\t\t\t\t\t\t

Microsoft Research Newsletter<\/h2>\n\t\t\t\t\n\t\t\t\t\t\t\t\t

Stay connected to the research community at Microsoft.<\/p>\n\t\t\t\t\n\t\t\t\t\t\t\t\t

\n\t\t\t\t\t
\n\t\t\t\t\t\t\n\t\t\t\t\t\t\tSubscribe today\t\t\t\t\t\t<\/a>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t\t<\/div>\n\t<\/div>\n\t<\/div>\n\t\n\n\n

How does it work?<\/h2>\n\n\n\n

Something very strange happened in our experiments. Usually, adding an extra constraint on the way an algorithm works can be expected to decrease<\/em> overall performance. And indeed, at the 120M parameter scale of the speedrun, we see Dion\u2019s update taking more time than Muon, while not yielding any significant gains. But at larger scales, we observed a different trend: Dion began to outperform Muon.<\/p>\n\n\n\n

\"Wall-clock
Figure 3. Wall-clock time speedup of Dion for 3B model training<\/figcaption><\/figure>\n\n\n\n

Why would adding a constraint improve<\/em> the update rule? The answer lies in what the constraint enforces. Dion achieves a much closer approximation to true orthonormalization than Muon. This precision, initially subtle, becomes increasingly important as the number of singular vectors grows. Over increasing model scale and training steps, this small advantage accumulates\u2014leading to a measurable improvement in performance.<\/p>\n\n\n\n

This edge further grows with batch size\u2014with larger batches the update quality tends to degrade, but notably more slowly with Dion than Muon (and Muon is already a significant improvement over AdamW).<\/p>\n\n\n\n

\"Scaling
Figure 4. Scaling of Dion across different batch sizes<\/figcaption><\/figure>\n\n\n\n

Here you can see how the number of steps to reach a pretraining loss compared to AdamW varies as batch size grows with full rank and \u00bc rank Dion (in orange) and Muon (in blue).   <\/p>\n\n\n\n

In our experiments, these benefits extend to various post-training regimes as well.<\/p>\n\n\n\n

We also experimented with rank, discovering empirically that larger models tolerate smaller rank well.<\/p>\n\n\n\n

\"Low-rank
Figure 5. Low-rank Dion across different model sizes<\/figcaption><\/figure>\n\n\n\n

Projecting this trend out to the scale of the LLaMA-3 (opens in new tab)<\/span><\/a> 405B parameter models suggests that Dion is fully effective even with rank fractions as low as 1\/16 or 1\/64<\/strong> for large dense models like LLaMA-3.    <\/p>\n\n\n\n

Using hardware timings of the individual update steps suggests a story that looks this:<\/p>\n\n\n\n

\"Estimated
Figure 6. Estimated wall-clock time of each optimizer step for Llama 3 405B. Lower is better. Muon is highlighted in orange as our baseline, next to Dion with varying rank fractions. Suggested rank fractions for a 405B parameter model are shown in blue. Using Dion with rank fraction 1\/16 or lower offers an order-of-magnitude speedup over Muon.<\/figcaption><\/figure>\n\n\n\n

We\u2019ve open-sourced a PyTorch FSDP2 + Tensor Parallel (TP) implementation of Dion<\/strong>, available via a simple pip install. Our goal is to make faster training with Dion accessible to everyone. As a bonus, the repository also includes a PyTorch FSDP2 implementation of Muon.<\/strong><\/p>\n\n\n\n

\n
Dion optimizer<\/a><\/div>\n<\/div>\n\n\n\n

Acknowledgements<\/h2>\n\n\n\n

We thank Riashat Islam and Pratyusha Sharma for their helpful feedback on the writing and presentation.<\/p>\n","protected":false},"excerpt":{"rendered":"

Dion is a new AI model optimization method that boosts scalability and performance over existing leading methods by orthonormalizing only a top rank subset of singular vectors, enabling more efficient training of large models such as LLaMA-3 with reduced overhead.<\/p>\n","protected":false},"author":43868,"featured_media":1147793,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","msr-author-ordering":[{"type":"user_nicename","value":"Kwangjun Ahn","user_id":"43950"},{"type":"user_nicename","value":"John Langford","user_id":"32204"}],"msr_hide_image_in_river":null,"footnotes":""},"categories":[1],"tags":[],"research-area":[13556],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[269148,243984,269142],"msr-impact-theme":[],"msr-promo-type":[],"msr-podcast-series":[],"class_list":["post-1146334","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-research-blog","msr-research-area-artificial-intelligence","msr-locale-en_us","msr-post-option-approved-for-river","msr-post-option-blog-homepage-featured","msr-post-option-include-in-river"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"","podcast_episode":"","msr_research_lab":[992148],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[],"related-projects":[],"related-events":[],"related-researchers":[{"type":"user_nicename","value":"John Langford","user_id":32204,"display_name":"John Langford","author_link":"John Langford<\/a>","is_active":false,"last_first":"Langford, John","people_section":0,"alias":"jcl"}],"msr_type":"Post","featured_image_thumbnail":"\"Three","byline":"Kwangjun Ahn and John Langford<\/a>","formattedDate":"August 12, 2025","formattedExcerpt":"Dion is a new AI model optimization method that boosts scalability and performance over existing leading methods by orthonormalizing only a top rank subset of singular vectors, enabling more efficient training of large models such as LLaMA-3 with reduced overhead.","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/1146334","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/users\/43868"}],"replies":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/comments?post=1146334"}],"version-history":[{"count":27,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/1146334\/revisions"}],"predecessor-version":[{"id":1147795,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/1146334\/revisions\/1147795"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/1147793"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=1146334"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/categories?post=1146334"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/tags?post=1146334"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=1146334"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=1146334"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=1146334"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=1146334"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=1146334"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=1146334"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=1146334"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=1146334"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}