ZeRO++: Extremely Efficient Collective Communication for Giant Model Training
Guanhua Wang, Heyang Qin, Sam Ade Jacobs, Connor Holmes, Samyam Rajbhandari, Olatunji Ruwase, Feng Yang, Lei Yang, Yuxiong He
ICLR 2024 | May 2024
Olatunji Ruwase
Team Lead
Publications
Publications by Year
-
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
Marah I Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Caio César Teodoro Mendes, Weizhu Chen, Vishrav Chaudhary, Parul Chopra, Allie Del Giorno, Gustavo de Rosa, Matthew Dixon, Ronen Eldan, Dan Iter, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Jamie Huynh, Mojan Javaheripi, Xin Jin, Piero Kauffmann, Nikos Karampatziakis, Dongwoo Kim, Mahmoud Khademi, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Chen Liang, Weishung Liu, Xihui (Eric) Lin, Zeqi Lin, Piyush Madan, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez-Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Corby Rosset, Sambudha Roy, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Xia Song, Olatunji Ruwase, Xin Wang, Rachel Ward, Guanhua Wang, Philipp Witte, Michael Wyatt, Can Xu, Jiahang Xu, Sonali Yadav, Fan Yang, Ziyi Yang, Donghan Yu, Chengruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yunan Zhang, Xiren Zhou
MSR-TR-2024-12 | April 2024
Published by Microsoft
-
DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale
Reza Yazdani Aminabadi, Samyam Rajbhandari, Minjia Zhang, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Jeff Rasley, Shaden Smith, Olatunji Ruwase, Yuxiong He
MSR-TR-2022-21 | June 2022
Published by Microsoft
-
SimiGrad: Fine-Grained Adaptive Batching for Large Scale Training using Gradient Similarity Measurement
Heyang Qin, Samyam Rajbhandari, Olatunji Ruwase, Feng Yan, Lei Yang, Yuxiong He
NeurIPS 2021 | November 2021
ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning
Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, Yuxiong He
April 2021
ZeRO-Offload: Democratizing Billion-Scale Model Training
Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, Yuxiong He
January 2021
-
ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, Yuxiong He
May 2020
-
HyperDrive: Exploring Hyperparameters with POP Scheduling
Jeff Rasley, Yuxiong He, Olatunji Ruwase, Feng Yan
Proceedings of the 18th ACM/IFIP/USENIX Middleware Conference | December 2017
Optimizing CNNs on Multicores for Scalability, Performance and Goodput
Samyam Rajbhandari, Yuxiong He, Olatunji Ruwase, Michael Carbin, Trishul Chilimbi
2017 Architectural Support for Programming Languages and Operating Systems | April 2017
-
SERF: Efficient Scheduling for Fast Deep Neural Network Serving via Judicious Parallelism
Feng Yan, Yuxiong He, Olatunji Ruwase, Evgenia Smirni
SC'16 | November 2016
ISBN: 978-1-4673-8815-3
-
Toward Accelerating Deep Learning at Scale Using Specialized Hardware in the Datacenter
Kalin Ovtcharov, Olatunji Ruwase, Joo-Young Kim, Jeremy Fowers, Karin Strauss, Eric Chung
Proceedings of the 27th IEEE HotChips Symposium on High-Performance Chips (HotChips 2015) | August 2015
Performance Modeling and Scalability Optimization of Distributed Deep Learning Systems
Feng Yan, Olatunji Ruwase, Yuxiong He, Trishul Chilimbi
SIGKDD International Conference on Knowledge Discovery and Data Mining | August 2015
Page Overlays: An Enhanced Virtual Memory Framework to Enable Fine-grained Memory Management
Vivek Seshadri, Gennady Pekhimenko, Olatunji Ruwase, Onur Mutlu, Phillip B. Gibbons, Michael A. Kozuch, Todd C. Mowry, Trishul Chilimbi
June 2015
Accelerating Deep Convolutional Neural Networks Using Specialized Hardware
Kalin Ovtcharov, Olatunji Ruwase, Joo-Young Kim, Jeremy Fowers, Karin Strauss, Eric Chung
February 2015
-
Guardrail: A High Fidelity Approach to Protecting Hardware Devices from Buggy Drivers
Olatunji Ruwase, Michael A. Kozuch, Phillip B. Gibbons, Todd C. Mowry
Architectural Support for Programming Languages and Operating Systems (ASPLOS) | March 2014
-
Decoupled Lifeguards: Enabling Path Optimizations for Dynamic Correctness Checking Tools
Olatunji Ruwase, Shimin Chen, Phillip B. Gibbons, Todd C. Mowry
Programming Language Design and Implementation (PLDI) | January 2010
-
Ditto: A System for Opportunistic Caching in Multi-Hop Wireless Mesh Networks
Fahad Dogar, Amar Phanishayee, Himabindu Pucha, Olatunji Ruwase, David Andersen
ACM MOBICOM 2008 | September 2008
Flexible Hardware Acceleration for Instruction-Grain Program Monitoring
Shimin Chen, Michael Kozuch, Theodoros Strigkos, Babak Falsafi, Phillip B. Gibbons, Todd C. Mowry, Vijaya Ramachandran, Olatunji Ruwase, Michael Ryan, Evangelos Vlachos
International Symposium on Computer Architecture | June 2008
Parallelizing Dynamic Information Flow Tracking
Olatunji Ruwase, Phillip B. Gibbons, Todd C. Mowry, Vijaya Ramachandran, Shimin Chen, Michael Kozuch, Michael Ryan
June 2008
-
A Practical Dynamic Buffer Overflow Detector
Olatunji Ruwase, Monica S. Lam
Network and Distributed Systems Security (NDSS) | February 2004
Publications by Research Area
-
ZeRO++: Extremely Efficient Collective Communication for Giant Model Training
Guanhua Wang, Heyang Qin, Sam Ade Jacobs, Connor Holmes, Samyam Rajbhandari, Olatunji Ruwase, Feng Yang, Lei Yang, Yuxiong He
ICLR 2024 | May 2024
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
Marah I Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Caio César Teodoro Mendes, Weizhu Chen, Vishrav Chaudhary, Parul Chopra, Allie Del Giorno, Gustavo de Rosa, Matthew Dixon, Ronen Eldan, Dan Iter, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Jamie Huynh, Mojan Javaheripi, Xin Jin, Piero Kauffmann, Nikos Karampatziakis, Dongwoo Kim, Mahmoud Khademi, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Chen Liang, Weishung Liu, Xihui (Eric) Lin, Zeqi Lin, Piyush Madan, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez-Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Corby Rosset, Sambudha Roy, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Xia Song, Olatunji Ruwase, Xin Wang, Rachel Ward, Guanhua Wang, Philipp Witte, Michael Wyatt, Can Xu, Jiahang Xu, Sonali Yadav, Fan Yang, Ziyi Yang, Donghan Yu, Chengruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yunan Zhang, Xiren Zhou
MSR-TR-2024-12 | April 2024
Published by Microsoft
DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale
Reza Yazdani Aminabadi, Samyam Rajbhandari, Minjia Zhang, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Jeff Rasley, Shaden Smith, Olatunji Ruwase, Yuxiong He
MSR-TR-2022-21 | June 2022
Published by Microsoft
SimiGrad: Fine-Grained Adaptive Batching for Large Scale Training using Gradient Similarity Measurement
Heyang Qin, Samyam Rajbhandari, Olatunji Ruwase, Feng Yan, Lei Yang, Yuxiong He
NeurIPS 2021 | November 2021
ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning
Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, Yuxiong He
April 2021
ZeRO-Offload: Democratizing Billion-Scale Model Training
Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, Yuxiong He
January 2021
ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, Yuxiong He
May 2020
Optimizing CNNs on Multicores for Scalability, Performance and Goodput
Samyam Rajbhandari, Yuxiong He, Olatunji Ruwase, Michael Carbin, Trishul Chilimbi
2017 Architectural Support for Programming Languages and Operating Systems | April 2017
Performance Modeling and Scalability Optimization of Distributed Deep Learning Systems
Feng Yan, Olatunji Ruwase, Yuxiong He, Trishul Chilimbi
SIGKDD International Conference on Knowledge Discovery and Data Mining | August 2015
Accelerating Deep Convolutional Neural Networks Using Specialized Hardware
Kalin Ovtcharov, Olatunji Ruwase, Joo-Young Kim, Jeremy Fowers, Karin Strauss, Eric Chung
February 2015
-
Optimizing CNNs on Multicores for Scalability, Performance and Goodput
Samyam Rajbhandari, Yuxiong He, Olatunji Ruwase, Michael Carbin, Trishul Chilimbi
2017 Architectural Support for Programming Languages and Operating Systems | April 2017
Decoupled Lifeguards: Enabling Path Optimizations for Dynamic Correctness Checking Tools
Olatunji Ruwase, Shimin Chen, Phillip B. Gibbons, Todd C. Mowry
Programming Language Design and Implementation (PLDI) | January 2010
Flexible Hardware Acceleration for Instruction-Grain Program Monitoring
Shimin Chen, Michael Kozuch, Theodoros Strigkos, Babak Falsafi, Phillip B. Gibbons, Todd C. Mowry, Vijaya Ramachandran, Olatunji Ruwase, Michael Ryan, Evangelos Vlachos
International Symposium on Computer Architecture | June 2008
Parallelizing Dynamic Information Flow Tracking
Olatunji Ruwase, Phillip B. Gibbons, Todd C. Mowry, Vijaya Ramachandran, Shimin Chen, Michael Kozuch, Michael Ryan
June 2008
A Practical Dynamic Buffer Overflow Detector
Olatunji Ruwase, Monica S. Lam
Network and Distributed Systems Security (NDSS) | February 2004
-
ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, Yuxiong He
May 2020
HyperDrive: Exploring Hyperparameters with POP Scheduling
Jeff Rasley, Yuxiong He, Olatunji Ruwase, Feng Yan
Proceedings of the 18th ACM/IFIP/USENIX Middleware Conference | December 2017
Optimizing CNNs on Multicores for Scalability, Performance and Goodput
Samyam Rajbhandari, Yuxiong He, Olatunji Ruwase, Michael Carbin, Trishul Chilimbi
2017 Architectural Support for Programming Languages and Operating Systems | April 2017
SERF: Efficient Scheduling for Fast Deep Neural Network Serving via Judicious Parallelism
Feng Yan, Yuxiong He, Olatunji Ruwase, Evgenia Smirni
SC'16 | November 2016
ISBN: 978-1-4673-8815-3
Toward Accelerating Deep Learning at Scale Using Specialized Hardware in the Datacenter
Kalin Ovtcharov, Olatunji Ruwase, Joo-Young Kim, Jeremy Fowers, Karin Strauss, Eric Chung
Proceedings of the 27th IEEE HotChips Symposium on High-Performance Chips (HotChips 2015) | August 2015
Performance Modeling and Scalability Optimization of Distributed Deep Learning Systems
Feng Yan, Olatunji Ruwase, Yuxiong He, Trishul Chilimbi
SIGKDD International Conference on Knowledge Discovery and Data Mining | August 2015
Page Overlays: An Enhanced Virtual Memory Framework to Enable Fine-grained Memory Management
Vivek Seshadri, Gennady Pekhimenko, Olatunji Ruwase, Onur Mutlu, Phillip B. Gibbons, Michael A. Kozuch, Todd C. Mowry, Trishul Chilimbi
June 2015
Accelerating Deep Convolutional Neural Networks Using Specialized Hardware
Kalin Ovtcharov, Olatunji Ruwase, Joo-Young Kim, Jeremy Fowers, Karin Strauss, Eric Chung
February 2015
Guardrail: A High Fidelity Approach to Protecting Hardware Devices from Buggy Drivers
Olatunji Ruwase, Michael A. Kozuch, Phillip B. Gibbons, Todd C. Mowry
Architectural Support for Programming Languages and Operating Systems (ASPLOS) | March 2014
Ditto: A System for Opportunistic Caching in Multi-Hop Wireless Mesh Networks
Fahad Dogar, Amar Phanishayee, Himabindu Pucha, Olatunji Ruwase, David Andersen
ACM MOBICOM 2008 | September 2008
Parallelizing Dynamic Information Flow Tracking
Olatunji Ruwase, Phillip B. Gibbons, Todd C. Mowry, Vijaya Ramachandran, Shimin Chen, Michael Kozuch, Michael Ryan
June 2008
-
Toward Accelerating Deep Learning at Scale Using Specialized Hardware in the Datacenter
Kalin Ovtcharov, Olatunji Ruwase, Joo-Young Kim, Jeremy Fowers, Karin Strauss, Eric Chung
Proceedings of the 27th IEEE HotChips Symposium on High-Performance Chips (HotChips 2015) | August 2015
Accelerating Deep Convolutional Neural Networks Using Specialized Hardware
Kalin Ovtcharov, Olatunji Ruwase, Joo-Young Kim, Jeremy Fowers, Karin Strauss, Eric Chung
February 2015
-
Accelerating Deep Convolutional Neural Networks Using Specialized Hardware
Kalin Ovtcharov, Olatunji Ruwase, Joo-Young Kim, Jeremy Fowers, Karin Strauss, Eric Chung
February 2015
Publications by Type
-
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
Marah I Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Caio César Teodoro Mendes, Weizhu Chen, Vishrav Chaudhary, Parul Chopra, Allie Del Giorno, Gustavo de Rosa, Matthew Dixon, Ronen Eldan, Dan Iter, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Jamie Huynh, Mojan Javaheripi, Xin Jin, Piero Kauffmann, Nikos Karampatziakis, Dongwoo Kim, Mahmoud Khademi, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Chen Liang, Weishung Liu, Xihui (Eric) Lin, Zeqi Lin, Piyush Madan, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez-Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Corby Rosset, Sambudha Roy, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Xia Song, Olatunji Ruwase, Xin Wang, Rachel Ward, Guanhua Wang, Philipp Witte, Michael Wyatt, Can Xu, Jiahang Xu, Sonali Yadav, Fan Yang, Ziyi Yang, Donghan Yu, Chengruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yunan Zhang, Xiren Zhou
MSR-TR-2024-12 | April 2024
Published by Microsoft
DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale
Reza Yazdani Aminabadi, Samyam Rajbhandari, Minjia Zhang, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Jeff Rasley, Shaden Smith, Olatunji Ruwase, Yuxiong He
MSR-TR-2022-21 | June 2022
Published by Microsoft
-
ZeRO++: Extremely Efficient Collective Communication for Giant Model Training
Guanhua Wang, Heyang Qin, Sam Ade Jacobs, Connor Holmes, Samyam Rajbhandari, Olatunji Ruwase, Feng Yang, Lei Yang, Yuxiong He
ICLR 2024 | May 2024
SimiGrad: Fine-Grained Adaptive Batching for Large Scale Training using Gradient Similarity Measurement
Heyang Qin, Samyam Rajbhandari, Olatunji Ruwase, Feng Yan, Lei Yang, Yuxiong He
NeurIPS 2021 | November 2021
HyperDrive: Exploring Hyperparameters with POP Scheduling
Jeff Rasley, Yuxiong He, Olatunji Ruwase, Feng Yan
Proceedings of the 18th ACM/IFIP/USENIX Middleware Conference | December 2017
Optimizing CNNs on Multicores for Scalability, Performance and Goodput
Samyam Rajbhandari, Yuxiong He, Olatunji Ruwase, Michael Carbin, Trishul Chilimbi
2017 Architectural Support for Programming Languages and Operating Systems | April 2017
SERF: Efficient Scheduling for Fast Deep Neural Network Serving via Judicious Parallelism
Feng Yan, Yuxiong He, Olatunji Ruwase, Evgenia Smirni
SC'16 | November 2016
ISBN: 978-1-4673-8815-3
Toward Accelerating Deep Learning at Scale Using Specialized Hardware in the Datacenter
Kalin Ovtcharov, Olatunji Ruwase, Joo-Young Kim, Jeremy Fowers, Karin Strauss, Eric Chung
Proceedings of the 27th IEEE HotChips Symposium on High-Performance Chips (HotChips 2015) | August 2015
Performance Modeling and Scalability Optimization of Distributed Deep Learning Systems
Feng Yan, Olatunji Ruwase, Yuxiong He, Trishul Chilimbi
SIGKDD International Conference on Knowledge Discovery and Data Mining | August 2015
Page Overlays: An Enhanced Virtual Memory Framework to Enable Fine-grained Memory Management
Vivek Seshadri, Gennady Pekhimenko, Olatunji Ruwase, Onur Mutlu, Phillip B. Gibbons, Michael A. Kozuch, Todd C. Mowry, Trishul Chilimbi
June 2015
Guardrail: A High Fidelity Approach to Protecting Hardware Devices from Buggy Drivers
Olatunji Ruwase, Michael A. Kozuch, Phillip B. Gibbons, Todd C. Mowry
Architectural Support for Programming Languages and Operating Systems (ASPLOS) | March 2014
Decoupled Lifeguards: Enabling Path Optimizations for Dynamic Correctness Checking Tools
Olatunji Ruwase, Shimin Chen, Phillip B. Gibbons, Todd C. Mowry
Programming Language Design and Implementation (PLDI) | January 2010
Ditto: A System for Opportunistic Caching in Multi-Hop Wireless Mesh Networks
Fahad Dogar, Amar Phanishayee, Himabindu Pucha, Olatunji Ruwase, David Andersen
ACM MOBICOM 2008 | September 2008
Flexible Hardware Acceleration for Instruction-Grain Program Monitoring
Shimin Chen, Michael Kozuch, Theodoros Strigkos, Babak Falsafi, Phillip B. Gibbons, Todd C. Mowry, Vijaya Ramachandran, Olatunji Ruwase, Michael Ryan, Evangelos Vlachos
International Symposium on Computer Architecture | June 2008
Parallelizing Dynamic Information Flow Tracking
Olatunji Ruwase, Phillip B. Gibbons, Todd C. Mowry, Vijaya Ramachandran, Shimin Chen, Michael Kozuch, Michael Ryan
June 2008
-
ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning
Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, Yuxiong He
April 2021
ZeRO-Offload: Democratizing Billion-Scale Model Training
Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, Yuxiong He
January 2021
ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, Yuxiong He
May 2020
Accelerating Deep Convolutional Neural Networks Using Specialized Hardware
Kalin Ovtcharov, Olatunji Ruwase, Joo-Young Kim, Jeremy Fowers, Karin Strauss, Eric Chung
February 2015
-
A Practical Dynamic Buffer Overflow Detector
Olatunji Ruwase, Monica S. Lam
Network and Distributed Systems Security (NDSS) | February 2004