DeepSpeed

DeepSpeed

Extreme Speed and Scale for DL Training and Inference

微软研究院博客

Figure 1. Trend of sizes of state-of-the-art NLP models over time
微软研究院博客

| Ali Alvi 和 Paresh Kharya

We are excited to introduce the DeepSpee…

DeepSpeed MoE powers eight times bigger models using expert-parallelism + ZeRO-Offload compared with expert-parallelism only. A graph shows supported model sizes on NVIDIA A100 GPUs. DeepSpeed MoE scales near-linearly with respect to the number of GPUs. Z-code MoE (10B) consistently outperforms other systems on BLEU scores for an in-house 50 language test dataset. Read more in the blog post. 
微软研究院博客

| DeepSpeed Team 和 Z-code Team

Today, we are proud to announce DeepSpee…

DeepSpeed multi GPU inference offers up to 6.9 times throughput improvement for large deep learning model inference. Progressive Layer Dropping offers 2.8 times faster convergence for large model training. 1-bit LAMB offers up to 4.6 times less communication overhead. Single GPU speedups for inference: 2.1 times on BERT Base, 4.4 times on BERT Large, 3.8 times on GPT 2, 3.5 times on GPT 2 XL, 1.9 times on GPT Neo. Multi GPU speedups for inference: 6.2 times for Turing NLG, 3.7 times for 175 billion parameter language model.
微软研究院博客

| DeepSpeed Team, Rangan Majumder, 和 Andrey Proskurin

Last month, the DeepSpeed Team announced…

微软研究院博客

| DeepSpeed Team, Rangan Majumder, 和 Junhua Wang

In February, we announced DeepSpeed, an …

simulated aerial drone navigating course
微软研究院博客

“This is a game changer for the big data…

微软研究院博客

| DeepSpeed Team, Rangan Majumder, 和 Junhua Wang

In February, we announced DeepSpeed, an …

微软研究院博客

| Corby Rosset

This figure was adapted from a similar i…

微软研究院博客

| DeepSpeed Team, Rangan Majumder, 和 Junhua Wang

The latest trend in AI is that larger na…