DeepSpeed

DeepSpeed

Extreme Speed and Scale for DL Training and Inference

Microsoft Research ブログ

Figure 1. Trend of sizes of state-of-the-art NLP models over time
Microsoft Research ブログ

| Ali Alvi と Paresh Kharya

We are excited to introduce the DeepSpee…

DeepSpeed MoE powers eight times bigger models using expert-parallelism + ZeRO-Offload compared with expert-parallelism only. A graph shows supported model sizes on NVIDIA A100 GPUs. DeepSpeed MoE scales near-linearly with respect to the number of GPUs. Z-code MoE (10B) consistently outperforms other systems on BLEU scores for an in-house 50 language test dataset. Read more in the blog post. 
Microsoft Research ブログ

| DeepSpeed Team と Z-code Team

Today, we are proud to announce DeepSpee…

DeepSpeed multi GPU inference offers up to 6.9 times throughput improvement for large deep learning model inference. Progressive Layer Dropping offers 2.8 times faster convergence for large model training. 1-bit LAMB offers up to 4.6 times less communication overhead. Single GPU speedups for inference: 2.1 times on BERT Base, 4.4 times on BERT Large, 3.8 times on GPT 2, 3.5 times on GPT 2 XL, 1.9 times on GPT Neo. Multi GPU speedups for inference: 6.2 times for Turing NLG, 3.7 times for 175 billion parameter language model.
Microsoft Research ブログ

| DeepSpeed Team, Rangan Majumder, と Andrey Proskurin

Last month, the DeepSpeed Team announced…

Microsoft Research ブログ

| DeepSpeed Team, Rangan Majumder, と Junhua Wang

In February, we announced DeepSpeed, an …

simulated aerial drone navigating course
Microsoft Research ブログ

“This is a game changer for the big data…

Microsoft Research ブログ

| DeepSpeed Team, Rangan Majumder, と Junhua Wang

In February, we announced DeepSpeed, an …

Microsoft Research ブログ

| Corby Rosset

This figure was adapted from a similar i…

Microsoft Research ブログ

| DeepSpeed Team, Rangan Majumder, と Junhua Wang

The latest trend in AI is that larger na…