{"id":991293,"date":"2023-12-12T06:00:00","date_gmt":"2023-12-12T14:00:00","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?p=991293"},"modified":"2023-12-15T16:20:52","modified_gmt":"2023-12-16T00:20:52","slug":"phi-2-the-surprising-power-of-small-language-models","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/phi-2-the-surprising-power-of-small-language-models\/","title":{"rendered":"Phi-2: The surprising power of small language models"},"content":{"rendered":"\n

Contributors<\/h3>\n\n\n\n

Marah Abdin<\/a>, Jyoti Aneja<\/a>, Sebastien Bubeck<\/a>, Caio C\u00e9sar Teodoro Mendes, Weizhu Chen<\/a>, Allie Del Giorno, Ronen Eldan<\/a>, Sivakanth Gopi<\/a>, Suriya Gunasekar<\/a>, Mojan Javaheripi<\/a>, Piero Kauffmann<\/a>, Yin Tat Lee<\/a>, Yuanzhi Li, Anh Nguyen<\/a>, Gustavo de Rosa<\/a>, Olli Saarikivi<\/a>, Adil Salim<\/a>, Shital Shah<\/a>, Michael Santacroce, Harkirat Singh Behl, Adam Taumann Kalai<\/a>, Xin Wang<\/a>, Rachel Ward<\/a>, Philipp Witte<\/a>, Cyril Zhang<\/a>, Yi Zhang<\/p>\n\n\n\n

\"Satya
Figure 1. <\/strong>Satya Nadella announcing Phi-2 at Microsoft Ignite 2023.<\/figcaption><\/figure>\n\n\n\n

Over the past few months, our Machine Learning Foundations team at Microsoft Research has released a suite of small language models (SLMs) called \u201cPhi\u201d that achieve remarkable performance on a variety of benchmarks. Our first model, the 1.3 billion parameter Phi-1<\/strong> (opens in new tab)<\/span><\/a>, achieved state-of-the-art performance on Python coding among existing SLMs (specifically on the HumanEval and MBPP benchmarks). We then extended our focus to common sense reasoning and language understanding and created a new 1.3 billion parameter model named Phi-1.5<\/strong> (opens in new tab)<\/span><\/a>, with performance comparable to models 5x larger.<\/p>\n\n\n\n

We are now releasing Phi-2<\/strong> (opens in new tab)<\/span><\/a>, a 2.7 billion-parameter language model that demonstrates outstanding reasoning and language understanding capabilities, showcasing state-of-the-art performance among base language models with less than 13 billion parameters. On complex benchmarks Phi-2 matches or outperforms models up to 25x larger, thanks to new innovations in model scaling and training data curation.<\/p>\n\n\n\n

With its compact size, Phi-2 is an ideal playground for researchers, including for exploration around mechanistic interpretability, safety improvements, or fine-tuning experimentation on a variety of tasks. We have made Phi-2<\/strong> (opens in new tab)<\/span><\/a> <\/strong>available in the Azure AI Studio model catalog to foster research and development on language models.<\/p>\n\n\n\n\t

\n\t\t\n\n\t\t

\n\t\tvideo series<\/span>\n\t<\/p>\n\t\n\t

\n\t\t\t\t\t\t
\n\t\t\t\t\n\t\t\t\t\t\"On\n\t\t\t\t<\/a>\n\t\t\t<\/div>\n\t\t\t\n\t\t\t
\n\n\t\t\t\t\t\t\t\t\t

On Second Thought<\/h2>\n\t\t\t\t\n\t\t\t\t\t\t\t\t

A video series with Sinead Bovell built around the questions everyone\u2019s asking about AI. With expert voices from across Microsoft, we break down the tension and promise of this rapidly changing technology, exploring what\u2019s evolving and what\u2019s possible.<\/p>\n\t\t\t\t\n\t\t\t\t\t\t\t\t

\n\t\t\t\t\t
\n\t\t\t\t\t\t\n\t\t\t\t\t\t\tExplore the series\t\t\t\t\t\t<\/a>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t\t<\/div>\n\t<\/div>\n\t<\/div>\n\t\n\n\n

Key Insights Behind Phi-2<\/h2>\n\n\n\n

The massive increase in the size of language models to hundreds of billions of parameters has unlocked a host of emerging capabilities that have redefined the landscape of natural language processing. A question remains whether such emergent abilities can be achieved at a smaller scale using strategic choices for training, e.g., data selection.<\/p>\n\n\n\n

Our line of work with the Phi models aims to answer this question by training SLMs that achieve performance on par with models of much higher scale (yet still far from the frontier models). Our key insights for breaking the conventional language model scaling laws with Phi-2 are twofold:<\/p>\n\n\n\n

Firstly, training data quality plays a critical role in model performance. This has been known for decades, but we take this insight to its extreme by focusing on \u201ctextbook-quality\u201d data, following upon our prior work \u201cTextbooks Are All You Need<\/a>.\u201d Our training data mixture contains synthetic datasets specifically created to teach the model common sense reasoning and general knowledge, including science, daily activities, and theory of mind, among others. We further augment our training corpus with carefully selected web data that is filtered based on educational value and content quality. Secondly, we use innovative techniques to scale up, starting from our 1.3 billion parameter model, Phi-1.5, and embedding its knowledge within the 2.7 billion parameter Phi-2. This scaled knowledge transfer not only accelerates training convergence but shows clear boost in Phi-2 benchmark scores.<\/p>\n\n\n\n

\"A
Figure 2. <\/strong>Comparison between Phi-2 (2.7B) and Phi-1.5 (1.3B) models. All tasks are evaluated in 0-shot except for BBH and MMLU which use 3-shot CoT and 5-shot, respectively.<\/figcaption><\/figure>\n\n\n\n

Training Details<\/h2>\n\n\n\n

Phi-2 is a Transformer-based model with a next-word prediction objective, trained on 1.4T tokens from multiple passes on a mixture of Synthetic and Web datasets for NLP and coding. The training for Phi-2 took 14 days on 96 A100 GPUs. Phi-2 is a base model that has not undergone alignment through reinforcement learning from human feedback (RLHF), nor has it been instruct fine-tuned. Despite this, we observed better behavior with respect to toxicity and bias compared to existing open-source models that went through alignment (see Figure 3). This is in line with what we saw in Phi-1.5 due to our tailored data curation technique, see our previous tech report<\/a> for more details on this. For more information about the Phi-2 model, please visit Azure AI | Machine Learning Studio (opens in new tab)<\/span><\/a>.<\/p>\n\n\n\n

\"A
Figure 3. <\/strong>Safety scores computed on 13 demographics from ToxiGen. A subset of 6541 sentences are selected and scored between 0 to 1 based on scaled perplexity and sentence toxicity. A higher score indicates the model is less likely to produce toxic sentences compared to benign ones.<\/figcaption><\/figure>\n\n\n\n

Phi-2 Evaluation<\/h2>\n\n\n\n

Below, we summarize Phi-2 performance on academic benchmarks compared to popular language models. Our benchmarks span several categories, namely, Big Bench Hard (BBH) (3 shot with CoT), commonsense reasoning (PIQA, WinoGrande, ARC easy and challenge, SIQA), language understanding (HellaSwag, OpenBookQA, MMLU (5-shot), SQuADv2 (2-shot), BoolQ), math (GSM8k (8 shot)), and coding (HumanEval, MBPP (3-shot)).<\/p>\n\n\n\n

With only 2.7 billion parameters, Phi-2 surpasses the performance of Mistral and Llama-2 models at 7B and 13B parameters on various aggregated benchmarks. Notably, it achieves better performance compared to 25x larger Llama-2-70B model on muti-step reasoning tasks, i.e., coding and math. Furthermore, Phi-2 matches or outperforms the recently-announced Google Gemini Nano 2, despite being smaller in size.<\/p>\n\n\n\n

Of course, we acknowledge the current challenges with model evaluation, and that many public benchmarks might leak into the training data. For our first model, Phi-1, we did an extensive decontamination study to discard this possibility, which can be found in our first report \u201cTextbooks Are All You Need<\/a>.\u201d Ultimately, we believe that the best way to judge a language model is to test it on concrete use cases. Following that spirit, we also evaluated Phi-2 using several Microsoft internal proprietary datasets and tasks, comparing it again to Mistral and Llama-2. We observed similar trends, i.e. on average, Phi-2 outperforms Mistral-7B, and the latter outperforms the Llama-2 models (7B, 13B, and 70B).<\/p>\n\n\n\n

Model<\/th>Size<\/th>BBH<\/th>Commonsense
Reasoning<\/th>		Language
Understanding<\/th>		Math<\/th>Coding<\/th><\/tr><\/thead>
Llama-2<\/td>7B<\/td>40.0<\/td>62.2<\/td>56.7<\/td>16.5<\/td>21.0<\/td><\/tr>
13B<\/td>47.8<\/td>65.0<\/td>61.9<\/td>34.2<\/td>25.4<\/td><\/tr>
70B<\/td>66.5<\/td>69.2<\/td>67.6<\/td>64.1<\/td>38.3<\/td><\/tr>
Mistral<\/td>7B<\/td>57.2<\/td>66.4<\/td>63.7<\/td>46.4<\/td>39.4<\/td><\/tr>
Phi-2<\/td>2.7B<\/td>59.2<\/td>68.8<\/td>62.0<\/td>61.1<\/td>53.7<\/td><\/tr><\/tbody><\/table>
Table 1.<\/strong> Averaged performance on grouped benchmarks compared to popular open-source SLMs.<\/center><\/figcaption><\/figure>\n\n\n\n
Model<\/th>Size<\/th>BBH<\/th>BoolQ<\/th>MBPP<\/th>MMLU<\/th><\/tr><\/thead>
Gemini Nano 2<\/td>3.2B<\/td>42.4<\/td>79.3<\/td>27.2<\/td>55.8<\/td><\/tr>
Phi-2<\/td>2.7B<\/td>59.3<\/td>83.3<\/td>59.1<\/td>56.7<\/td><\/tr><\/tbody><\/table>
Table 2.<\/strong> Comparison between Phi-2 and Gemini Nano 2 Model on Gemini\u2019s reported benchmarks.<\/center><\/figcaption><\/figure>\n\n\n\n

In addition to these benchmarks, we also performed extensive testing on commonly used prompts from the research community. We observed a behavior in accordance with the expectation we had given the benchmark results. For example, we tested a prompt used to probe a model\u2019s ability to solve physics problems, most recently used to evaluate the capabilities of the Gemini Ultra model, and achieved the following result:<\/p>\n\n\n\n

\"An
Figure 4. <\/strong>Phi-2’s output on a simple physics problem, which includes an approximately correct square root calculation.<\/figcaption><\/figure>\n\n\n\n
\"The
Figure 5. <\/strong>Similarly to Gemini\u2019s test we also further queried Phi-2 with a student\u2019s wrong answer to see if Phi-2 could identify where the mistake is (it did, despite Phi-2 being not fine-tuned for chat or instruction-following). We note however that it is not fully an apple-to-apple comparison with the Gemini Ultra\u2019s output described in the Gemini report, in particular in the latter case the student\u2019s answer was given as an image with handwritten text rather than raw text in our case.<\/figcaption><\/figure>\n","protected":false},"excerpt":{"rendered":"

Phi-2 is now accessible on the Azure model catalog. Its compact size and new innovations in model scaling and training data curation make it ideal for exploration around mechanistic interpretability, safety improvements, and fine-tuning experimentation on a variety of tasks.<\/p>\n","protected":false},"author":42183,"featured_media":991311,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","msr-author-ordering":[{"type":"user_nicename","value":"Mojan Javaheripi","user_id":"42777"},{"type":"user_nicename","value":"S\u00e9bastien Bubeck","user_id":"33570"}],"msr_hide_image_in_river":0,"footnotes":""},"categories":[1],"tags":[],"research-area":[13556],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[243984],"msr-impact-theme":[],"msr-promo-type":[],"msr-podcast-series":[],"class_list":["post-991293","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-research-blog","msr-research-area-artificial-intelligence","msr-locale-en_us","msr-post-option-blog-homepage-featured"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"","podcast_episode":"","msr_research_lab":[],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[],"related-projects":[],"related-events":[968280],"related-researchers":[],"msr_type":"Post","featured_image_thumbnail":"\"Satya","byline":"Mojan Javaheripi and S\u00e9bastien Bubeck","formattedDate":"December 12, 2023","formattedExcerpt":"Phi-2 is now accessible on the Azure model catalog. Its compact size and new innovations in model scaling and training data curation make it ideal for exploration around mechanistic interpretability, safety improvements, and fine-tuning experimentation on a variety of tasks.","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/991293","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/users\/42183"}],"replies":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/comments?post=991293"}],"version-history":[{"count":26,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/991293\/revisions"}],"predecessor-version":[{"id":993270,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/991293\/revisions\/993270"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/991311"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=991293"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/categories?post=991293"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/tags?post=991293"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=991293"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=991293"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=991293"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=991293"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=991293"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=991293"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=991293"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=991293"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}