{"id":1134179,"date":"2025-03-19T09:00:00","date_gmt":"2025-03-19T16:00:00","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?p=1134179"},"modified":"2025-08-05T07:19:36","modified_gmt":"2025-08-05T14:19:36","slug":"claimify-extracting-high-quality-claims-from-language-model-outputs","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/claimify-extracting-high-quality-claims-from-language-model-outputs\/","title":{"rendered":"Claimify: Extracting high-quality claims from language model outputs"},"content":{"rendered":"\n
\"Gradient<\/figure>\n\n\n\n
\n
Watch Dasha’s Claimify Explainer<\/a><\/div>\n<\/div>\n\n\n\n

This research was accepted by the 63rd annual meeting of the Association for Computational Linguistics (ACL 2025), the premier conference on natural language processing.<\/em><\/p>\n\n\n\n

While large language models (LLMs) are capable of synthesizing vast amounts of information, they sometimes produce inaccurate or unsubstantiated content. To mitigate this risk, tools like Azure AI\u2019s Groundedness Detection (opens in new tab)<\/span><\/a> can be used to verify LLM outputs.\u202f<\/p>\n\n\n\n

A common strategy for fact-checking LLM-generated texts \u2013 especially complex, highly detailed outputs \u2013 is claim extraction<\/strong>: instead of evaluating the entire text at once, it\u2019s broken down into simple factual statements that can be verified independently. However, the effectiveness of this approach depends on the quality of the extracted claims: if they are inaccurate or incomplete, the fact-checking results may be compromised. <\/p>\n\n\n\n

Our paper \u201cTowards Effective Extraction and Evaluation of Factual Claims<\/a>\u201d proposes a framework for evaluating claim extraction and introduces Claimify<\/strong>, a novel LLM-based method that outperforms prior solutions. In this blog post, we discuss common claim extraction issues and explore Claimify\u2019s approach.<\/p>\n\n\n\n

Key issues in claim extraction<\/h2>\n\n\n\n

As a case study, we prompted OpenAI\u2019s GPT-4o model to \u201cprovide an overview of challenges in emerging markets<\/em>\u201d based on a collection of news articles. Here is its output:<\/p>\n\n\n\n

# Emerging market challenges <\/strong>
Emerging markets face a myriad of challenges that span economic, social, and environmental dimensions. These challenges are often interconnected, exacerbating the difficulties faced by these regions.

## Economic instability and inflation <\/strong>
Several emerging markets are grappling with severe economic instability. For instance, Argentina’s rampant inflation, with monthly rates reaching as high as 25.5%, has made many goods unobtainable and plunged the value of the currency, causing severe economic hardship. Some experts estimate that the annual inflation rate could potentially double to 300%, while others predict even higher rates.

## Food insecurity and agricultural challenges <\/strong>
Food insecurity is another critical issue in emerging markets. Nigeria, for example, is striving to become self-sufficient in wheat production but is hindered by climate change and violence, exacerbated by high grain prices due to the suspension of the Black Sea Grain Initiative. Climate change has played a pivotal role in creating food insecurity and economic instability in farming-dependent economies, such as Zambia and Mozambique.

## Natural disasters and public health emergencies <\/strong>
Public health crises, as demonstrated by the experience of Derna, Libya, remain a significant concern in emerging markets. Catastrophic flooding caused by the collapse of two dams has led to significant destruction, complicating access to healthcare. The U.N. found that the resulting contaminated water caused many residents to fall ill, highlighting the need for improved water management. Countries like Afghanistan and Sudan have experienced similar challenges to those of Libya.

## Conclusion <\/strong>
Addressing emerging market challenges will require comprehensive strategies that consider economic stability, food security, and public health. International cooperation and targeted interventions are crucial in fostering sustainable development in emerging markets.<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

We then applied a simple prompt[1]<\/a> that extracted the following claims from the answer:<\/p>\n\n\n\n

1. Emerging markets face economic challenges.
2. Emerging markets face social challenges.
3. Emerging markets face environmental challenges.
4. Argentina experiences severe economic instability.
5. Argentina has a monthly inflation rate as high as 25.5%.
6. Argentina’s inflation has made many goods unobtainable.
7. Argentina’s currency value has plunged.
8. Some experts estimate Argentina’s annual inflation rate could reach 300%.
9. Food insecurity is a critical issue in emerging markets.
10. Nigeria is striving to become self-sufficient in wheat production.
11. Nigeria’s wheat production is hindered by climate change.
12. Nigeria’s wheat production is hindered by violence.
13. High grain prices affect Nigeria due to the suspension of the Black Sea Grain Initiative.
14. Climate change creates food insecurity in Zambia.
15. Climate change creates food insecurity in Mozambique.
16. Climate change causes economic instability in farming-dependent economies.
17. Public health crises are a concern in emerging markets.
18. Derna, Libya, experienced catastrophic flooding.
19. The collapse of two dams caused flooding in Derna, Libya.
20. Flooding in Derna, Libya, complicated access to healthcare.
21. The U.N. found contaminated water in Derna, Libya.
22. Contaminated water in Derna, Libya, caused many residents to fall ill.
23. Afghanistan has experienced challenges similar to those in Libya.
24. Sudan has experienced challenges similar to those in Libya.
25. Addressing emerging market challenges requires comprehensive strategies.
26. Comprehensive strategies should consider economic stability.
27. Comprehensive strategies should consider food security.
28. Comprehensive strategies should consider public health.
29. International cooperation is crucial for sustainable development in emerging markets.
30. Targeted interventions are crucial for sustainable development in emerging markets.<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

These claims appear to reflect the answer, but a closer look reveals four critical issues:<\/p>\n\n\n\n

#<\/th>Issue<\/th>Example<\/th><\/tr><\/thead>
1<\/td>Some claims are not factual statements that can be verified as true or false.<\/td>Excerpt:<\/strong> \u201cAddressing emerging market challenges will require comprehensive strategies that consider economic stability, food security, and public health.<\/em>\u201d

Claims:<\/strong>
  • Addressing emerging market challenges requires comprehensive strategies.<\/li>
  • Comprehensive strategies should consider economic stability.<\/li>
  • Comprehensive strategies should consider food security.<\/li>
  • Comprehensive strategies should consider public health.<\/li><\/ul>Explanation: <\/strong>These claims are not verifiable because they are opinions.<\/td><\/tr>
2<\/td>Some claims are missing or incomplete.<\/td>Excerpt:<\/strong> \u201cArgentina’s rampant inflation, with monthly rates reaching as high as 25.5%, has made many goods unobtainable and plunged the value of the currency, causing severe economic hardship<\/u>. Some experts estimate that the annual inflation rate could potentially double to 300%, while others predict even higher rates<\/u>.<\/em>\u201d

Claims:<\/strong>
  • Argentina has a monthly inflation rate as high as 25.5%.<\/li>
  • Argentina’s inflation has made many goods unobtainable.<\/li>
  • Argentina’s currency value has plunged.<\/li>
  • Some experts estimate Argentina\u2019s annual inflation rate could reach 300%.<\/li><\/ul> Explanation: <\/strong>The phrases \u201ccausing severe economic hardship<\/em>\u201d and \u201cothers predict even higher rates<\/em>\u201d are not reflected in any of the claims. The third claim also omits the fact that inflation caused the currency depreciation.<\/td><\/tr>
3<\/td>Some claims are inaccurate.<\/td>Excerpt: <\/strong>\u201cThe U.N. found that the resulting contaminated water caused many residents to fall ill, highlighting the need for improved water management<\/em>.\u201d

Claims:<\/strong>
  • The U.N. found contaminated water in Derna, Libya.<\/li>
  • Contaminated water in Derna, Libya, caused many residents to fall ill.<\/li><\/ul> Explanation: <\/strong>The first claim is inaccurate because the U.N. found the link between contaminated water and illness, not the contaminated water itself. The second claim also misrepresents the sentence since it shifts the meaning from a viewpoint of a specific entity (the U.N.) to a general assertion about the effects of contaminated water in Derna, Libya.<\/td><\/tr>
4<\/td>Some claims cannot be understood without additional context.<\/td>Excerpt: <\/strong>\u201cCountries like Afghanistan and Sudan have experienced similar challenges to those of Libya.<\/em>\u201d

Claims:<\/strong>