{"id":1133691,"date":"2025-03-18T09:00:00","date_gmt":"2025-03-18T16:00:00","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?p=1133691"},"modified":"2025-03-18T07:28:45","modified_gmt":"2025-03-18T14:28:45","slug":"introducing-kblam-bringing-plug-and-play-external-knowledge-to-llms","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/introducing-kblam-bringing-plug-and-play-external-knowledge-to-llms\/","title":{"rendered":"Introducing KBLaM: Bringing plug-and-play external knowledge to LLMs"},"content":{"rendered":"\n
\"KBLaM<\/figure>\n\n\n\n

Large language models (LLMs) have demonstrated remarkable capabilities in reasoning, language understanding, and even creative tasks. Yet, a key challenge persists: how to efficiently integrate external knowledge.<\/p>\n\n\n\n

Traditional methods such as fine-tuning and Retrieval-Augmented Generation (RAG) come with trade-offs\u2014fine-tuning demands costly retraining, while RAG introduces separate retrieval modules that increase complexity and prevent seamless, end-to-end training. In-context learning, on the other hand, becomes increasingly inefficient as knowledge bases grow, facing quadratic computational scaling that hinders its ability to handle large repositories. A comparison of these approaches can be seen in Figure 1.<\/p>\n\n\n\n

A new way to integrate knowledge<\/h2>\n\n\n\n

To address these challenges, we introduce the Knowledge Base-Augmented Language Model (KBLaM)<\/strong><\/a> \u2014a novel approach that integrates structured knowledge bases into pre-trained LLMs. Instead of relying on external retrieval modules or costly fine-tuning, KBLaM encodes knowledge into continuous key-value vector pairs<\/strong>, efficiently embedding them within the model\u2019s attention layers using a specialized rectangular attention mechanism<\/strong>, which implicitly performs retrieval in an integrated manner.<\/p>\n\n\n\n

We use structured knowledge bases to represent the data, allowing us to consolidate knowledge <\/strong>and leverage structure<\/strong>. This design allows it to scale linearly<\/strong> with the size of the knowledge base while maintaining dynamic<\/strong> updates<\/strong> without retraining, making it far more efficient than existing methods.<\/p>\n\n\n\n\t

\n\t\t\n\n\t\t

\n\t\tPODCAST SERIES<\/span>\n\t<\/p>\n\t\n\t

\n\t\t\t\t\t\t
\n\t\t\t\t\n\t\t\t\t\t\"Illustrated\n\t\t\t\t<\/a>\n\t\t\t<\/div>\n\t\t\t\n\t\t\t
\n\n\t\t\t\t\t\t\t\t\t

AI Testing and Evaluation: Learnings from Science and Industry<\/h2>\n\t\t\t\t\n\t\t\t\t\t\t\t\t

Discover how Microsoft is learning from other domains to advance evaluation and testing as a pillar of AI governance.<\/p>\n\t\t\t\t\n\t\t\t\t\t\t\t\t

\n\t\t\t\t\t
\n\t\t\t\t\t\t\n\t\t\t\t\t\t\tListen now\t\t\t\t\t\t<\/a>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t\t<\/div>\n\t<\/div>\n\t<\/div>\n\t\n\n\n

Scalable, efficient, and future-ready<\/h2>\n\n\n\n

At its core, KBLaM is designed to integrate structured knowledge into LLMs, making them more efficient and scalable. It achieves this by converting external knowledge bases\u2014collections of facts structured as triples consisting of an entity, a property, and a value\u2014into a format that LLMs can process naturally.  Such knowledge bases allow for consolidated, reliable sources of knowledge.<\/p>\n\n\n\n

To create these knowledge bases, we first extract structured data in JSON format using small language models. We then apply Project Alexandria<\/a>\u2019s probabilistic clustering. Once we have this structured knowledge base, KBLaM follows a three-step pipeline:<\/p>\n\n\n\n

    \n
  1. Knowledge Encoding:<\/strong> Each knowledge triple is mapped into a key-value vector pair using a pre-trained sentence encoder<\/strong> with lightweight linear adapters<\/strong>. The key vector, derived from the entity name and property, encodes \u201cindex information,\u201d while the value vector captures the corresponding property value. This allows us to create continuous, learnable key-value representations<\/strong>.<\/li>\n\n\n\n
  2. Integration with LLMs:<\/strong> These key-value pairs, or knowledge tokens<\/em><\/strong>, are augmented into the model\u2019s attention layers using a specialized rectangular attention structure<\/strong>. Unlike traditional transformer models that process all tokens equally and come with quadratic cost\u2014such as GPT-4, Phi, and Llama\u2014rectangular attention enables the model to attend over knowledge with linear cost, as illustrated in Figure 2. Compared to standard attention mechanisms in generative language models, where each token attends to all preceding tokens, our approach introduces a more efficient structure. In this setup, language tokens (such as those from a user\u2019s question) attend to all knowledge tokens. However, knowledge tokens do not attend to one another, nor do they attend back to the language tokens. This selective attention pattern significantly reduces computational cost while preserving the model\u2019s ability to incorporate external knowledge effectively.

    This linear cost, which is crucial for the efficiency of KBLaM, effectively amounts to treating each fact independently\u2014an assumption that holds for most facts. For example, the model\u2019s name, KBLaM, and the fact that the research was conducted at Microsoft Research are very weakly correlated. This rectangular attention is implemented as an extension of standard attention. During training, we keep the base model\u2019s weights frozen, ensuring that when no knowledge tokens are provided, the model functions exactly as it did originally.<\/li>\n\n\n\n
  3. Efficient Knowledge Retrieval:<\/strong> Through this rectangular attention, the model learns to dynamically retrieve relevant knowledge tokens during inference, eliminating the need for separate retrieval steps.<\/li>\n<\/ol>\n\n\n\n
    \"Figure
    Figure 1: KBLaM allows for attention over the entire knowledge base instead of having an external retriever.<\/em><\/em><\/figcaption><\/figure>\n\n\n\n
    \"Figure
    Figure 2: By having the user\u2019s question attend to the knowledge base, while treating facts in the knowledge base independently, KBLaM scales efficiently and linearly with the size of the knowledge base.<\/em><\/em><\/figcaption><\/figure>\n\n\n\n

    Unlike RAG, which appends retrieved document chunks to prompts, KBLaM allows for direct integration<\/strong> of knowledge into the model. Compared to in-context learning,  KBLaM\u2019s rectangular attention maintains a linear memory footprint<\/strong>, making it vastly more scalable for large knowledge bases. <\/p>\n\n\n\n

    Its efficiency is a game-changer. While traditional in-context learning methods struggle with quadratic memory growth due to self-attention overhead, KBLaM\u2019s linear overhead means we can store much more knowledge in the context. In practice, this means KBLaM can store and process over 10,000 knowledge triples<\/strong>, the equivalent of approximately 200,000 text tokens on a single GPU<\/strong>\u2014a feat that would be computationally prohibitive with conventional in-context learning. The results across a wide range of triples and can be seen in Figure 3. Remarkably, it achieves this while extending a base model that has a context length of only 8K tokens<\/strong>. Additionally, KBLaM enables dynamic updates: modifying <\/strong>a single knowledge triple does not require retraining or re-computation of the entire knowledge base. <\/p>\n\n\n\n

    \"Figure
    Figure 3: KBLaM is much faster and uses much less memory than adding the equivalent number of triples in the context using conventional RAG-like approaches. In particular, we have lower time to first token with 4,096 tripes in the context with KBLaM than we would with 5 triples in the context.<\/em><\/em><\/figcaption><\/figure>\n\n\n\n

    Enhancing interpretability and reliability<\/h2>\n\n\n\n

    Another major benefit of KBLaM is its interpretability<\/strong>. Unlike in-context learning, where knowledge injection is opaque, KBLAM\u2019s attention weights<\/strong> provide clear insights into how the model utilizes knowledge tokens. Experiments show that KBLaM assigns high attention scores to relevant knowledge triples, effectively mimicking a soft retrieval process.<\/p>\n\n\n\n

    Furthermore, KBLaM enhances model reliability<\/strong> by learning through its training examples when not to answer a question if the necessary information is missing from the knowledge base. In particular, with knowledge bases larger than approximately 200 triples, we found that the model refuses to answer questions it has no knowledge about more precisely than a model given the information as text in context. This feature helps reduce hallucinations<\/strong>, a common problem in LLMs that rely on internal knowledge alone, making responses more accurate and trustworthy.<\/p>\n\n\n\n

    The future of knowledge-augmented AI<\/h2>\n\n\n\n

    KBLaM represents a major step forward in integrating structured knowledge into LLMs. By offering a scalable, efficient, and interpretable alternative to existing techniques, it paves the way for AI systems that can stay up to date and provide reliable, knowledge-driven responses. In fields where accuracy and trust are critical\u2014such as medicine, finance, and scientific research\u2014this approach has the potential to transform how language models interact with real-world information.<\/p>\n\n\n\n

    As AI systems increasingly rely on dynamic knowledge rather than static model parameters, we hope KBLaM will serve as a bridge between raw computational power and real-world understanding.<\/p>\n\n\n\n

    However, there is still work to be done before it can be deployed at scale. Our current model has been trained primarily on factual question-answer pairs, and further research is needed to expand its capabilities across more complex reasoning tasks and diverse knowledge domains.<\/p>\n\n\n\n

    To accelerate progress, we are releasing KBLaM\u2019s code and datasets<\/strong> (opens in new tab)<\/span><\/a> to the research community, and we are planning integrations with the Hugging Face transformers library. By making these resources available, we hope to inspire further research and adoption of scalable, efficient knowledge augmentation<\/strong> for LLMs. The future of AI isn\u2019t just about generating text\u2014it\u2019s about generating knowledge that is accurate, adaptable, and deeply integrated with the evolving world. KBLaM is a step in that direction.<\/p>\n","protected":false},"excerpt":{"rendered":"

    Introducing KBLaM, an approach that encodes and stores structured knowledge within an LLM itself. By integrating knowledge without retraining, it offers a scalable alternative to traditional methods.<\/p>\n","protected":false},"author":43518,"featured_media":1134055,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","msr-author-ordering":null,"msr_hide_image_in_river":null,"footnotes":""},"categories":[1],"tags":[],"research-area":[13556],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[269148,269142],"msr-impact-theme":[],"msr-promo-type":[],"msr-podcast-series":[],"class_list":["post-1133691","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-research-blog","msr-research-area-artificial-intelligence","msr-locale-en_us","msr-post-option-approved-for-river","msr-post-option-include-in-river"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"","podcast_episode":"","msr_research_lab":[199561],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[1142643],"related-projects":[],"related-events":[],"related-researchers":[{"type":"guest","value":"xi-wang","user_id":"1133697","display_name":"Xi Wang","author_link":"Xi Wang<\/a>","is_active":true,"last_first":"Wang, Xi","people_section":0,"alias":"xi-wang"},{"type":"guest","value":"liana-mikaelyan","user_id":"1133703","display_name":"Liana Mikaelyan","author_link":"Liana Mikaelyan<\/a>","is_active":true,"last_first":"Mikaelyan, Liana","people_section":0,"alias":"liana-mikaelyan"},{"type":"guest","value":"mathew-salvaris","user_id":"1133699","display_name":"Mathew Salvaris","author_link":"Mathew Salvaris<\/a>","is_active":true,"last_first":"Salvaris, Mathew","people_section":0,"alias":"mathew-salvaris"},{"type":"user_nicename","value":"James Hensman","user_id":43404,"display_name":"James Hensman","author_link":"James Hensman<\/a>","is_active":false,"last_first":"Hensman, James","people_section":0,"alias":"jameshensman"}],"msr_type":"Post","featured_image_thumbnail":"\"KBLaM","byline":"","formattedDate":"March 18, 2025","formattedExcerpt":"Introducing KBLaM, an approach that encodes and stores structured knowledge within an LLM itself. By integrating knowledge without retraining, it offers a scalable alternative to traditional methods.","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/1133691","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/users\/43518"}],"replies":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/comments?post=1133691"}],"version-history":[{"count":39,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/1133691\/revisions"}],"predecessor-version":[{"id":1134400,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/1133691\/revisions\/1134400"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/1134055"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=1133691"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/categories?post=1133691"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/tags?post=1133691"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=1133691"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=1133691"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=1133691"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=1133691"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=1133691"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=1133691"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=1133691"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=1133691"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}