{"id":1132281,"date":"2025-03-01T19:30:59","date_gmt":"2025-03-02T03:30:59","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?post_type=msr-project&p=1132281"},"modified":"2025-06-11T12:57:17","modified_gmt":"2025-06-11T19:57:17","slug":"secom","status":"publish","type":"msr-project","link":"https:\/\/www.microsoft.com\/en-us\/research\/project\/secom\/","title":{"rendered":"SeCom: On Memory Construction & Retrieval for Personalized Conversational Agents"},"content":{"rendered":"
\n\t
\n\t\t
\n\t\t\t\"feature_image\"\t\t<\/div>\n\t\t\n\t\t
\n\t\t\t\n\t\t\t
\n\t\t\t\t\n\t\t\t\t
\n\t\t\t\t\t\n\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\n

SeCom<\/h1>\n\n\n\n

On Memory Construction & Retrieval <\/strong>for Personalized Conversational Agents<\/p>\n\n\t\t\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t<\/div>\n<\/section>\n\n\n\n\n\n

How can conversational agents better retain and retrieve past interactions<\/strong> for more coherent and personalized<\/strong> experiences? Our latest work – SeCom <\/strong>on Memory Construction & Retrieval<\/strong> tackles this challenge head-on!<\/p>\n\n\n\n

Existing approaches typically perform retrieval augmented response generation by constructing memory banks from conversation history at either the turn-level, session-level, or through summarization. In SeCom, we present two key findings<\/strong>: (1) The granularity of memory unit matters: Turn-level, session-level, and summarization-based methods each exhibit limitations in both memory retrieval accuracy and the semantic quality of the retrieved content. (2) Prompt compression methods, such as LLMLingua-2<\/em>, can effectively serve as a denoising mechanism, enhancing memory retrieval accuracy across different granularities.<\/p>\n\n\n\n

Building on these insights, we propose SeCom<\/strong>, a method that constructs the memory bank at segment level by introducing a conversation Se<\/strong>gmentation model that partitions long-term conversations into topically coherent segments, while applying Com<\/strong>pression based denoising on memory units to enhance memory retrieval. Experimental results show that SeCom<\/strong> exhibits a significant performance advantage over baselines on long-term conversation benchmarks LOCOMO and Long-MT-Bench+. Additionally, the proposed conversation segmentation method demonstrates superior performance on dialogue segmentation datasets such as DialSeg711, TIAGE, and SuperDialSeg. <\/p>\n\n\n\n

Key Takeaways<\/h2>\n\n\n\n

\ud83d\udca1 Memory granularity matters: <\/strong>Turn-level, session-level & summarization-based memory struggle with retrieval accuracy and the semantic integrity or relevance of the context.<\/p>\n\n\n\n

\ud83d\udca1 Prompt compression methods<\/strong> (e.g., LLMLingua-2 (opens in new tab)<\/span><\/a>) can denoise memory retrieval, boosting <\/strong>both retrieval accuracy <\/strong>and response quality.<\/strong><\/p>\n\n\n\n

\u2705 SeCom<\/strong> \u2013 an approach that segments conversations topically<\/strong> for memory construction <\/strong>and performs memory retrieval<\/strong> based on compressed memory units<\/strong>.<\/p>\n\n\n\n

\ud83d\udcca Result<\/strong> \u2013 superior performance on long-term conversation benchmarks such as LOCOMO and Long-MT-Bench+!<\/p>\n\n\n\n

<\/div>\n\n\n\n

What’s the Impact of Memory Granularity?<\/h3>\n\n\n\n

We first systematically investigate the impact of different memory granularities on conversational agents within the paradigm of retrieval augmented response generation. Our findings indicate that turn-level, session-level, and summarization-based methods all exhibit limitations in terms of the accuracy of the retrieval module as well as the semantics of the retrieved content, which ultimately lead to sub-optimal responses.<\/p>\n\n\n\n

\ud83d\udca1 Long conversations are naturally composed of coherent discourse units. To capture this structure, we introduce a conversation segmentation model that partitions long-term conversations into topically coherent segments, constructing the memory bank at the segment level. During response generation, we directly concatenate the retrieved segment-level memory units as the context, bypassing summarization to avoid the information loss that often occurs when converting dialogues into summaries.<\/p>\n\n\n\n

\"Illustration
Figure 1. Illustration of retrieval augmented response generation with different memory granularities.<\/strong>\u00a0Turn-level memory<\/em>\u00a0is too fine-grained, leading to fragmentary and incomplete context.\u00a0Session-level memory\u00a0<\/em>is too coarse-grained, containing too much irrelevant information.\u00a0Summary based methods<\/em>\u00a0suffer from information loss that occurs during summarization.\u00a0Ours (segment-level memory)<\/em>\u00a0can better capture topically coherent units in long conversations, striking a balance between including more relevant, coherent information while excluding irrelevant content. \ud83c\udfaf indicates the retrieved memory units at\u00a0turn level\u00a0or\u00a0segment level\u00a0under the same context budget. [0.xx]: similarity between target query and history content.<\/figcaption><\/figure>\n\n\n\n
<\/div>\n\n\n\n
\"Illustration
Figure 2. How memory granularity affects (a) the response quality and (b, c) retrieval accuracy.<\/strong><\/figcaption><\/figure>\n\n\n\n
<\/div>\n\n\n\n

Does Memory Denoising Help?<\/h3>\n\n\n\n

Inspired by the notion that natural language tends to be inherently redundant, we hypothesize that such redundancy can act as noise for retrieval systems, complicating the extraction of key information. Therefore, we propose removing such redundancy from memory units prior to retrieval by leveraging prompt compression methods such as LLMLingua-2<\/em>. Figure 3 shows the results.<\/p>\n\n\n\n

\"Illustration
Figure 3. Prompt compression method (LLMLingua-2) can serve as an effective denoising technique to enhance the memory retrieval system<\/strong>\u00a0by: (a) improving the retrieval recall with varying context budget K; (b) benefiting the retrieval system by increasing the similarity between the query and relevant segments while decreasing the similarity with irrelevant ones.<\/figcaption><\/figure>\n\n\n\n
<\/div>\n\n\n\n

SeCom<\/h2>\n\n\n\n

To address the above challenges, we present SeCom<\/strong>, a system that constructs memory bank at segment level<\/strong> by introducing a Conversation Segmentation Model<\/em>, while applying Compression-Based Denoising<\/em> on memory units to enhance memory retrieval.<\/p>\n\n\n\n

Conversation Segmentation Model<\/h3>\n\n\n\n

Given a conversation session \\(c\\), the conversation segmentation model \\(f_\\mathcal{I}\\) aims to identify a set of segment indices \\(\\mathcal{I}=\\{(p_k, q_k)\\} {K \\atop{k}}\\) where \\(K\\) denotes the total number of segments within the session \\(c,p_k\\) and \\(q_k\\) represent the indexes of the first and last interaction turns for the \\(k\\)-th segment \\(s_k,\\) with \\(p_k \\; {\\le} \\; q_{k}, p_{k+1} = q_k + 1\\). This can be formulated as:<\/p>\n\n\n\n

\\(f_{\\mathcal{I}}(c) = \\{s_k\\}{K\\atop{k=1}},\\) where \\(s_k = \\{t_{p_k}, t_{{p_k}+1},…, t_{q_k}\\}.\\)<\/p>\n\n\n\n

We employ GPT-4<\/strong> as the conversation segmentation model \\(f_\\mathcal{I}\\). We find that more lightweight models, such as Mistral-7B (opens in new tab)<\/span><\/a> and even RoBERTa (opens in new tab)<\/span><\/a> scale models, can also perform segmentation well, making our apporach applicable in resource-constrained environments.<\/p>\n\n\n\n

Compression-Based Memory Denoising<\/h3>\n\n\n\n

Given a target user request \\(u^*\\) and context budget \\(N\\), the memory retrieval system \\(f_R\\) retrieves \\(N\\) memory units \\(\\{m_n \\; \\in \\; \\mathcal{M}\\}{N\\atop{n=1}}\\) from the memory bank \\(\\mathcal{M}\\) as the context in response to the user request \\(u^*\\). With the consideration that the inherent redundancy in natural language can act as noise for the retrieval system, we denoise memory units by removing such redundancy via a prompt compression model \\(f_{Comp}\\) before retrieval:<\/p>\n\n\n\n

\\(\\{m_n \\; \\in \\; \\mathcal{M}\\}{N\\atop{n=1}} \\leftarrow f_R(u^*, f_{Comp}(\\mathcal{M}),N).\\)<\/p>\n\n\n\n

Experiments<\/h2>\n\n\n\n

We evaluate SeCom<\/strong> against four intuitive approaches and four state-of-the-art models on long-term conversation benchmarks: LOCOMO (opens in new tab)<\/span><\/a> and Long-MT-Bench+ (opens in new tab)<\/span><\/a>.<\/p>\n\n\n\n

Main Results<\/h3>\n\n\n\n

As shown in the following table, SeCom<\/strong> outperforms all baseline approaches, exhibiting a significant performance advantage, particularly on the long-conversation benchmark LOCOMO<\/em>. Interestingly, there is a significant performance disparity in turn-Level<\/em> and session-Level<\/em> methods when using different retrieval models. In contrast, SeCom<\/strong> enjoys greater robustness in terms of the deployed retrieval system.<\/p>\n\n\n\n

\"Table<\/figure>\n\n\n\n

The figure below presents the pairwise comparison result by instructing GPT-4 to determine the superior response. SeCom achieves a higher win rate compared to all baseline methods. We attribute this to the fact that topical segments in SeCom can strike a balance between including more relevant, coherent information while excluding irrelevant content, thus leading to more robust and superior retrieval performance.<\/p>\n\n\n\n

\"Figure<\/figure>\n\n\n\n

Impact of the Memory Unit Granularity<\/h3>\n\n\n\n

The figure below compares QA performance across different memory granularities under varying context budgets, demonstrating the superiority of segment-level memory<\/strong> over turn-level and session-level memory.<\/p>\n\n\n\n

\"Figure<\/figure>\n\n\n\n

Evaluation of Conversation Segmentation Model<\/h3>\n\n\n\n

We evaluate the conversation segmentation module<\/strong> independently on widely used dialogue segmentation datasets: DialSeg711 (opens in new tab)<\/span><\/a>, TIAGE (opens in new tab)<\/span><\/a> and SuperDialSeg (opens in new tab)<\/span><\/a>. The following table presents the result, showing that our segmentation model consistently outperforms baselines in the unsupervised segmentation setting.<\/p>\n\n\n\n

\"Table<\/figure>\n\n\n\n

The Effect of Compression-Based Memory Denoising<\/h3>\n\n\n\n

As shown in table below, removing the proposed compression-based memory denoising mechanism will result in a performance drop up to 9.46 points of GPT4Score on LOCOMO<\/em>, highlighting the critical role of this denoising mechanism: by effectively improving the retrieval system, it significantly enhances the overall effectiveness of the system.<\/p>\n\n\n\n

\"Table<\/figure>\n\n\n\n

BibTex<\/h2>\n\n\n\n
@inproceedings{pan2025secom,\n    title={SeCom: On Memory Construction and Retrieval for Personalized Conversational Agents},\n    author={Zhuoshi Pan and Qianhui Wu and Huiqiang Jiang and Xufang Luo and Hao Cheng and Dongsheng Li and Yuqing Yang and Chin-Yew Lin and H. Vicky Zhao and Lili Qiu and Jianfeng Gao},\n    booktitle={The Thirteenth International Conference on Learning Representations},\n    year={2025},\n    url={https:\/\/openreview.net\/forum?id=xKDZAW0He3}\n}<\/code><\/pre>\n\n\n","protected":false},"excerpt":{"rendered":"
On Memory Construction & Retrieval for Personalized Conversational Agents How can conversational agents better retain and retrieve past interactions for more coherent and personalized experiences? Our latest work – SeCom on Memory Construction & Retrieval tackles this challenge head-on! Existing approaches typically perform retrieval augmented response generation by constructing memory banks from conversation history at […]<\/p>\n","protected":false},"featured_media":1133142,"template":"","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","footnotes":""},"research-area":[13556,13545,13554],"msr-locale":[268875],"msr-impact-theme":[],"msr-pillar":[],"class_list":["post-1132281","msr-project","type-msr-project","status-publish","has-post-thumbnail","hentry","msr-research-area-artificial-intelligence","msr-research-area-human-language-technologies","msr-research-area-human-computer-interaction","msr-locale-en_us","msr-archive-status-active"],"msr_project_start":"","related-publications":[1133157],"related-downloads":[],"related-videos":[],"related-groups":[],"related-events":[],"related-opportunities":[],"related-posts":[1135033],"related-articles":[],"tab-content":[],"slides":[],"related-researchers":[{"type":"guest","display_name":"Zhuoshi Pan","user_id":1133154,"people_section":"People","alias":""},{"type":"user_nicename","display_name":"Qianhui Wu","user_id":40741,"people_section":"People","alias":"qianhuiwu"},{"type":"user_nicename","display_name":"Hao Cheng","user_id":39922,"people_section":"People","alias":"chehao"},{"type":"user_nicename","display_name":"Dongsheng Li","user_id":39402,"people_section":"People","alias":"dongsli"},{"type":"user_nicename","display_name":"Yuqing Yang","user_id":40654,"people_section":"People","alias":"yuqyang"},{"type":"user_nicename","display_name":"Chin-Yew Lin","user_id":31493,"people_section":"People","alias":"cyl"},{"type":"guest","display_name":"H. Vicky Zhao","user_id":1133175,"people_section":"People","alias":""},{"type":"user_nicename","display_name":"Jianfeng Gao","user_id":32246,"people_section":"People","alias":"jfgao"}],"msr_research_lab":[199560,199565],"msr_impact_theme":[],"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/1132281","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/msr-project"}],"version-history":[{"count":59,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/1132281\/revisions"}],"predecessor-version":[{"id":1141893,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/1132281\/revisions\/1141893"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/1133142"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=1132281"}],"wp:term":[{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=1132281"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=1132281"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=1132281"},{"taxonomy":"msr-pillar","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-pillar?post=1132281"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}