{"id":1136235,"date":"2025-04-14T12:00:00","date_gmt":"2025-04-14T19:00:00","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?p=1136235"},"modified":"2025-04-29T16:57:24","modified_gmt":"2025-04-29T23:57:24","slug":"engagement-user-expertise-and-satisfaction-key-insights-from-the-semantic-telemetry-project","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/engagement-user-expertise-and-satisfaction-key-insights-from-the-semantic-telemetry-project\/","title":{"rendered":"Engagement, user expertise, and satisfaction: Key insights from the Semantic Telemetry Project"},"content":{"rendered":"\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"2100\" height=\"1182\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Semantic-Telemetry-2-BlogHeroFeature-1400x788-1.jpg\" alt=\"The image features four white icons on a gradient background that transitions from blue on the left to green on the right. The first icon is a network or molecule structure with interconnected nodes. The second icon is a light bulb, symbolizing an idea or innovation. The third icon is a checklist with three items and checkmarks next to each item. The fourth icon consists of two overlapping speech bubbles, representing communication or conversation.\" class=\"wp-image-1136384\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Semantic-Telemetry-2-BlogHeroFeature-1400x788-1.jpg 2100w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Semantic-Telemetry-2-BlogHeroFeature-1400x788-1-300x169.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Semantic-Telemetry-2-BlogHeroFeature-1400x788-1-1024x576.jpg 1024w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Semantic-Telemetry-2-BlogHeroFeature-1400x788-1-768x432.jpg 768w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Semantic-Telemetry-2-BlogHeroFeature-1400x788-1-1536x865.jpg 1536w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Semantic-Telemetry-2-BlogHeroFeature-1400x788-1-2048x1153.jpg 2048w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Semantic-Telemetry-2-BlogHeroFeature-1400x788-1-1066x600.jpg 1066w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Semantic-Telemetry-2-BlogHeroFeature-1400x788-1-655x368.jpg 655w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Semantic-Telemetry-2-BlogHeroFeature-1400x788-1-240x135.jpg 240w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Semantic-Telemetry-2-BlogHeroFeature-1400x788-1-640x360.jpg 640w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Semantic-Telemetry-2-BlogHeroFeature-1400x788-1-960x540.jpg 960w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Semantic-Telemetry-2-BlogHeroFeature-1400x788-1-1280x720.jpg 1280w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Semantic-Telemetry-2-BlogHeroFeature-1400x788-1-1920x1080.jpg 1920w\" sizes=\"auto, (max-width: 2100px) 100vw, 2100px\" \/><\/figure>\n\n\n\n<p>The <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/project\/semantic-telemetry\/\" target=\"_blank\" rel=\"noreferrer noopener\">Semantic Telemetry Project<\/a> aims to better understand complex, turn-based human-AI interactions in Microsoft Copilot using a new data science approach.&nbsp;<\/p>\n\n\n\n<p>This understanding is crucial for recognizing how individuals utilize AI systems to address real-world tasks. It provides actionable insights, enhances key use cases, and identifies opportunities for system improvement.<\/p>\n\n\n\n<p>In a recent <a href=\"https:\/\/www.microsoft.com\/en-us\/research\/blog\/semantic-telemetry-understanding-how-users-interact-with-ai-systems\/?msockid=0f3bfc3f6f3464c71ab7e9906e196520\">blog post<\/a>, we shared our approach for classifying chat log data using large language models (LLMs), which allows us to analyze these interactions at scale and in near real time. We also introduced two of our LLM-generated classifiers: Topics and Task Complexity.&nbsp;<\/p>\n\n\n\n<p>This blog post will examine how our suite of LLM-generated classifiers can serve as early indicators for user engagement and highlight how usage and satisfaction varies based on AI and user expertise.<\/p>\n\n\n\n<p><strong>The key findings from our research are:&nbsp;<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>When users engage in more professional, technical, and complex tasks, they are more likely to continue utilizing the tool and increase their level of interaction with it.&nbsp;<\/li>\n\n\n\n<li>Novice users currently engage in simpler tasks, but their work is gradually becoming more complex over time.&nbsp;<\/li>\n\n\n\n<li>More expert users are satisfied with AI responses only where AI expertise is on par with their own expertise on the topic, while novice users had low satisfaction rates regardless of AI expertise.&nbsp;<\/li>\n<\/ul>\n\n\n\n<p>Read on for more information on these findings. Note that all analyses were conducted on anonymous Copilot in Bing interactions containing no personal information.&nbsp;<\/p>\n\n\n\n<div style=\"padding-bottom:0; padding-top:0\" class=\"wp-block-msr-immersive-section alignfull row has-background has-lighter-gray-background-color has-text-color has-black-color wp-block-msr-immersive-section\">\n\t\n\t<div class=\"container\">\n\t\t<div class=\"wp-block-msr-immersive-section__inner wp-block-msr-immersive-section__inner--narrow\">\n\t\t\t<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<p><em>Classifiers mentioned in article:&nbsp;<\/em><\/p>\n\n\n\n<p><strong>Knowledge work classifier<\/strong>: Tasks that involve creating artifacts related to information work typically requiring creative and analytical thinking. Examples include strategic business planning, software design, and scientific research.&nbsp;<\/p>\n\n\n\n<p><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/blog\/semantic-telemetry-understanding-how-users-interact-with-ai-systems\/?msockid=0f3bfc3f6f3464c71ab7e9906e196520\" target=\"_blank\" rel=\"noreferrer noopener\"><strong>Task complexity classifier<\/strong><\/a>: Assesses the cognitive complexity of a task if a user performs it without the use of AI. We group into two categories: <em>low complexity<\/em> and <em>high complexity<\/em>.&nbsp;<\/p>\n\n\n\n<p><strong>Topics classifier<\/strong>: A single label for the primary topic of the conversation.<\/p>\n\n\n\n<p><strong>User expertise: <\/strong>Labels the user\u2019s expertise on the primary topic within the conversation as one of the following categories: Novice (no familiarity with the topic), Beginner (little prior knowledge or experience), Intermediate (some basic knowledge or familiarity with the topic), Proficient (can apply relevant concepts from conversation), and Expert (deep and comprehensive understanding of the topic).&nbsp;<\/p>\n\n\n\n<p><strong>AI expertise: <\/strong>Labels the AI agent expertise based on the same criteria as user expertise above.&nbsp;<\/p>\n\n\n\n<p><strong>User satisfaction: <\/strong>A 20-question satisfaction\/dissatisfaction rubric that the LLM evaluates to create an aggregate score for overall user satisfaction.&nbsp;<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\t\t<\/div>\n\t<\/div>\n\n\t<\/div>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"what-keeps-bing-chat-users-engaged\">What keeps Bing Chat users engaged?&nbsp;<\/h2>\n\n\n\n<p>We conducted a study of a random sample of 45,000 anonymous Bing Chat users during May 2024. The data was grouped into three cohorts based on user activity over the course of the month:&nbsp;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><em>Light <\/em>(1 active chat session per week)&nbsp;<\/li>\n\n\n\n<li><em>Medium<\/em> (2-3 active chat sessions per week)&nbsp;<\/li>\n\n\n\n<li><em>Heavy<\/em> (4+ active chat sessions per week)&nbsp;<\/li>\n<\/ul>\n\n\n\n<p><strong>The key finding is that <\/strong><strong><em>heavy users<\/em><\/strong><strong> are doing more professional, complex work.<\/strong>&nbsp;<\/p>\n\n\n\n<p>We utilized our <strong>knowledge work classifier<\/strong> to label the chat log data as relating to knowledge work tasks. What we found is knowledge work tasks were higher in all cohorts, with the highest percentage in <em>heavy users<\/em>.&nbsp;<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"2308\" height=\"1429\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_1_Engagement-Knowledge-Work.png\" alt=\"Bar chart illustrating knowledge work distribution across three engagement cohorts: light, medium, and heavy. The chart shows that all three cohorts engage in more knowledge work compared to the 'Not knowledge work' and 'Both' categories, with heavy users performing the most knowledge work. \" class=\"wp-image-1136247\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_1_Engagement-Knowledge-Work.png 2308w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_1_Engagement-Knowledge-Work-300x186.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_1_Engagement-Knowledge-Work-1024x634.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_1_Engagement-Knowledge-Work-768x476.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_1_Engagement-Knowledge-Work-1536x951.png 1536w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_1_Engagement-Knowledge-Work-2048x1268.png 2048w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_1_Engagement-Knowledge-Work-240x149.png 240w\" sizes=\"auto, (max-width: 2308px) 100vw, 2308px\" \/><figcaption class=\"wp-element-caption\"><em>Figure 1: Knowledge work based on engagement cohort<\/em><\/figcaption><\/figure>\n\n\n\n<p>Analyzing task complexity, we observed that users with higher engagement frequently perform the highest number of tasks with high complexity, while users with lower engagement performed more tasks with low complexity.&nbsp;<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1798\" height=\"1087\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_2_Engagement-task-complexity.png\" alt=\"Bar chart illustrating task complexity distribution across three engagement cohorts: light, medium, and heavy. The chart shows all three cohorts perform more high complexity tasks than low complexity tasks, with heavy users performing the greatest number of high complexity tasks. \" class=\"wp-image-1136248\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_2_Engagement-task-complexity.png 1798w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_2_Engagement-task-complexity-300x181.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_2_Engagement-task-complexity-1024x619.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_2_Engagement-task-complexity-768x464.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_2_Engagement-task-complexity-1536x929.png 1536w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_2_Engagement-task-complexity-240x145.png 240w\" sizes=\"auto, (max-width: 1798px) 100vw, 1798px\" \/><figcaption class=\"wp-element-caption\"><em>Figure 2: High complexity and low complexity tasks by engagement cohort+<\/em>&nbsp;<\/figcaption><\/figure>\n\n\n\n<p>Looking at the overall data, we can filter on heavy users and see higher numbers of chats where the user was performing <em>knowledge work<\/em> tasks. Based on task complexity, we see that most <em>knowledge work <\/em>tasks seek to apply a solution to an existing problem, primarily within <em>programming and scripting<\/em>. This is in line with our top overall topic, <em>technology<\/em>, which we discussed in the previous post.&nbsp;<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"2161\" height=\"1219\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_3_Engagement-heavy-users-tree-view.png\" alt=\"Tree diagram illustrating how heavy users are engaging with Bing Chat. The visual selects the most common use case for heavy users: knowledge work, \u201capply\u201d complexity and related topics.  \" class=\"wp-image-1136249\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_3_Engagement-heavy-users-tree-view.png 2161w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_3_Engagement-heavy-users-tree-view-300x169.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_3_Engagement-heavy-users-tree-view-1024x578.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_3_Engagement-heavy-users-tree-view-768x433.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_3_Engagement-heavy-users-tree-view-1536x866.png 1536w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_3_Engagement-heavy-users-tree-view-2048x1155.png 2048w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_3_Engagement-heavy-users-tree-view-1066x600.png 1066w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_3_Engagement-heavy-users-tree-view-655x368.png 655w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_3_Engagement-heavy-users-tree-view-240x135.png 240w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_3_Engagement-heavy-users-tree-view-640x360.png 640w\" sizes=\"auto, (max-width: 2161px) 100vw, 2161px\" \/><figcaption class=\"wp-element-caption\"><em>Figure 3: Heavy users tree diagram<\/em>&nbsp;<\/figcaption><\/figure>\n\n\n\n<p>In contrast, <em>light users<\/em> tended to do more <em>low complexity <\/em>tasks (<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/blog\/semantic-telemetry-understanding-how-users-interact-with-ai-systems\/?msockid=061a2f481be2670621233a8a1a5866f2\">\u201cRemember\u201d<\/a>),&nbsp;using Bing Chat like a traditional search engine and engaging more in topics like <em>business and finance <\/em>and <em>computers and electronics.<\/em><\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"580\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_4_Engagement-light-users-tree-view-1024x580.png\" alt=\"Tree diagram illustrating how light users are engaging with Bing Chat. The visual selects the most common use case for light users: knowledge work, \u201cremember\u201d complexity and related topics. \" class=\"wp-image-1136250\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_4_Engagement-light-users-tree-view-1024x580.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_4_Engagement-light-users-tree-view-300x170.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_4_Engagement-light-users-tree-view-768x435.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_4_Engagement-light-users-tree-view-1536x870.png 1536w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_4_Engagement-light-users-tree-view-2048x1161.png 2048w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_4_Engagement-light-users-tree-view-240x136.png 240w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><figcaption class=\"wp-element-caption\"><em>Figure 4: Light users tree diagram<\/em>&nbsp;<\/figcaption><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"novice-queries-are-becoming-more-complex\">Novice queries are becoming more complex&nbsp;<\/h2>\n\n\n\n<p>We looked at Bing Chat data from January through August 2024 and we classified chats using our <strong>User Expertise<\/strong> classifier. When we looked at how the different user expertise groups were using the tool for professional tasks, we discovered that proficient and expert users tend to do more professional tasks with high complexity in topics like <em>programming and scripting, professional writing and editing<\/em>, <em>and physics and chemistry<\/em>.&nbsp;<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1576\" height=\"1135\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_5_Proficent-Expert-top-topics.png\" alt=\"Bar chart illustrating top topics for proficient and expert users with programming and scripting (18.3%), professional writing and editing (10.4%), and physics and chemistry (9.8%) as top three topics. \" class=\"wp-image-1136251\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_5_Proficent-Expert-top-topics.png 1576w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_5_Proficent-Expert-top-topics-300x216.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_5_Proficent-Expert-top-topics-1024x737.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_5_Proficent-Expert-top-topics-768x553.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_5_Proficent-Expert-top-topics-1536x1106.png 1536w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_5_Proficent-Expert-top-topics-240x173.png 240w\" sizes=\"auto, (max-width: 1576px) 100vw, 1576px\" \/><figcaption class=\"wp-element-caption\"><em>Figure 5: Top topics for proficient\/expert users<\/em>&nbsp;<\/figcaption><\/figure>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1329\" height=\"667\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_6_Proficent-expert-task-complexity.png\" alt=\"Bar chart showing task complexity for proficient and expert users. The chart shows a greater number of high complexity chats than low complexity chats, with the highest percentage in categories \u201cUnderstand\u201d (30.8%) and \u201cApply\u201d (29.3%). \" class=\"wp-image-1136252\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_6_Proficent-expert-task-complexity.png 1329w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_6_Proficent-expert-task-complexity-300x151.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_6_Proficent-expert-task-complexity-1024x514.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_6_Proficent-expert-task-complexity-768x385.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_6_Proficent-expert-task-complexity-240x120.png 240w\" sizes=\"auto, (max-width: 1329px) 100vw, 1329px\" \/><figcaption class=\"wp-element-caption\"><em>Figure 6: Task complexity for proficient\/expert<\/em>&nbsp;<\/figcaption><\/figure>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1882\" height=\"1386\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_7_Novice-top-topics.png\" alt=\"Bar chart illustrating top topics for novice users with business and finance (12.5%), education and learning (10.0%), and computers and electronics (9.8%) as top three topics. \" class=\"wp-image-1136253\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_7_Novice-top-topics.png 1882w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_7_Novice-top-topics-300x221.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_7_Novice-top-topics-1024x754.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_7_Novice-top-topics-768x566.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_7_Novice-top-topics-1536x1131.png 1536w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_7_Novice-top-topics-80x60.png 80w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_7_Novice-top-topics-240x177.png 240w\" sizes=\"auto, (max-width: 1882px) 100vw, 1882px\" \/><figcaption class=\"wp-element-caption\"><em>Figure 7: Top topics for novices<\/em>&nbsp;<\/figcaption><\/figure>\n\n\n\n<p>In contrast, novice users engaged more in professional tasks relating to <em>business and finance<\/em> and <em>education and learning, <\/em>mainly using the tool to recall information.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1588\" height=\"799\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_8_novice-task-complexity.png\" alt=\"Bar chart showing task complexity for novice users. The chart shows a greater number of low complexity chats than high complexity chats, with the highest percentage in categories \u201cRemember\u201d (48.6%). \" class=\"wp-image-1136254\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_8_novice-task-complexity.png 1588w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_8_novice-task-complexity-300x151.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_8_novice-task-complexity-1024x515.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_8_novice-task-complexity-768x386.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_8_novice-task-complexity-1536x773.png 1536w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_8_novice-task-complexity-240x121.png 240w\" sizes=\"auto, (max-width: 1588px) 100vw, 1588px\" \/><figcaption class=\"wp-element-caption\"><em>Figure 8: Task complexity for novices<\/em>&nbsp;<\/figcaption><\/figure>\n\n\n\n<p>However, novices are targeting increasingly more complex tasks over time. Over the eight-month period, we see the percentage of high complexity tasks rise from about 36% to 67%, revealing that novices are learning and adapting quickly (see Figure 9).&nbsp;<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"672\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_9_weekly-high-complexity-novice-1024x672.png\" alt=\"Line chart showing weekly percentage of high complexity chats for novice users from January-August 2024. The line chart starts at 35.9% in January and ends at 67.2% in August.  \" class=\"wp-image-1136255\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_9_weekly-high-complexity-novice-1024x672.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_9_weekly-high-complexity-novice-300x197.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_9_weekly-high-complexity-novice-768x504.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_9_weekly-high-complexity-novice-1536x1007.png 1536w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_9_weekly-high-complexity-novice-240x157.png 240w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_9_weekly-high-complexity-novice.png 1807w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><figcaption class=\"wp-element-caption\"><em>Figure 9: High complexity for novices Jan-Aug 2024<\/em>&nbsp;<\/figcaption><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"how-does-user-satisfaction-vary-according-to-expertise\">How does user satisfaction vary according to expertise?&nbsp;<\/h2>\n\n\n\n<p>We classified both the user expertise and AI agent expertise for anonymous interactions in Copilot in Bing. We compared the level of user and AI agent expertise with our <strong>user satisfaction classifier<\/strong>.&nbsp;<\/p>\n\n\n\n<p><strong>The key takeaways are:<\/strong>&nbsp;<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Experts and proficient users are only satisfied with AI agents with similar expertise (expert\/proficient).&nbsp;<\/li>\n\n\n\n<li>Novices are least satisfied, regardless of the expertise of the AI agent.&nbsp;<\/li>\n<\/ul>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"2548\" height=\"859\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_10_Sat-by-User-AI.png\" alt=\"Table illustrating user satisfaction based on expertise level of user and agent. Each row if the table is the user expertise group (novice, beginner, intermediate, proficient, expert) and on the columns is AI expertise group (novice, beginner, intermediate, proficient, expert). The table illustrates that novice users are least satisfied overall and expert\/proficient users are satisfied with AI expertise of proficient\/expert.  \" class=\"wp-image-1136256\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_10_Sat-by-User-AI.png 2548w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_10_Sat-by-User-AI-300x101.png 300w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_10_Sat-by-User-AI-1024x345.png 1024w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_10_Sat-by-User-AI-768x259.png 768w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_10_Sat-by-User-AI-1536x518.png 1536w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_10_Sat-by-User-AI-2048x690.png 2048w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Figure_10_Sat-by-User-AI-240x81.png 240w\" sizes=\"auto, (max-width: 2548px) 100vw, 2548px\" \/><figcaption class=\"wp-element-caption\"><em>Figure 10: Copilot in Bing satisfaction intersection of AI expertise and User expertise (August-September 2024)<\/em>&nbsp;<\/figcaption><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"conclusion\">Conclusion<\/h2>\n\n\n\n<p>Understanding these metrics is vital for grasping user behavior over time and relating it to real-world business indicators. Users are finding value from complex professional knowledge work tasks, and novices are quickly adapting to the tool and finding these high value use-cases. By analyzing user satisfaction in conjunction with expertise levels, we can tailor our tools to better meet the needs of different user groups. Ultimately, these insights can help improve user understanding across a variety of tasks.&nbsp;&nbsp;<\/p>\n\n\n\n<p>In our next post, we will examine the engineering processes involved in LLM-generated classification.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Semantic Telemetry Project data show that people who use AI for more professional and complex tasks are more likely to keep using the tool and to use it more often. Novice AI users engage in simpler tasks, but their usage is becoming more complex.<\/p>\n","protected":false},"author":43518,"featured_media":1136384,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","msr-author-ordering":[{"type":"user_nicename","value":"Amber Hoak","user_id":"37992"},{"type":"user_nicename","value":"Scott Counts","user_id":"31471"},{"type":"user_nicename","value":"Kate Lytvynets","user_id":"38073"},{"type":"user_nicename","value":"David Tittsworth","user_id":"38064"},{"type":"user_nicename","value":"Siddharth Suri","user_id":"33766"},{"type":"user_nicename","value":"Nirupama Chandrasekaran","user_id":"33115"},{"type":"user_nicename","value":"Ben Cutler","user_id":"31188"},{"type":"user_nicename","value":"Weiwei Yang","user_id":"40138"}],"msr_hide_image_in_river":null,"footnotes":""},"categories":[1],"tags":[],"research-area":[13556,13554],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[269148,243984,269142],"msr-impact-theme":[],"msr-promo-type":[],"msr-podcast-series":[],"class_list":["post-1136235","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-research-blog","msr-research-area-artificial-intelligence","msr-research-area-human-computer-interaction","msr-locale-en_us","msr-post-option-approved-for-river","msr-post-option-blog-homepage-featured","msr-post-option-include-in-river"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"","podcast_episode":"","msr_research_lab":[199565],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[],"related-projects":[1119417],"related-events":[],"related-researchers":[{"type":"user_nicename","value":"Amber Hoak","user_id":37992,"display_name":"Amber Hoak","author_link":"<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/amhoak\/?lang=zh-cn\" aria-label=\"\u8bbf\u95ee\u4e2a\u4eba\u8d44\u6599\u9875\u9762\u4e86\u89e3Amber Hoak\">Amber Hoak<\/a>","is_active":false,"last_first":"Hoak, Amber","people_section":0,"alias":"amhoak"},{"type":"user_nicename","value":"Scott Counts","user_id":31471,"display_name":"Scott Counts","author_link":"<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/counts\/?lang=zh-cn\" aria-label=\"\u8bbf\u95ee\u4e2a\u4eba\u8d44\u6599\u9875\u9762\u4e86\u89e3Scott Counts\">Scott Counts<\/a>","is_active":false,"last_first":"Counts, Scott","people_section":0,"alias":"counts"},{"type":"user_nicename","value":"Kate Lytvynets","user_id":38073,"display_name":"Kate Lytvynets","author_link":"<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/kalytv\/?lang=zh-cn\" aria-label=\"\u8bbf\u95ee\u4e2a\u4eba\u8d44\u6599\u9875\u9762\u4e86\u89e3Kate Lytvynets\">Kate Lytvynets<\/a>","is_active":false,"last_first":"Lytvynets, Kate","people_section":0,"alias":"kalytv"},{"type":"user_nicename","value":"David Tittsworth","user_id":38064,"display_name":"David Tittsworth","author_link":"<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/datittsw\/?lang=zh-cn\" aria-label=\"\u8bbf\u95ee\u4e2a\u4eba\u8d44\u6599\u9875\u9762\u4e86\u89e3David Tittsworth\">David Tittsworth<\/a>","is_active":false,"last_first":"Tittsworth, David","people_section":0,"alias":"datittsw"},{"type":"user_nicename","value":"Siddharth Suri","user_id":33766,"display_name":"Siddharth Suri","author_link":"<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/suri\/?lang=zh-cn\" aria-label=\"\u8bbf\u95ee\u4e2a\u4eba\u8d44\u6599\u9875\u9762\u4e86\u89e3Siddharth Suri\">Siddharth Suri<\/a>","is_active":false,"last_first":"Suri, Siddharth","people_section":0,"alias":"suri"},{"type":"user_nicename","value":"Nirupama Chandrasekaran","user_id":33115,"display_name":"Nirupama Chandrasekaran","author_link":"<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/niruc\/?lang=zh-cn\" aria-label=\"\u8bbf\u95ee\u4e2a\u4eba\u8d44\u6599\u9875\u9762\u4e86\u89e3Nirupama Chandrasekaran\">Nirupama Chandrasekaran<\/a>","is_active":false,"last_first":"Chandrasekaran, Nirupama","people_section":0,"alias":"niruc"},{"type":"user_nicename","value":"Weiwei Yang","user_id":40138,"display_name":"Weiwei Yang","author_link":"<a href=\"https:\/\/www.microsoft.com\/en-us\/research\/people\/weiwya\/?lang=zh-cn\" aria-label=\"\u8bbf\u95ee\u4e2a\u4eba\u8d44\u6599\u9875\u9762\u4e86\u89e3Weiwei Yang\">Weiwei Yang<\/a>","is_active":false,"last_first":"Yang, Weiwei","people_section":0,"alias":"weiwya"}],"msr_type":"Post","featured_image_thumbnail":"<img width=\"960\" height=\"540\" src=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Semantic-Telemetry-2-BlogHeroFeature-1400x788-1-960x540.jpg\" class=\"img-object-cover\" alt=\"The image features four white icons on a gradient background that transitions from blue on the left to green on the right. The first icon is a network or molecule structure with interconnected nodes. The second icon is a light bulb, symbolizing an idea or innovation. The third icon is a checklist with three items and checkmarks next to each item. The fourth icon consists of two overlapping speech bubbles, representing communication or conversation.\" decoding=\"async\" loading=\"lazy\" srcset=\"https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Semantic-Telemetry-2-BlogHeroFeature-1400x788-1-960x540.jpg 960w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Semantic-Telemetry-2-BlogHeroFeature-1400x788-1-300x169.jpg 300w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Semantic-Telemetry-2-BlogHeroFeature-1400x788-1-1024x576.jpg 1024w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Semantic-Telemetry-2-BlogHeroFeature-1400x788-1-768x432.jpg 768w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Semantic-Telemetry-2-BlogHeroFeature-1400x788-1-1536x865.jpg 1536w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Semantic-Telemetry-2-BlogHeroFeature-1400x788-1-2048x1153.jpg 2048w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Semantic-Telemetry-2-BlogHeroFeature-1400x788-1-1066x600.jpg 1066w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Semantic-Telemetry-2-BlogHeroFeature-1400x788-1-655x368.jpg 655w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Semantic-Telemetry-2-BlogHeroFeature-1400x788-1-240x135.jpg 240w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Semantic-Telemetry-2-BlogHeroFeature-1400x788-1-640x360.jpg 640w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Semantic-Telemetry-2-BlogHeroFeature-1400x788-1-1280x720.jpg 1280w, https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2025\/04\/Semantic-Telemetry-2-BlogHeroFeature-1400x788-1-1920x1080.jpg 1920w\" sizes=\"auto, (max-width: 960px) 100vw, 960px\" \/>","byline":"","formattedDate":"April 14, 2025","formattedExcerpt":"Semantic Telemetry Project data show that people who use AI for more professional and complex tasks are more likely to keep using the tool and to use it more often. Novice AI users engage in simpler tasks, but their usage is becoming more complex.","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/1136235","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/users\/43518"}],"replies":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/comments?post=1136235"}],"version-history":[{"count":22,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/1136235\/revisions"}],"predecessor-version":[{"id":1138182,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/1136235\/revisions\/1138182"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/1136384"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=1136235"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/categories?post=1136235"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/tags?post=1136235"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=1136235"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=1136235"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=1136235"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=1136235"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=1136235"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=1136235"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=1136235"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=1136235"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}