{"id":973395,"date":"2023-10-12T09:00:00","date_gmt":"2023-10-12T16:00:00","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/microsoft-at-vl-hcc-2023-focus-on-co-audit-tools-for-spreadsheets\/"},"modified":"2025-06-19T07:11:08","modified_gmt":"2025-06-19T14:11:08","slug":"microsoft-at-vl-hcc-2023-focus-on-co-audit-tools-for-spreadsheets","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/microsoft-at-vl-hcc-2023-focus-on-co-audit-tools-for-spreadsheets\/","title":{"rendered":"Microsoft at VL\/HCC 2023: Focus on co-audit tools for spreadsheets"},"content":{"rendered":"\n

These research papers were presented at the <\/em><\/strong>IEEE Symposium on Visual Languages and Human-Centric Computing<\/em><\/strong> (opens in new tab)<\/span><\/a> (VL\/HCC 2023), a premier forum for design, theory, and application of computing technologies for programming, modelling, and communication.<\/em><\/strong><\/p>\n\n\n\n

\"Microsoft<\/figure>\n\n\n\n

Large language models (LLMs) have revolutionized the way novice programmers and everyday computer users tap into the capabilities of natural language for programming. Among the tools used in this context, spreadsheets stand out as the preferred choice. The integration of LLMs into spreadsheets promises to substantially enhance their functionality and the user experience. At the same time, it’s well known that spreadsheet users commonly though inadvertently introduce errors (opens in new tab)<\/span><\/a>, and this can carry significant risks. For example, in 2010, a spreadsheet used in a Harvard economic analysis (opens in new tab)<\/span><\/a> to inform austerity measures imposed on Greece was discovered to contain multiple errors (opens in new tab)<\/span><\/a>.<\/p>\n\n\n\n

Microsoft is actively pursuing (opens in new tab)<\/span><\/a> research focused on developing co-auditing tools and techniques, with an initial emphasis on spreadsheets. These tools are designed to help users verify the results generated by LLMs. At VL\/HCC 2023 (opens in new tab)<\/span><\/a>, we introduce two new spreadsheet tools, ColDeco and FxD, specifically built to help users thoroughly examine and debug their programs within spreadsheets. Additionally, it is worth mentioning that the paper on FxD was awarded the Honorable Mention (opens in new tab)<\/span><\/a>.<\/p>\n\n\n\n

ColDeco: An end-user inspection tool<\/h3>\n\n\n\n

Working with tables in spreadsheets is a common task, and the ability to add a calculated column can be incredibly useful. A calculated column not only adds information but also facilitates tasks like filtering and sorting. Generative AI can enable users to create sophisticated calculated columns in tables. However, verification of AI-generated code in this scenario is crucial because AI can misinterpret the user\u2019s intent or overlook important data. <\/p>\n\n\n\n

In our paper, \u201cColDeco: An End User Spreadsheet Inspection Tool for AI-Generated Code<\/a>,\u201d we introduce ColDeco, a no-code inspection tool for calculated columns. ColDeco uses helper columns and row grouping to help users understand how an AI-generated column works and locate any errors. <\/p>\n\n\n\n

To describe how ColDeco works, we\u2019ll use an example table containing people\u2019s first, middle, and last names in separate columns. Our user asks the system to \u201ccreate a column called \u2018Abbreviation\u2019 that takes the first letter of each part of the name.\u201d In this example, there\u2019s an error in the generated code that fails to handle rows with no middle names, causing some Abbreviation cells to be empty.  <\/p>\n\n\n\n

First, the model generates a program that computes an abbreviation for each row and adds it to the new Abbreviation column. ColDeco\u2019s interface automatically opens as a side panel, as shown in Figure 1. <\/p>\n\n\n\n

The Inspect Columns view displays any generated columns, accompanied by a natural language description of the generated code. The Inspect Rows view displays a subset of the table, organized by behavior. The Row Inspection view uses dataflow analysis to group rows, highlighting key distinct execution behaviors. In our example, this view quickly draws the user\u2019s attention to the two rows that fail to calculate an abbreviation.<\/p>\n\n\n\n

\"Two
Figure 1. The initial view of the ColDeco side panel. An Abbreviation program is generated by the AI and added to the table as a new column. The Inspect Columns view (1a) shows the column generated by the AI, including a description of how the code works. The Inspect Rows view (1b) groups rows into different behaviors, indicating that there are errors in two rows.<\/figcaption><\/figure>\n\n\n\n

If our user wants to investigate an error, they can expand a generated column into multiple helper columns, illustrated in Figure 2. These helper columns are visible in both the table (2a) and the side panel (2b), and they show the intermediate values. The user can now see that the missing abbreviations are caused by an error that occurred when the system tried to take the first and middle initials.<\/p>\n\n\n\n

\"Two
Figure 2. The ColDeco side panel after a user expands the Abbreviation column into two additional helper columns. Each additional column has a description.<\/figcaption><\/figure>\n\n\n\n\t
\n\t\t\n\n\t\t

\n\t\tSpotlight: Event Series<\/span>\n\t<\/p>\n\t\n\t

\n\t\t\t\t\t\t
\n\t\t\t\t\n\t\t\t\t\t\"Research\n\t\t\t\t<\/a>\n\t\t\t<\/div>\n\t\t\t\n\t\t\t
\n\n\t\t\t\t\t\t\t\t\t

Microsoft Research Forum<\/h2>\n\t\t\t\t\n\t\t\t\t\t\t\t\t

Join us for a continuous exchange of ideas about research in the era of general AI. Watch the latest episodes on demand.<\/p>\n\t\t\t\t\n\t\t\t\t\t\t\t\t

\n\t\t\t\t\t
\n\t\t\t\t\t\t\n\t\t\t\t\t\t\tWatch on-demand\t\t\t\t\t\t<\/a>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t\t<\/div>\n\t<\/div>\n\t<\/div>\n\t\n\n\n

FxD: A functional debugger <\/h3>\n\n\n\n

Not every spreadsheet task involves generating a new table column. Moreover, many users are already well acquainted with spreadsheet formulas. This brings us to our second tool, a spreadsheet formula debugger, introduced in the paper, \u201cFxD: a functional debugger for dysfunctional spreadsheets<\/a>.\u201d <\/p>\n\n\n\n

We employed a user-centered approach when designing FxD, extensively reviewing existing literature on functional programming debuggers. This informed the four key features we implemented into FxD: <\/p>\n\n\n\n

Live debugging<\/strong>. FxD dynamically updates as a user edits a formula, allowing for quick formula modification and exploration (Figure 3, image 1).<\/p>\n\n\n\n

Hybrid formula tracing<\/strong>. <\/strong>The debugger combines step-based evaluation (Figure 3, image 1) with tree-based derivations (Figure 3, image 3) to provide a step-by-step breakdown of the formula. Substeps are hidden behind expandable cards to prevent user overload.  <\/p>\n\n\n\n

Subformula coloring<\/strong>. <\/strong>Color coding highlights changes in a formula as FxD evaluates it. This facilitates the tracking of these updates when a user hovers over a step (Figure 3, images 2 and 4). <\/p>\n\n\n\n

Information inspector<\/strong>. <\/strong>Context-aware tooltips improve the user experience. One example is table previews when a user hovers over ranges in functions like VLOOKUP. These tooltips offer insights into the range, surrounding context, and the lookup column used by the containing function (Figure 3, image 3).<\/p>\n\n\n\n

\"Four
Figure 3. The FxD debugger. Image 1 shows the edited formula and evaluation steps. The steps update as a user edits the formula. Image 2 shows subformula coloring, which highlights a subformula and its value upon hovering. Image 3 shows an information inspector that previews the range referenced in a formula. Image 4 shows the concurrent evaluation of multiple subformulas. When the user hovers over a value, the corresponding subformula is underlined.<\/figcaption><\/figure>\n\n\n\n

Growing importance of AI code verification <\/h3>\n\n\n\n

As the complexity of AI-generated code rises, the need for tools to verify accuracy becomes increasingly critical. In response, we developed these two co-audit tools tailored to spreadsheets. Moving forward, a key consideration lies in managing the complexity of these tools. Our vision is that debugging tools will become infused with generative AI to assist users in both generating and verifying workflows. <\/p>\n\n\n\n

Review our paper<\/a> on co-auditing in general to learn more.<\/p>\n","protected":false},"excerpt":{"rendered":"

These research papers were presented at the IEEE Symposium on Visual Languages and Human-Centric Computing (opens in new tab) (VL\/HCC 2023), a premier forum for design, theory, and application of computing technologies for programming, modelling, and communication. Large language models (LLMs) have revolutionized the way novice programmers and everyday computer users tap into the capabilities […]<\/p>\n","protected":false},"author":42735,"featured_media":974691,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","msr-author-ordering":null,"msr_hide_image_in_river":null,"footnotes":""},"categories":[204945],"tags":[],"research-area":[13556,13554],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[],"msr-impact-theme":[264846],"msr-promo-type":[],"msr-podcast-series":[],"class_list":["post-973395","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-conferences","msr-research-area-artificial-intelligence","msr-research-area-human-computer-interaction","msr-locale-en_us"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"","podcast_episode":"","msr_research_lab":[],"msr_impact_theme":["Computing foundations"],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[1142579],"related-projects":[511097],"related-events":[],"related-researchers":[{"type":"user_nicename","value":"Jack Williams","user_id":40156,"display_name":"Jack Williams","author_link":"Jack Williams<\/a>","is_active":false,"last_first":"Williams, Jack","people_section":0,"alias":"johnwilliams"},{"type":"guest","value":"kasra-ferdowsi","user_id":"973404","display_name":"Kasra Ferdowsi","author_link":"Kasra Ferdowsi<\/a>","is_active":true,"last_first":"Ferdowsi, Kasra","people_section":0,"alias":"kasra-ferdowsi"}],"msr_type":"Post","featured_image_thumbnail":"\"Microsoft","byline":"Jack Williams<\/a>, Ian Drosos, and Kasra Ferdowsi<\/a>","formattedDate":"October 12, 2023","formattedExcerpt":"These research papers were presented at the IEEE Symposium on Visual Languages and Human-Centric Computing (opens in new tab) (VL\/HCC 2023), a premier forum for design, theory, and application of computing technologies for programming, modelling, and communication. Large language models (LLMs) have revolutionized the way…","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/973395","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/users\/42735"}],"replies":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/comments?post=973395"}],"version-history":[{"count":31,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/973395\/revisions"}],"predecessor-version":[{"id":1142609,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/posts\/973395\/revisions\/1142609"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media\/974691"}],"wp:attachment":[{"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/media?parent=973395"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/categories?post=973395"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/tags?post=973395"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=973395"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=973395"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=973395"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=973395"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=973395"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=973395"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=973395"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/www.microsoft.com\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=973395"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}