{"id":1004586,"date":"2024-03-07T09:00:00","date_gmt":"2024-03-07T17:00:00","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?p=1004586"},"modified":"2024-03-05T14:17:27","modified_gmt":"2024-03-05T22:17:27","slug":"improving-llm-understanding-of-structured-data-and-exploring-advanced-prompting-methods","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/improving-llm-understanding-of-structured-data-and-exploring-advanced-prompting-methods\/","title":{"rendered":"Improving LLM understanding of structured data and exploring advanced prompting methods"},"content":{"rendered":"\n

This research paper was presented at the <\/strong><\/em>17th ACM International Conference on Web Search and Data Mining (opens in new tab)<\/span><\/a><\/em><\/strong> (WSDM 2024), the premier conference on web-inspired research on search and data mining.<\/strong><\/em><\/p>\n\n\n\n

\"WSDM<\/figure>\n\n\n\n

In today\u2019s data-driven landscape, tables are indispensable for organizing and presenting information, particularly text. They streamline repetitive content, enhance data manageability, enable easier data analysis, and improve machine processing capabilities. Meanwhile, large language models (LLMs) are advancing in their ability to tackle challenges associated with natural language, but the degree to which they understand tables included in their prompts remains an open question. Our research aims to explore this question and improve how LLMs use and work with table-based data.<\/p>\n\n\n\n

Our paper, \u201cTable Meets LLM: Can Large Language Models Understand Structured Table Data? A Benchmark and Empirical Study<\/a>,\u201d presented at WSDM 2024 (opens in new tab)<\/span><\/a>, investigates what kinds of prompts most effectively enable LLMs to understand tables; how much LLMs inherently detect structured data; and how LLMs\u2019 existing knowledge can be harnessed to improve this understanding. We also analyze the complex trade-off among multiple combinations of input designs and overall performance.<\/p>\n\n\n\n

To address these questions, we propose a new benchmark called Structural Understanding Capabilities (SUC), shown in Figure 1 (a), which focuses on specific tasks to assess LLMs\u2019 ability to understand structured data in tables and compare different types of prompts. We conducted a series of experiments using different prompt designs. Our findings, detailed in the paper<\/a>, evaluate how each design enhances LLMs\u2019 ability to work with tables. <\/p>\n\n\n\n

\"The<\/a>
Figure 1. The SUC benchmark and prompt designs for evaluation.<\/figcaption><\/figure>\n\n\n\n

Insights and findings using the SUC benchmark<\/h2>\n\n\n\n

Based on humans’ perception of tables, we developed tasks to evaluate how LLMs understand them. We conducted evaluations on GPT-3.5 and GPT-4 and discovered that the results depended on certain input factors, such as table format, content order, and partition marks. The findings, detailed in Tables 1 and 2, reveal some notable and unexpected findings:<\/p>\n\n\n\n