{"id":1153693,"date":"2025-11-12T04:00:20","date_gmt":"2025-11-12T12:00:20","guid":{"rendered":"https:\/\/www.microsoft.com\/en-us\/research\/?p=1153693"},"modified":"2025-11-12T10:17:41","modified_gmt":"2025-11-12T18:17:41","slug":"mmctagent-enabling-multimodal-reasoning-over-large-video-and-image-collections","status":"publish","type":"post","link":"https:\/\/www.microsoft.com\/en-us\/research\/blog\/mmctagent-enabling-multimodal-reasoning-over-large-video-and-image-collections\/","title":{"rendered":"MMCTAgent: Enabling multimodal reasoning over large video and image collections"},"content":{"rendered":"\n
\"Three<\/figure>\n\n\n\n

Modern multimodal AI models can recognize objects, describe scenes, and answer questions about images and short video clips, but they struggle with long-form and large-scale visual data, where real-world reasoning requires moving beyond object recognition and short-clip analysis.<\/p>\n\n\n\n

Real-world reasoning increasingly involves analyzing long-form video content, where context spans minutes or hours, far beyond the context limits of most models.\u202fIt also entails querying across massive multimodal libraries of videos, images, and transcripts, where finding and integrating relevant evidence requires more than retrieval\u2014it requires strategic reasoning. Existing models typically perform single-pass inference, producing one-shot answers. This limits their ability to handle tasks that require temporal reasoning, cross-modal grounding, and iterative refinement.<\/p>\n\n\n\n

MMCTAgent<\/h2>\n\n\n\n

To meet these challenges, we developed the Multi-modal Critical Thinking Agent<\/a>, or MMCTAgent, for structured reasoning over long-form video and image data, available on GitHub (opens in new tab)<\/span><\/a> and featured on Azure AI Foundry Labs (opens in new tab)<\/span><\/a>.<\/p>\n\n\n\n

Built on AutoGen<\/a>, Microsoft\u2019s open-source multi-agent system, MMCTAgent provides multimodal question-answering with a Planner\u2013Critic architecture. This design enables planning, reflection, and tool-based reasoning, bridging perception and deliberation in multimodal tasks. It links language, vision, and temporal understanding, transforming static multimodal tasks into dynamic reasoning workflows.  <\/p>\n\n\n\n

Unlike conventional models that produce one-shot answers, MMCTAgent has modality-specific agents, including ImageAgent and VideoAgent, which include tools like get_relevant_query_frames() or object_detection-tool(). These agents perform deliberate, iterative reasoning\u2014selecting the right tools for each modality, evaluating intermediate results, and refining conclusions through a Critic loop. This enables MMCTAgent to analyze complex queries across long videos and large image libraries with explainability, extensibility, and scalability.<\/p>\n\n\n\n

\n
MMCTAgent on Azure AI Foundry Labs<\/a><\/div>\n<\/div>\n\n\n\n\t
\n\t\t\n\n\t\t

\n\t\tvideo series<\/span>\n\t<\/p>\n\t\n\t

\n\t\t\t\t\t\t
\n\t\t\t\t\n\t\t\t\t\t\"On\n\t\t\t\t<\/a>\n\t\t\t<\/div>\n\t\t\t\n\t\t\t
\n\n\t\t\t\t\t\t\t\t\t

On Second Thought<\/h2>\n\t\t\t\t\n\t\t\t\t\t\t\t\t

A video series with Sinead Bovell built around the questions everyone\u2019s asking about AI. With expert voices from across Microsoft, we break down the tension and promise of this rapidly changing technology, exploring what\u2019s evolving and what\u2019s possible.<\/p>\n\t\t\t\t\n\t\t\t\t\t\t\t\t

\n\t\t\t\t\t
\n\t\t\t\t\t\t\n\t\t\t\t\t\t\tExplore the series\t\t\t\t\t\t<\/a>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t\t<\/div>\n\t<\/div>\n\t<\/div>\n\t\n\n\n

How MMCTAgent works<\/h2>\n\n\n\n

MMCTAgent integrates two coordinated agents, Planner and Critic, orchestrated through AutoGen. The Planner agent decomposes a user query, identifies the appropriate reasoning tools, performs multimodal operations, and drafts a preliminary answer. The Critic agent reviews the Planner\u2019s reasoning chain, validates evidence alignment, and refines or revises the response for factual accuracy and consistency.<\/p>\n\n\n\n

This iterative reasoning loop enables MMCTAgent to improve its answers through structured self-evaluation\u2014bringing reflection into AI reasoning. A key strength of MMCTAgent lies in its modular extensibility. Developers can easily integrate new, domain-specific tools\u2014such as medical image analyzers, industrial inspection models, or specialized retrieval modules\u2014by adding them to ImageQnATools or VideoQnATools. This design makes MMCTAgent adaptable across domains.<\/p>\n\n\n\n

VideoAgent: From ingestion to long-form multimodal reasoning<\/h3>\n\n\n\n
\"MMCTAgent\u2019s
Figure 1. MMCTAgent\u2019s Planner\u2013Critic architecture enables multimodal reasoning over long-form video through structured ingestion, retrieval, and iterative feedback<\/figcaption><\/figure>\n\n\n\n

The VideoAgent extends this architecture to long-form video reasoning. It operates in two connected phases: library creation (ingestion) and query-time reasoning.<\/p>\n\n\n\n

\n\n\n\n

Phase 1 \u2013 Video ingestion and library creation<\/h4>\n\n\n\n

Before reasoning, long-form videos undergo an ingestion pipeline that aligns multimodal information for retrieval and understanding:<\/p>\n\n\n\n

    \n
  1. Transcription <\/strong>and translation<\/strong>: Converts audio to text and, if multilingual, translates transcripts into a consistent language <\/li>\n\n\n\n
  2. Key-frame identification<\/strong>: Extracts representative frames marking major visual or scene changes<\/li>\n\n\n\n
  3. Semantic chunking <\/strong>and chapter generation<\/strong>: Combines transcript segments and visual summaries into coherent, semantically segmented chapters with associated key frames. Inspired by Microsoft\u2019s Deep Video Discovery agentic search tool<\/a>, this step also extracts detailed descriptions of objects, on-screen text, and characters present within each video segment, integrating these insights directly into the corresponding chapters. <\/li>\n\n\n\n
  4. Multimodal embedding creation<\/strong>: Generates image embeddings for key frames, linking them to their corresponding transcript and chapter data<\/li>\n<\/ol>\n\n\n\n

    All structured metadata, including transcripts, visual summaries, chapters, and embeddings, is indexed in the Multimodal Knowledgebase using Azure AI Search (opens in new tab)<\/span><\/a>, which forms the foundation for scalable semantic retrieval and downstream reasoning.<\/p>\n\n\n\n

    \n\n\n\n

    Phase 2 \u2013 Video question answering and reasoning<\/h4>\n\n\n\n

    When a user submits a query, the VideoAgent retrieves, analyzes, and reasons across the indexed video content using specialized planner and critic tools.<\/p>\n\n\n\n

    Planner tools<\/h5>\n\n\n\n