Every IT team knows the feeling: A project is on track, and then testing becomes the wall everything stacks up against. Slow cycles, fragmented tools, and manual validation are some of the most persistent bottlenecks in enterprise technology.
AI tools for testing are everywhere, but for most organizations, they’re still experiments, not infrastructure.
When our Commerce Platforms team, which owns the company’s core commerce and financial platform services, found itself staring down a critical SAP migration with a hard deadline, they decided it was time to try a different approach.
They wanted to know if enterprise-grade AI testing was actually achievable.
A growing challenge
Behind the scenes, we run much of our global business on SAP. From finance and supply chain management to HR and procurement, SAP serves as a critical enterprise platform across the company.
Testing across our 40-plus SAP service lines had long relied on third-party tools. Our licensed infrastructure had finite capacity and required manual test-case authoring. And the vendor we were working with wasn’t able to move as quickly as we wanted.
As a result, costs compounded in every direction. Reliance on third-party testing tools to automate SAP regression testing came with high licensing costs, infrastructure requirements, and operational complexity. Together, those tools cost us about $500,000 a year to maintain. With only 80 third-party tool licenses spread across more than 40 service lines, teams were perpetually competing for capacity.
Additionally, security vulnerabilities and compliance findings in third-party testing tools became an ongoing liability. Vendor remediation timelines stretched to 9 or 10 months, far beyond our requirements.
Then came the deadline that forced us to act.
One of the first major tests came from the Global Trade Services (GTS) system, which handles customs and invoice generation processes required to move products across international borders. As part of our broader SAP modernization effort, GTS had to migrate from SAP ERP Central Component to SAP Business Suite 4 HANA before a December 2025 deadline or shipments would stop. The team had 10 months. The traditional testing approach wouldn’t get us there.
The time had come to explore AI as a solution for a testing model that was becoming unsustainable. With an ambitious goal, and the GTS team’s blessing, we developed Enterprise Test Platform, an AI-driven testing solution.
Building with an incremental approach
Our team started narrow. Rather than replacing their third-party testing platform overnight, they focused first on test-case authoring. It was a practical starting point: Teams already had years of business process documentation, but turning that knowledge into structured, executable test cases still required significant manual effort.
“We realized that enterprise teams already had the documentation AI needs. Once we saw that AI could convert that knowledge into reliable, structured test cases, it changed the trajectory of the entire project.”
Neeraj Tripathi, principal program manager, Commerce Platforms team
Process documentation, API references, business requirement documents, and functional flows were already there, scattered across systems. The missing piece was converting that existing knowledge into structured, executable test assets.
“We realized that enterprise teams already had the documentation AI needs,” says Neeraj Tripathi, a principal program manager on the Commerce Platforms team. “Once we saw that AI could convert that knowledge into reliable, structured test cases, it changed the trajectory of the entire project.”
Early results exceeded expectations. That momentum gave the team confidence to expand beyond test-case authoring to eventually encompass execution, validation, orchestration, and reporting, consolidating fragmented testing capabilities into a unified first-party platform.
Making a dramatic impact
The GTS migration became the proof point. The team ran three full mock drills using the Enterprise Test Platform throughout, and the numbers tell the story of a team learning in real time.
“The migration of delta was completed in 11 minutes. We have set an industry benchmark with this project.”
Priyanka Alva, program manager lead, Global Trade Services
In the first mock drill, the system cutover alone took 144 hours. In the second, it was 36 hours. In the third, 11 hours. Finally, the GTS system went live on SAP S4 Migration on Azure in six hours, with SAP Support flying a team in from overseas to be on site.
“The migration of delta was completed in 11 minutes,” Alva says. “We have set an industry benchmark with this project.”
The in-house migration tool and Enterprise Test Platform contributed directly. Weekly regression testing that previously consumed three full days dropped to under an hour. Alva’s team achieved 57% automation across the full migration effort, a result that came during the first pilot, while the tool was still being actively developed in parallel. The go-live produced zero post-launch defects.
Since then, the Enterprise Test Platform has been used for other service lines. The broader organizational results are just as striking:
- Across service lines that have adopted the platform, teams are seeing 80% efficiency gains in end-to-end testing cycles. More than 10,000 test cases now execute in 10 to 12 minutes.
- Hundreds of thousands of dollars spent annually on third-party licensing and infrastructure are coming off the books as each service line retires its legacy tools.
- The security compliance problem that once left findings open for the better part of a year is resolved; first-party tooling means we now own the remediation timeline.
SAP has since cited the GTS migration at TechEd and Sapphire as a model for S4 migrations at scale.
“We’ve achieved something the industry has been struggling with,” Tripathi says. “Our journey is a case study for other organizations working toward their S4 migrations. That story is now being told by SAP to their customers.”
Solving the challenge AI skeptics raise first
Generative AI is probabilistic by nature. For compliance-sensitive enterprise testing, that’s a disqualifying concern. The Commerce Platforms team knew it, and they built their answer into the architecture.
When a tester prompts the agent to generate test cases, the agent draws on the service line’s knowledge base and proposes test steps for human review. Nothing is committed until a person approves. Once approved, the agent produces a test execution context, or TC JSON file: a lightweight, human-readable script that locks in every step, API mapping, input payload, expected output, and validation assertion for that test case.
“The nature of AI is probabilistic, but testing requires deterministic responses,” Tripathi says. “Every time that test case runs, it runs the same way.”
“Each time some action is done by the AI, another agent comes and challenges it. The agent probes more questions and makes sure the output is accurate.”
Ram Mohan Reddy Gudupalli, software engineering manager, Commerce Platforms team
That handles execution. But what about authoring itself?
To solve that, the team added a second layer: A dedicated “policing” agent built into the multi-agent architecture that re-interrogates every output before it’s finalized.
“Each time some action is done by the AI, another agent comes and challenges it,” says Ram Mohan Reddy Gudupalli, the software engineering manager who led the build. “The agent probes more questions and makes sure the output is accurate. The real innovation wasn’t adding AI to testing, it was making AI accountable through deterministic execution, independent validation, and enterprise governance.”
The TC JSON makes execution deterministic and auditable. The policing agent makes the authoring step trustworthy. Together, they give a probabilistic technology the governance profile that enterprise requires.
The Enterprise Test Platform connects existing business knowledge to AI-assisted test creation, deterministic execution, traceable results, and automated validation/insights:
- Knowledge base: Business process docs, wikis, APIs, and test artifacts become the source of truth.
- Test case authoring: AI proposes test scenarios and creates Azure DevOps test cases for human review.
- Test process engine: Creates a Test Execution Context file that standardizes execution logic, inputs, and validations.
- Deterministic execution: Runs approved test cases at scale in Azure DevOps through controlled, repeatable execution.
- Results and traceability: Captures outcomes, evidence, and summaries back in Azure DevOps for end-to-end traceability.
- Validation, analysis, and insights: Validates results across enterprise systems, identifies anomalies, and provides actionable testing insights.
At execution time, an orchestration engine fans test cases out for parallel execution across multiple simultaneous instances. Results flow back to Azure DevOps, where the agent analyzes outputs, flags mismatches, and automatically creates bug work items with AI-recommended fixes.
“We could have chosen a safer pilot. But if GTS succeeded, it would become a much bigger story. They came through with flying colors, and now they speak on our behalf.”
Adrivit Bhattacharya, program manager lead, Commerce Platforms team
Adoption is its own engineering problem
Building the platform was the easier part of the project. Adrivit Bhattacharya, program manager lead for the platform team, chose GTS as the first pilot not because it was the safest bet, but because it was the highest-stakes one.
“We could have chosen a safer pilot,” Bhattacharya says. “But if GTS succeeded, it would become a much bigger story. They came through with flying colors, and now they speak on our behalf.”
Before claiming any efficiency gains, our team ran the same test cases manually to establish a real baseline, then had service lines review and sign off on the comparison.
As of now, seven service lines are fully onboarded, 11 are in progress, and more than 15 are in the pipeline.
“Building solutions is easy; adoption is where the challenges are because there is no one size fits all. If a service line only needs one of our solutions, they should be able to leverage it without having to connect to everything we offer.”
Umamaheswar Charabudla, engineering group manager, Commerce Platforms team
Umamaheswar Charabudla, the engineering group manager on the team, made the platform modular from the start, so teams can adopt one capability at a time—authoring, execution, or validation—without committing to the full stack up front.
“Building solutions is easy; adoption is where the challenges are because there is no one size fits all,” Charabudla says. “If a service line only needs one of our solutions, they should be able to leverage it without having to connect to everything we offer. There should be no ‘all or nothing.’”
What’s next—and why it matters beyond Microsoft
Our team built this solution under pressure, proved it on one of the most business-critical systems in the company, and is now fielding interest from enterprises around the world facing identical problems. The tool that started as a scrappy experiment to save a migration timeline has become a case study that SAP is presenting to its own customer base.
For organizations navigating the same pressures, the Commerce Platforms team playbook is open.
Key takeaways
If your organization is navigating similar testing challenges, here is what the Commerce Platforms team learned:
- Start with your knowledge base, not your tools. Most organizations already have the documentation that AI needs: process specs, API references, business requirements. The opportunity is connecting that existing knowledge to AI orchestration. If it’s scattered or incomplete, invest in capturing it first.
- Start with one phase. The team began with test case authoring, proved the value, and expanded from there. Trying to automate the full lifecycle on day one increases complexity and risk without a foundation to build on.
- Make your AI deterministic. Generative AI is inherently probabilistic; enterprise testing cannot be. A test execution context file that locks in inputs, expected outputs, and validation logic is what makes AI-generated test cases repeatable, auditable, and trustworthy for compliance environments.
- Keep humans in the loop at every stage. The agent proposes; a person approves. Speed and scale don’t require removing human judgment; they require applying it at the right moments.
- Choose your first customer carefully. Your pilot partner sets the story you tell every team that follows. Pick a team facing a real, urgent problem. If they succeed, they become your most credible advocates.
- Earn your numbers. Before claiming efficiency gains, run the same test cases the old way to establish a real baseline. Have your customers agree to the comparison in writing. Fuzzy numbers don’t survive the trip to senior leadership.
- Build a pluggable architecture. The AI landscape changes fast. Design your system so models, tools, and execution engines can be swapped without rebuilding everything. What’s best today may not be the best option in six months.
Try it out
Related links
- See how Microsoft and SAP are accelerating migrations with Azure.
- Explore the Model Context Protocol (MCP) documentation to learn how to build MCP-integrated solutions.
- Discover how Microsoft Foundry provides governance, observability, and evaluation capabilities for enterprise AI.
- Learn about our Secure Future Initiative and how security is built into first-party AI development at Microsoft.
