Consulting services and software company Notion Solutions wanted to improve the quality of its software products, but was having difficulty ensuring adequate test coverage across different client and server operating system versions. The company deployed Microsoft Visual Studio 2010 Ultimate to all developer desktops; upgraded to Visual Studio Team Foundation Server 2010 for application lifecycle management; deployed Visual Studio Lab Management 2010 to virtualize its test environment; and adopted Microsoft Test Manager—included in the Ultimate edition—to manage test cases and capture test results. With its new test environment, tools, and processes, Notion Solutions is benefiting from improved software quality, increased productivity, new developer capabilities, and simplified licensing of the software required for development and testing.
Founded in 2004, Microsoft Gold Certified Partner Notion Solutions provides consulting services that help software development organizations get the most out of the Microsoft Visual Studio development system. Services include migration planning workshops, “Jumpstart” consulting engagements, training courses, and more. Notion Solutions also develops and sells Notion Tools, a suite of extensions for Visual Studio that helps teams better plan and manage the software development process.
As soon as the initial prerelease version of Visual Studio 2010 became available, Notion Solutions began working to adopt it for the company’s own product development efforts—both as a means of improving internal product development and to better understand how to help customers do the same. In determining where to start, the company chose to focus on one of its own problem areas: software testing and quality.
“Our software requires lots of testing on various combinations of server and client operating system versions,” says Chris Menegay, Vice President of Consulting at Notion Solutions. “We initially thought we had good test coverage across all the different operating systems we need to support, but the number of customer support incidents told us otherwise.”
Given the desire to improve the testing process, Notion Solutions came up with three objectives. “First, we wanted to virtualize our test configurations so that we could manage and deploy them more efficiently,” says Menegay. “Second, we wanted to be able to easily deploy new builds into those environments. Third, we needed an easy way for people to jump right in and begin testing. We have a very dynamic team, with all developers stepping in and out of testing roles between client engagements.”Solution
To improve its software testing processes, Notion upgraded its development tools to the latest versions and deployed new virtualization and management technologies for its test environment. Specifically, the company:
Desktop and Server Software Upgrades
- Upgraded all developer desktops from Visual Studio Team System 2008 Team Suite to Visual Studio 2010 Ultimate.
- Upgraded from Visual Studio Team System 2008 Team Foundation Server to Visual Studio Team Foundation Server 2010 for better version control, workflow management, build automation, automated unit testing, bug tracking, and test case management.
- Used Visual Studio Lab Management 2010 to virtualize all required test environments, and configured Team Foundation Server 2010 to automatically deploy builds into those test environments.
- Adopted Microsoft Test Manager (included in Visual Studio 2010 Ultimate) to manage test cases, assign them, and capture test results.
The move to Visual Studio 2010 began in April 2009 with an upgrade of all developer desktops from Visual Studio Team System 2008 Team Suite to Visual Studio 2010 Ultimate. Developers then worked for a week or so to resolve the few build issues that emerged. Following that, Menegay personally upgraded the production server from Visual Studio Team System 2008 Team Foundation Server to Visual Studio Team Foundation Server 2010.
Upgrading to the latest version of Team Foundation Server was easy,” says Menegay. “I first used the physical-to-virtual tools provided with Microsoft System Center Virtual Machine Manager 2008 to move our existing Team Foundation Server 2008 instance to a virtual machine. Then I installed the upgrade, with the entire process taking about 15 minutes of effort over a two-hour period.”
||We’re finding and fixing many more bugs than before—most being the kind that are hard to find but easy to fix.
Vice President of Consulting, Notion Solutions
Several months later, when Team Foundation Server 2010 Beta 2 came out, Notion Solutions decided to move it to a faster server and took the opportunity to perform a clean install. “We disconnected the Team Foundation Server Beta 1 instance from the database, installed Beta 2 running under Hyper-V on Windows Server 2008 R2, and reconnected the database,” says Menegay.Virtualized Test Environment
After the upgrade to Team Foundation Server 2010 Beta 2, Notion Solutions deployed Visual Studio Lab Management 2010 to facilitate the management and rapid provisioning of virtual lab environments for test execution. Lab Management 2010 uses System Center Virtual Machine Manager 2008 R2, a license for which is included with Lab Management 2010 for use in development and test deployments. The company also configured Team Foundation Server 2010 to automatically deploy new builds into the virtualized test environments.
With Lab Management 2010, the developers at Notion Solutions can rapidly set up, tear down, and restore virtual environments to a known state for testing. “Lab Management 2010 extends build automation by automating virtual machine provisioning, build deployment, and build verification—all in a highly integrated manner,” says Menegay. “It also allows us to file bugs with rich data and links to environment checkpoints that can be used to recreate test environments, effectively reducing wasted time and resources in the development lifecycle.”
In configuring its test environment, Notion Solutions first created virtual machines (VMs) running the 32-bit and 64-bit editions of the Windows XP, Windows Vista, and Windows 7 client operating systems, and the Windows Server 2003, Windows Server 2003 R2, Windows Server 2008, and Windows Server 2008 R2 server operating systems. The company then grouped the VMs into different combinations of client and server operating systems against which to test, with 32-bit client hitting 64-bit server, 64-bit client hitting 32-bit server, and so on, with each combination also including a third VM running a domain controller. “Creating physical environments for all of this testing would have cost well over $10,000,” says Menegay. “By easily creating these environments as VMs, we were able to do testing we otherwise likely would have not done.”
Continues Menegay, “Testers use the network isolation feature of Lab Management 2010 to launch clones of the VMs in a way that enables them to work alongside each other without conflicts. The greatest new value we’ve received from Visual Studio 2010 so far is definitely from the lab management capabilities, in that they put all the capabilities we need to deploy and manage a complex test environment at our fingertips. Lab Management 2010 is really slick and works like magic—it’s hard to describe; you’ve got to see it to believe it.” Rigorous Testing
Having previously managed test cases using Microsoft Word documents, the first step was to move all 450 or so test cases into Microsoft Test Manager. Notion Solutions now uses Test Manager to articulate a test plan for each test environment, manage the testing process, and collect test results.
“We now have approximately 100 ‘smoke tests’ that developers need their code to pass before checking it in. For example, they have to make sure that they didn’t break the installer,” says Menegay. “Beyond that, we have about 350 regression tests that we run as a part of each full test pass. We just started using the Test Impact Analysis feature and are anxious to see how it can aid us in determining which unit tests are impacted by new code changes.”
With Test Manager, Notion Solutions maintains a test plan for each target test environment, with the corresponding test suite composed of individual test cases. “We used the metadata fields in the test cases to categorize each one. For example, we designated all smoke tests as Priority 1,” says Menegay. “That way, we can use query-based test suites to examine all test cases and generate the required set of tests.”
During testing, Notion Solutions uses the test data collection features provided in Visual Studio 2010 to capture relevant data. ”When a bug is filed, the information required to reproduce the bug is automatically attached to the entry,” says Menegay. “The way that Visual Studio 2010 automates test data collection saves considerable time, both in the speed at which we can test and by giving developers the data they need to quickly reproduce and correct the problem.”Physical Architecture
From an architectural perspective, the company’s test environment is supported by three servers:
- Team Foundation Server 2010 runs in a virtualized environment based on Windows Server 2008 R2 with Hyper-V, alongside several other VMs that host production applications.
- The data store for Team Foundation Server 2010 resides alongside several other databases on a shared database server that runs Microsoft SQL Server 2008 Standard and the 64-bit version of Windows Server 2003 Standard Edition.
- The virtualized lab environment is hosted on a dedicated server that runs Windows Server 2008 Enterprise with Hyper-V.
Notion Solutions is significantly improving the quality of its software offerings by using Visual Studio 2010 to streamline and manage its test processes. “Version 1.2 of Notion Tools is now in the final test phase, and we’re finding and fixing many more bugs than before—most being the kind that are hard to find but easy to fix,” says Menegay. “We now have a much firmer grasp on the testing process and are taking advantage of the new capabilities provided by Visual Studio 2010 to ensure a much higher level of software quality.”Improved Software Quality
In testing Notion Tools version 1.2, Notion Solutions has seen a significant increase in the number of bugs filed—and fixed. “We’re finding and resolving bugs faster than ever before,” says Menegay. “Most of our support calls today are small issues—primarily related to different operating system versions—that we should have caught in the past but somehow missed. Considering that we’re finding and closing 10 times as many bugs as before, we expect quality to be significantly higher and the number of customer support calls to decrease dramatically.”Increased Testing Productivity
Notion Solutions has significantly increased the productivity of developers who assist in the testing process, making it possible for them to get more done in less time. “As new people are available for testing, we assign them test cases inside Test Manager, and they simply deploy their own VM environment and begin testing,” says Menegay. “We can do more testing than ever before because we can spin-up new test environments so quickly and easily. The lab management capabilities we now enjoy are saving each person who assists with testing about four hours per week.”New Developer Tools
Beyond the release of Notion Tools 1.2—the impetus for which is improved quality—developers will start working on new features for version 2.0, during which they plan to take advantage of other Visual Studio 2010 tools and features. “In developing Notion Tools 2.0, we’ll be using the profiling tools in Visual Studio 2010 Ultimate to improve application performance,” says Menegay. “We’re also starting to use the software modeling tools in Visual Studio 2010, and are beginning to develop automated user interface tests that will run as part of the build verification process.” Simplified Licensing
With its new test environment, Notion Solutions is getting more than ever out of its development tools budget, including increased value from MSDN subscriptions. “Team Foundation Server 2010 is now included with most MSDN subscription levels, so there’s no need to purchase it,” says Menegay. “And the Client Access Licenses are included with most MSDN subscriptions as well, so there’s no cost there either. Most importantly, our MSDN subscriptions include access to all the Microsoft software that we need for development and test purposes. With our new lab environment, we’re running all different kinds of Microsoft software for testing purposes, and, because everyone has an MSDN subscription, we can download and install almost whatever we want.”
Microsoft Visual Studio 2010
Microsoft Visual Studio 2010 is an integrated development system that helps simplify the entire development process from design to deployment. Unleash your creativity with powerful prototyping, modeling, and design tools that help you bring your vision to life. Work within a personalized environment that helps accelerate the coding process and supports the use of your existing skills, and target a growing number of platforms, including Microsoft SharePoint Server 2010 and cloud services. Also, work more efficiently thanks to integrated testing and debugging tools that you can use to find and fix bugs quickly and easily to help ensure high-quality solutions.
For more information about Visual Studio 2010, go to For More Information
For more information about Microsoft products and services, call the Microsoft Sales Information Center at (800) 426-9400. In Canada, call the Microsoft Canada Information Centre at (877) 568-2495. Customers in the United States and Canada who are deaf or hard-of-hearing can reach Microsoft text telephone (TTY/TDD) services at (800) 892-5234. Outside the 50 United States and Canada, please contact your local Microsoft subsidiary. To access information using the World Wide Web, go to:
For more information about Notion Solutions, call (972) 607-4830 or visit the Web site at: