2 page Case Study
Posted: 5/22/2012
Rate This Evidence:

Questionmark Software Company Boosts Productivity 67 Percent with Virtual Test Labs

Questionmark needed to streamline application lifecycle management and team collaboration among developers. It consolidated test environments on TeamDevCentral, a Microsoft Visual Studio 2010 Team Foundation Server and Microsoft Visual Studio 2010 Lab Management environment hosted by rational eyes, a Microsoft partner. Using virtual lab management to automate build-deploy-test workflows, Questionmark achieved more productive, agile software development.

Business Needs
Every year, Questionmark customers run 100 million assessments using either the on-premises Questionmark Perception solution, or Questionmark OnDemand, a software as a service offering. Both solutions provide test scheduling, delivery, and analytics, in addition to an open authoring tool called Questionmark Live that subject matter experts use to create and review assessment content.

“Our success rests on product innovation and features that our customers can trust,” says Eric Shepherd, Chief Executive Officer at Questionmark. “To achieve this, we must give our developers and QA teams the best tools for increasing quality and efficiency within our application lifecycle management (ALM) processes.”

Questionmark has four development environments: a developer’s own PC, as well as application, integration, and staging environments. It uses Scrum agile software development methodology with team projects organized in two-week sprints for each of the authoring, scheduling, delivery, and analytics functional areas in Questionmark products. The company used Microsoft Visual Studio Team System 2008 Team Foundation Server to store product backlogs and user stories, which outline use case scenarios, but globally-dispersed test environments made it difficult to coordinate efficient build-deploy-test workflows.

“Environments available for testing weren’t easily discoverable by other teams, so they set up their own,” says Mack Male, Software Engineering Manager at Questionmark. “We had to wait for operations to manually set up test environments. Generating a build from Team Foundation Server was automated, but we copied files to the test environment manually, which opened up room for errors, like accidentally overwriting a configuration file.”

Questionmark is promoting its popular OnDemand product, which has a quarterly release cycle. “We need to be more efficient to improve our QA processes and meet our OnDemand release schedule,” says Shepherd. “To achieve that, we had to consolidate our test environments and automate ALM processes.”

* We are offering more features... more frequently and more reliability on the same head count, because of the build-deploy-test efficiencies brought about by Lab Management. *
Eric Shepherd
Chief Executive Officer, Questionmark
At first, Questionmark tried running test environments on the Amazon Elastic Compute Cloud platform. Testers could quickly provision environments, but there were three key concerns: personal identifiable information used during QA process needed protection, the servers required administration, and the solution did not interoperate with Team Foundation Server. Questionmark turned to rational eyes, a member of the Microsoft Partner Network with Gold competencies.

Rational eyes recommended an integrated ALM and virtual lab management solution: Microsoft Visual Studio Team Foundation Server 2010, along with Microsoft Visual Studio 2010 Lab Management. Questionmark was already familiar with Team Foundation Server source control management, build automation, and work item tracking. Lab Management adds the promise of virtualized, on-demand test environments to deliver more efficient build-deploy-test processes. Questionmark developers can deploy their builds into a Lab Management virtual machine in a pre-defined test environment, and testers can file bug reports with links to environment snapshots for developers—all with a few clicks.

“We loved the solution’s ability to automate workflows and streamline collaboration between development, QA, and operations, and while we could run this ourselves for the same benefit, we wanted to consolidate our global test environments into one hosted location to streamline team collaboration,” says Shepherd. “When rational eyes told us they could host the entire Microsoft solution for us, we were thrilled.”

In early 2010, Questionmark consolidated its global test environments on TeamDevCentral, an outsourced Team Foundation Server and Lab Management development collaboration environment hosted by rational eyes. “We provisioned three build servers and defined capacities for our test environments,” says Male. “We use a virtual private network to access our Microsoft ALM solution, and rational eyes takes care of all the infrastructure maintenance. It’s been great!”

Using Team Foundation Server and Lab Management, Questionmark has achieved significant improvements in developer productivity and ALM. “We are offering more features to the marketplace more frequently and more reliability on the same head count, because of the build-deploy-test efficiencies brought about by Lab Management,” says Shepherd.

Improved Productivity by 67 Percent
Authoring team developers and QA work closely together to continuously build, integrate, and verify changes—as well as find and fix more bugs earlier in the build-deploy-test cycle. A developer checks in code to Team Foundation Server, which builds the code and runs some automated tests, then clicks on a button to automatically deploy the build to a virtual machine in Lab Management, where the QA team can perform exploratory testing. “We reduced the overall build-deploy-test cycle by 67 percent and significantly reduced human error,” says Male. “It took 30 minutes to manually deploy a test version of Questionmark Live. Now that we’ve hooked up the builds to Lab Management, it takes five minutes.”

Improved Business Agility
Developers and testers accomplish more every two-week release cycle. “To offer customers the latest, highest quality and most maintainable features, we need to make changes quickly,” says Male. “Using Lab Management and Team Foundation Server, we can work more efficiently to produce features of higher quality.”

Gained a Single Integrated Test Environment
Questionmark achieved its initial goal of consolidating all of its environments in one place, where they are readily available to anyone. “Next, we want to get the environments talking to each other,” says Shepherd. “We are working with rational eyes to help teams reduce duplication of effort. We are only scratching the surface of what Lab Management and Team Foundation Server can deliver.”

For more information about other Microsoft customer successes, please visit:

Solution Overview

Organization Size: 100 employees

Organization Profile

Questionmark provides assessment technologies and services that enable organizations to securely measure employees’ knowledge, skills, and attitudes for certification and regulatory compliance and successful learning outcomes.

Software and Services
  • Microsoft Visual Studio Lab Management 2010
  • Microsoft Visual Studio Team Foundation Server 2010

Vertical Industries
IT Services

United States

Business Need
Business Productivity


rational eyes