Workshop on Industrial Software Testing @ FSE 2016

About

As part of the 24th ACM SIGSOFT international symposium in the foundations of software engineering, the goal of this workshop is to discuss common, open issues in industrial software testing. This workshop is not about completed research studies or a showcase of the latest achievements in software testing. With this workshop, we want to provide a platform for industry and research to discuss unsolved testing and verification issues that occur in industry.

Therefore, we seek problem statements that go beyond unit and functional correctness testing and target the entire verification process including the expensive field of system and integration testing (a topic that is underrepresented in our field but that seem to cause huge challenges in practice). We are inviting a mixed audience of industry and researchers looking for opportunities to engage with industry partners or want to get feedback on open research tracks.

Organizers

Kim Herzig, Microsoft, Redmond
kimh/microsoft/com

Peter Rigby, Concordia University, Montreal
peter.rigby/concordia/ca

Program Committee

Academia
  • Margaret-Anne Storey (University of Victoria)
  • Gordon Fraser (University of Sheffield)
  • Sascha Just (Saarland University)
  • Emad Shihab (Concordia University)
Industry
  • Pete Rotella (Cisco)
  • Suresh Thummalapenta (Microsoft)
  • Michaela Greiler (Microsoft)
  • Chris Griffiths (Ericsson)
  • Gary McKenna (Ericsson)

Call for Contributions

In recent years, the pressure on software producers to cut their time to market for software products has increased significantly. At the same time, software products get more complex. From this perspective, software testing and verification is a lower bound on how fast we can ship software. While there have been outstanding advances in build systems, software testing processes have not changed significantly, as it rarely has been the primary bottleneck blocking software development agility.

We seek for contributions articulating and highlighting common issues in industrial testing. How do we test software, where are bottlenecks, and how should testing scale looking forward? We want to focus on open or only partially solved testing issues at industry scale. This workshop is about problem, not solutions.

Instead of focusing on one particular subset of testing, we encourage problem statements that go beyond unit and functional correctness testing, but rather target the entire verification process including the expensive field of system and integration testing. Our industrial partners have the feeling that discussions about open but common software testing issues as industry scale are underrepresented in at software engineering conferences but that seem to cause huge challenges in practice.

Potential and relevant topics might include:

  • Flaky tests: Tests failing due to any other reason than code defects pose a big threat to the efficiency and effectiveness of any verification process. There are many root causes for flaky tests but very little efficient solutions that help engineers to fix flaky tests and to design them in a more robust way in the first place. We are interested in common solutions and issues around the topic of flaky tests.
  • Test selection: While there is a long list of research on test selection and test prioritization, many of these research studies did not consider the economic aspect of test selection and the potential risk of elapsing defects into later stages of the development process. We are interested in strategies at large scale, for both unit as well as system and integration tests (system \& integration tests are widely ignored in research).
  • Verification processes: Testing is only one piece of the larger verification process for software products. We are interested in overall development process strategies and how testing (unit \& system and integration) fits into these processes. We seek common problems and bottlenecks in these processes as well as side effects from or to other processes, such as build.
  • Missing tool support: What test tools are currently available and are there any common gaps that need to be closed to allow engineers and testers to be more effective and efficient in creating, running, debugging, and triaging tests.
  • Migrating system & integration tests towards unit tests. By nature, system and integration tests are long running usually finding few code defects and if finding them, finding them late in the development process causing extra cost.

Submission Types

The workshop is open to the entire community and the following types of submissions are sought:

  • Practitioner Talk Abstracts (500 words) are only open to participants from industry and should describe, in 500 words or less, a talk (approx. 15 minutes in length) on a key aspect of testing in their organization.  These talks should be primarily experience-based and should be used as a means of communicating challenges or possible techniques that are in need of research.
  • Technical Papers (4 pages) should identify challenges, discuss opposing viewpoints, outline processes, or present solutions related to practical aspects of testing.

How to Submit

At the time of submission all papers must conform to the FSE 2016 Format and Submission Guidelines.

Papers must be submitted electronically through the workshops submission site.
Each submission will be reviewed by at least three members of the Program Committee. Submissions will be evaluated on the basis of relevance to industrial software testing, importance of the problem statement or contribution, quality of presentation and appropriate comparison to related work. The program committee as a whole will make final decisions about which submissions to accept for presentation at the conference.

Accepted paper will not be included in the conference proceedings. We are aiming to create a combined research statement and a combined industry problem catalogue as result of the workshop.

All publications are subject to the ACM Author Representations policy.

Workshop Format

For the program, we plan a keynote (not yet determined) and short talks from both communities (research & industry) in the morning and brainstorming sessions in the afternoon. The goal of the afternoon session is to let people engage on a more personal level and to discuss common issues and solutions in more depth.

As outcome of this workshop we plan to have a joint research problem statement letter as well as a summary of actionable results from industrial participants.

Participating Companies

These companies showed interest and are likely to participate:

  • Microsoft
  • Cisco
  • Ericsson
  • Intel
  • Amazon
  • Sendwithus

We are still spreading the word and hope to extend the list in the coming months.