If every test manager would just...
Updated: October 15, 2004
On This Page
Donald Burn: ...consider these points.
1. Teach your company that testing is critical
Various studies indicate that the cost to the company of a bug found by a customer is anywhere from 100 to 10,000 times more expensive than finding it in-house. Likewise a bug found in QA testing costs anywhere from 10 to 100 times more than finding it during development.
2. Require a complete interface specification
A driver cannot be fully tested unless all inputs and outputs are known. This includes supported functions, IOCTLS, device interfaces, and registry settings the driver depends on.
3. Start test development and testing early
Test planning and development needs to be concurrent with the development of the driver. Testing by development should begin as soon as possible. A complete driver is not needed for development testing because a single code module can be tested in many cases.
4. Accept only "clean" drivers
A driver should not leave development until it runs cleanly on the checked and free builds with driver verifier. The driver should run with CUV with any complaints from that tool documented as to why they do not matter.
5. Contact WHQL early
You can work with WHQL to get around special driver issues, but it takes time. Starting the discussion with WHQL early is the best way to be sure to have a signed driver when your product ships.
6. Provide the tools to make tests easy to write and run
Providing a framework for writing tests in a standard manner will pay back in the end. Having a common method for logging results and invoking tests can make things easier for everyone.
7. Regression test everything
Nothing frustrates a user more than a bug that reappears. Make sure all bug reports result in regression tests that become part of the release cycle.
8. Consider the problems of reuse
Driver developers leverage existing code. Recognizing this can help testing and quality because a bug in one driver is likely to manifest itself in other drivers with the same code. Plan driver testing to utilize regression tests from drivers of the same lineage.
9. Recognize that quality is everyone's responsibility
Testing does not produce quality; it can only test for its absence. Quality comes from all functions involved with the driver working together to produce a quality product.
Stephan Wolf: ...insist on engineering verification tests.
Most test engineers I have seen are too good natured. They arrange for an easy life for software engineers by allowing them to generate new software versions all the time and throw these over to the test group untested.
But test engineers should not accept any software that the developers have not tested themselves before. The test group of course verifies all features and trivial setups, but the major task of the test group is to find bugs or anomalies that only arise in special, usually extreme, situations or contexts.
It is clear that a developer cannot have a test environment with hundreds of workstations. But a developer should have at least a minimal test bed for fundamental testing.
This leads to the following, if oversimplified, test road map:
Software developer performs engineering verification test (EVT).
EVT is complete only when it reveals no more obvious failures.
Software goes to the Test and Verification group.
Test and Verification group tests software intensively.