Wednesday, March 11, 2009

Rating testers based on number of bugs they find

Should we measure the quality of testers—their productivity, efficiency and skill—by counting how many bugs they find? Suppose that mid-project we compared the bug counts per day from exploratory testing with those obtained from regression testing (reuse of test cases that the program has previously passed) of the same area of the program. We might discover that exploratory testing yielded a much higher bug-finding rate, and this might affect our choice of testing strategies.
* Please read this attached PDF. click here to download

Few of the things that can and probably will go wrong if we measure individuals by counting their bug reports.
1. People are good at tailoring their behavior to things they are measured against. If you ask a tester for more bugs, you’ll probably get more bugs. The additional bugs might be minor, or similar to already reported bugs, or design quibbles. But the bug count will go up.
2. People know that other people tailor their behavior. Put a tester under incentive to report more bugs, and programmers will become more skeptical of the value of the reports they receive. Does this tester believe in this bug, they ask, or is she just inflating her bug count? Bugcounting creates political problems (especially if you also count bugs per programmer).
3. You can make a tester look good or bad just by choosing what type of testing she should do (regression testing often yields fewer bugs than exploratory testing) or what area she should test (fewer bugs to find in less buggy code). If raises and promotions are influenced by bug counts, project assignments will often be seen as unfair.
4. Bug counting creates incentives for superficial testing (test cases that are quick and easy to create). Bug counts punish testers who take the time to look for harder-tofind but more important bugs.
5. Such a system also penalizes testers who support other testers. It takes time to coach another tester or to help him build a tool that will make him more effective. The tester who does this has less time to find bugs.
6. Time spent on any process (such as documenting test cases) that doesn’t lead to more bugs faster is time that counts against the tester.

To Read Completely please click this link:

1 comment:

Pradeep Soundararajan said...

There are very few testers who attribute and credit other testers for their work and you seem to be one among the few.

I would love to read your blog if you are going to share your experiences as a tester.