Rex Black

Only honest data yields worthwhile information.

One development engineer told me that he “felt very depressed” after reading the risk analysis documents my test staff had prepared. I find it more depressing, however, to ignore the possibility of failure during development, only to live with the reality of preventable failure after release.

Skip. You've decided to skip the test for this pass, typically because it's a relatively low priority. Again, the Comments column of the test case summary worksheet should reflect why the test is not run. One way to think of the Skip state is “I could have run the test, but I didn't want to.”

You also might want to think twice about certain types of metrics—in particular, those that address questions such as “Who found the most bugs overall?”, “Who spent the most time testing?”, and “Who found the most bugs per hour spent?” I find these numbers dangerous, and I don't generate them. As soon as you begin analyzing employee performance data, you take risks. If people know you are measuring them this way, they will soon figure out how to manipulate the numbers.

The non sequitur interview involves asking some question that seems to be analogous or related to testing. The classic example is, “Tell me how you would test a salt shaker.” I find this silly. I call this a non sequitur interview style because it does not follow that, just because someone can craft a clever tale about testing simple real-world objects, they can test complex software and systems. You are hiring a test professional, not a raconteur.

I sometimes hire people whom I think will grow into a job, but I never hire people I think are trying to con their way into a job.

People, regardless of where they are, are not widgets. They want responsibility. They want respect. They want to be valued. They want us to be interested in developing their skills and their careers, even if they work for a third-party contractor. They want feedback.

The most dangerous kind of wrong is the kind of wrong that sounds reasonable, as I've said elsewhere in this book.

Don't take bugs personally, and don't become emotional about them. The times I have anointed myself the defender of quality and acted holier-than-thou with my colleagues are moments in my professional life that I look back on with embarrassment and regret. Even if you're right on the issue, you're doing the wrong thing because you end up alienating people with whom you need to maintain a professional and trusting relationship

Irreproducible Bug

The challenge with irreproducible bugs comes in two flavors. First, some bugs simply refuse to reproduce their symptoms consistently. This is especially the case in system testing, in which complex combinations of conditions are required to re-create problems. Sometimes these types of failures occur in clusters. If you see a bug three times in one day and then don't see it for a week, has it disappeared, or is it just hiding? Tempting as it is to dismiss this problem, be sure to write up these bugs. Random, intermittent failures—especially ones that result in system crashes or any other data loss—can have a significant effect on customers.

The second category of irreproducible bugs involves problems that seem to disappear with new revisions of the system, although no specific fix was made for them. I refer to these as “bugs fixed by accident.” You will find that more bugs are fixed by accident than you expect, but that fewer are fixed by accident than some project Pollyannas suggest. If the bug is an elusive one, you might want to keep the bug report active until you're convinced it's actually gone.