Testing false positives and negatives. Which one's which?

 

We all know tests need to be reliable (don't we?!). One of the worst things any test can do is give false results, whether consistently or intermittently.

A test that passes when it should fail gives a false sense of security. A test that fails when everything is OK leads to wild goose chases trying to find the non-existent problem. This should be obvious.

We call these false positive and false negative results. 

But just recently I have been thinking about what is a false positive and a false negative? It's obvious - until it isn't. So hopefully writing this down will clear it up in my mind, and some others. I'll add that this is my thinking right now after discussing the issue with several people - just make sure the definition inside your own company is consistent.

Turns out, whether something is a positive or negative, false or otherwise, depends entirely on what signal is being detected. What is the 'positive' signal? 

For testing, what is a test designed to do? Either:

A test is designed to fail when there is an issue. So the 'signal', the positive case when present, is a test failure.

or....

A test is designed to show that the system under test is OK. So the signal being checked is a working system - a test pass.

For me a test fits into the latter category - it is designed to show that everything is working. That is the signal I am looking for in order to ensure a release will be clean and bug free. So what does this do to the definitions of false positives and negatives? This picture sums it up.