The Insane Pesticide Paradox

The pesticide paradox: continuing to run the same tests won’t find new bugs.

As part of the Seven Testing Principles of the ISTQB Foundation Level Syllabus, on the pesticide paradox principle.

“If the same tests are repeated over and over again, eventually these tests no longer find any new defects. To detect new defects, existing tests and test data may need changing, and new tests may need to be written. (Tests are no longer effective at finding defects, just as pesticides are no longer effective at killing insects after a while.) In some cases, such as automated regression testing, the pesticide paradox has a beneficial outcome, which is the relatively low number of regression defects.” ISTQB Foundation Syllabus

If a large number of tests are executed at the same time, it is likely that they will produce many false positives. False positives are not dangerous and can be ignored, but they are inconvenient; we are likely to ignore them or stop using the test entirely.

An example of a false positive that is likely to be ignored is if a test is designed to find a certain type of defect but occasionally produces a false positive. If a test is designed to find a certain type of defect but occasionally produces a false positive.

However, in other cases, like exploratory testing, the pesticide paradox is definitely not desirable. When exploratory tests are no longer effective at finding new defects, new exploratory tests may need to be written. In fact, this is the only way to ensure that exploratory testing keeps on delivering new defects to be fixed.

And finally, in most cases, the pesticide paradox is unwanted. We may want to find new defects every time. If we do not change our tests, we will not find new defects. But it is not that easy to change the tests.