Don't Test Users, Test Hypotheses

User testing typically consists of a sort of fishing trip. We lower a lure (the user) into the water (the application or site) and see what critters (defects) bite. This is a valuable and time-tested approach. But when we start fishing for defects, we are left with some tough questions. For instance: When are we finished? How many defects do we need to find before we have fully tested the site or application? If we find a defect, how do we know how severe it is, and by what measure? In iterative testing, how do we compare results from the test of the current version with results from testing earlier versions?

Soudack, Avi. Boxes and Arrows (2003). Articles>Usability>Methods>Testing