Experimentation Works

If you really want to understand the value of an experiment, just look at the difference between its expected outcome and its actual result. If you thought something was going to happen and it happened, then you haven’t learned much. If you thought something was going to happen and it didn’t, then you’ve learned something meaningful. And if you thought something minor was going to happen, and the results are a major surprise and lead to a breakthrough, you’ve learned something highly valuable

Twyman’s law: “Any figure that looks interesting or different is usually wrong.”

categorize the different levels of understanding causality, Judea Pearl and Dana Mackenzie propose a three-tier ladder in The Book of Why.19 The first and lowest causality tier, association, is about finding regularities in observations. One event is associated, or correlated, with another if observing one changes the likelihood of observing the other. The authors place modern-day analytics and big data in this tier. The second, intervention, requires changing one or more variables and observing changes in outcomes. Experiments are such interventions. The third and highest tier, counterfactuals, includes the strongest test of causality. Instead of just asking, “Did A cause B?” a higher standard includes the counterfactual, “Would B have occurred if not for A?”

If a company expects the cause (for example, a change in store name) to have a large effect (a substantial increase in sales), the sample can be smaller. If the expected effect is small, the sample must be larger. This might seem counterintuitive, but think of it this way: the smaller the expected effect, the greater the number of observations that are required to distinguish it from the surrounding noise with the desired statistical confidence.

That’s what the Dutch technology company Philips realized when it conducted a focus group of teenagers to assess their color preferences for a new boom box. During the session, most teenagers selected “yellow” as their preferred color. After the session, the teenagers received a boom box as a reward for their participation and were offered a choice of two colors: yellow and black. Most participants selected “black,” even though they had chosen “yellow” when their preference was posed as a hypothetical question.9 When it comes to behavior, it’s usually better to trust the experiment.

(1) Does the experiment have a testable hypothesis? (2) Have stakeholders made a commitment to abide by the results? (3) Is the experiment doable? (4) How can we ensure reliable results? (5) Do we understand cause and effect? (6) Have we gotten the most value out of the experiment? And finally, (7) Are experiments really driving our decisions?