Academic studies in the social sciences often find very different results. For one, chance errors could affect a study’s results. Researchers may also consciously or subconsciously bias their results. All these sources of variability have led to fears of a “replication crisis” in psychology and other social sciences relevant to business. Given this variability, how should we consume evidence?
The immediate answer is to not rely too much on any one study. Whenever possible, look for meta-analyses or systematic reviews that synthesize results from many studies, as they can provide more-credible evidence and sometimes suggest reasons results differ.
When considering how much weight to give a study and its results, pay attention to its sample size. Studies are particularly likely to fail to replicate if they were based on a small sample.
Smaller studies also often target the exact sample that would yield the biggest effects. More generally, it can be helpful to think about what things might be different if the intervention were scaled up. For example, small interventions are unlikely to affect the broader market, but if scaled up, competitors or regulators might change their behavior in response.
Similarly, consider peculiarities of the sample, context and implementation. How did the researchers come to study the people, firms or products they did?
If the study was evaluating an intervention, how that intervention was implemented is very important. You may also have more confidence in the results of a study if there is some clear, causal mechanism that explains the findings and is constant across settings.
Finally, if it sounds too good to be true, it probably is. This might sound like a cliché, but it’s based on a principle from Bayesian statistics: Stranger claims should require stronger evidence in order to change one’s beliefs, or “priors.”
What if there haven’t been many studies? If that’s the case, you may wish to consider other sources of evidence, such as advice or predictions from others.
Overall, trust a mix of your experience and the evidence, but be careful not to be overconfident in your assessments. Most people could benefit by weighing the evidence more, even when results vary.
(Eva Vivalt is a research fellow and lecturer at the Australian National University.)
