Four years ago, a study in Current Biology about religious upbringings and generosity made headlines in more than 80 newspapers around the world — from the Economist and the Guardian to the Boston Globe and the Los Angeles Times. (Vox did not cover the study.) The research found that around the world, religious children were less generous.
How a study based on a typo made news everywhere — and the retraction didn’t
A paper arguing religious kids are less generous turned out to be the result of a typo.
Kelsey Piper is a contributing editor at Future Perfect, Vox’s effective altruism-inspired section on the world’s biggest challenges. She explores wide-ranging topics like climate change, artificial intelligence, vaccine development, and factory farms, and also writes the Future Perfect newsletter.
But even at the time, some researchers raised questions about the study by the University of Chicago’s Jean Decety and co-authors. University of Oregon psychologist Azim Shariff told Science Magazine in 2015 that he was confused by the results as they didn’t match previous research in the field. “It doesn’t fit in easily with what’s been out there so far. So I’ve got to do some thinking — other people have got to do some thinking — with how it does fit,” he told the magazine.
Later, he requested the original authors share their data so he could figure out why they were getting different results from his own results.
The answer? The entire result was literally the product of a typo. The researchers had collected data in many different countries — the US, Canada, China, Jordan, Turkey, and South Africa. When coding in their results, they used numbers to represent each country — 1 for the US, 2 for Canada, and so on. Then, they tried to control for the country in evaluating their results. But instead of controlling for country, Psychology Today reports, they “just treated it as a single continuous variable so that, for example “Canada” (coded as 2) was twice the “United States” (coded as 1).”
Oops.
Shariff published a reanalysis in 2016 that pointed out the problem, but the paper was not retracted until August of 2019. “When we reanalyzed these data to correct this error,” the authors admit in the retraction notice, “we found that country of origin, rather than religious affiliation, is the primary predictor of several of the outcomes.”
But their retraction has not gotten a fraction of the attention that the original paper received. Most of the outlets that covered the original study have not covered the retraction, and almost all of the original articles about the study are still live for readers to stumble across.
In fact, the original study is still being cited: Harvard professor Tyler J. VanderWeele pointed out in Psychology Today that two new articles about it, on Buzzworthy and another news aggregation site, went up after the formal retraction was published.
The study shows a big problem in science and science reporting: Mistakes will inevitably be made, and often they’ll circulate much more widely than the corrections that follow later.
How errors circulate faster than corrections
The original mistake made by Decety and co-authors is a really embarrassing one. Ideally, it would have been caught in the peer review process rather than making it into a published paper.
But the scientific process on the whole did its job here, even if it was slower than we might have hoped. Another researcher in the field — Shariff — heard about the results, wondered why they were inconsistent with previous research, and requested Decety’s data. Decety shared his data and the error was discovered. That’s science: a little messy but fundamentally self-correcting.
The public record is not quite so self-correcting. Even though Shariff published his analysis in 2016, VanderWeele found that only four media outlets published it — compared to the more than 80 that published the original study. The retraction in August got even less coverage, and the misinformation is still spreading, even though there’s now a prominent retraction notice at the top of the original study.
How could we do better? For one thing, science reporters should always link to the original study. While they often won’t notice every retraction and go back and edit their story, if they link to the original study, it will at least be possible for readers to learn that it’s since been retracted. The Buzzworthy story that went up after the retraction doesn’t link to the original study anywhere.
But a larger problem might be that we don’t consider retractions, refutations, and failed replications of studies to be stories just as much as the original story. A surprising and interesting result — like that religious children aren’t as generous as nonreligious children — will make lots more headlines than the fact that this was a coding error and differences in generosity are mostly explained by country of origin.
The whole situation is a reminder to take any given study with a good deal of skepticism. Lots of published research doesn’t actually stand up to further scrutiny — and while scientists often get to the right answer eventually, wrong answers can circulate faster than right ones.
Sign up for the Future Perfect newsletter. Twice a week, you’ll get a roundup of ideas and solutions for tackling our biggest challenges: improving public health, decreasing human and animal suffering, easing catastrophic risks, and — to put it simply — getting better at doing good.
See More:
