The Data Detective: Ten Easy Rules to Make Sense of Statistics – Tim Harford

Thoughts: The Data Detective is pitched as a corrective to Darrell Huff’s How to Lie with Statistics, which spends lots of time pointing out the ways that stats can be used to deceive, but doesn’t really include examples of statistics being used responsibly. While it touched on many themes I was already familiar with, like Phil Tetlock’s studies on forecasting (also in The Signal and the Noise and Enlightenment Now) and motivated reasoning (also in The Scout Mindset), I still found a bunch of new and useful ideas. A solid book, which I’d be happy recommending to anyone interested in thinking clearly in today’s world.

(The notes below are not a summary of the book, but rather raw notes - whatever I thought, at the time, might be worth remembering. I read this as an e-book, so page numbers are as they appeared in the app I used, Libby.)

Harford, Tim. 2021. The Data Detective: Ten Easy Rules to Make Sense of Statistics. Riverhead Books.

Introduction: How to Lie With Statistics

• 23: study by Kari Edwards and Edward Smith, underlining the power of doubt: “[Political] biases tended to appear more clearly in negative arguments. Disbelief flowed more fluidly than belief. The experimental subjects found it much easier to argue against positions they disliked than in favour of those they supported.”

Rule One: Search Your Feelings

• 45: People can more easily engage in motivated reasoning the more detail they have supporting position. 1979 study by Charles Lord et al. that gave people invented studies / other evidence arguing in favor of a position that they already held found that “the more detail people were presented with—graphs, research methods, commentary by other fictional academics—the easier they found it to disbelieve unwelcome evidence. If doubt is the weapon, detail is the ammunition.”
  • “The counterintuitive result is that presenting people with a detailed and balanced account of both sides of the argument may actually push people away from the centre rather than pull them in.”

Rule Two: Ponder Your Personal Experience

• 60: importance of choosing the right measure of centre: most people’s experience of a phenomenon can be quite different than the “average” experience: “For an extreme illustration, imagine a hypothetical train line with ten trains a day. One rush-hour train has a thousand people crammed onto it. What’s the average occupancy of these trains? A hundred people…. But what is the experience of the typical passenger in this scenario? Every single person rode on a crowded train.”
• 65: Harford quotes from Thinking Fast and Slow: “When faced with a difficult question, we often answer an easier one instead, usually without noticing the substitution.”
  • example he gives: when a person is asked how likely they are to be killed in an act of terrorism, the person will often instead ask themselves whether they can imagine being killed by terrorists, or whether they’ve recently heard a news report about an act of terrorism.
• 68: when is it best to rely on personal experience/intuition rather than running the numbers? situations that are novel and/or rapidly changing, and situations with hard-to-quantify details
• 72: to explore: Dollar Street, which uses images of people’s houses, household objects, etc. to give people an intuition of what it means to live on N dollars per month

Rule Three: Avoid Premature Enumeration

• 79: Harford coins the term “premature enumeration” for situations where one is evaluating a statistical claim, and instead of asking what’s being measured/counted and how that thing is defined, people jump straight to questions of trends and error bars and sample sizes.
• 84: responding to the aphorism that “the death of one man is a tragedy, the death of millions is just a statistic”, Harford states that “Premature enumeration is not just an intellectual failure. Not asking what is statistic actually means is a failure of empathy, too.”
  • 86: “Awful tragedies lie behind [many statistical figures]. Pinning down the definitions is vital if we want to understand what is happening and, perhaps, how we might make life better. That is, after all, while we’re collecting the numbers.”
• 89: “Net wealth is a great way to measure riches, but not such a good way to measure poverty” (since anyone with more debt than assets, e.g. many students, has a less-than-zero net wealth and are likely to be lumped together with the poorest of the poor)
• 93: There are many different measures of inequality, and they can trend in different directions at the same time. e.g. in the UK from 1990-2017, the income of the richest 1% of the population grew relative to the other 99%, at the same time as the average income of the poorest 10% tended to catch up to the median income value
• 93: resource Harford recommends: World Inequality Database

Rule Four: Step Back and Enjoy the View

Rule Five: Get the Backstory

• 138: the study that demonstrated the backfire effect was striking, but later studies found that the effect size is not nearly as large as initially stated.
  • “One summary of the research concluded: ‘Generally debunking can make peoples belief in specific claim is more accurate.’”
• 142: recommended resource: the Cochrane Collection - database of systematic research reviews related to medicine. Better to search the Cochrane Collection with medical questions, which will give you summaries of the current balance of the evidence, rather than googling them, which will likely turn up individual studies or sources citing only a few of them.
  • 144: the Campbell Collaboration does something similar for social policy studies

Rule Six: Ask Who Is Missing

• 149: Book mentioned, about how people collecting data and/or designing things have often only considered men and not women: Caroline Criado Perez - Invisible Women

Rule Seven: Demand Transparency When the Computer Says No

• 185: in the early days of science, science and alchemy were often pursued by the same people (e.g. Robert Boyle, Isaac Newton). David Wootton suggests the main difference that explains the contrasting development of science and alchemy was that alchemy tended to be done in secret, while scientists published their findings and scrutinized each other’s experiments.
  • 188: “Just as the most brilliant thinkers of the age failed to make progress while practising in secret, secret algorithms based on secret data are likely to lead to missed opportunities for improvement”
  • 195: One possible way to allow scrutiny and peer review while not giving everyone access to data companies went to a bunch of trouble to gather could be to force private data sets to be released several years after they were initially gathered “Three-year-old data are stale for many commercial purposes but may still be of tremendous scientific value.”

Rule Eight: Don’t Take Statistical Bedrock for Granted

• 214: book mentioned: James C. Scott’s Seeing Like a State - Argues that the data that states tend to gather is often too general/to standardized to capture relevant local variations.
  • “Because the state is powerful, its misperceptions of the world I can take physical form, producing well-meaning but clumsy and oppressive modernist schemes that ignore local knowledge and stifle local autonomy”
• 216: book mentioned: Harford’s Messy.
  • “States should be humble. Bureaucrats must recognize the limits of their knowledge. There is always a risk that the bird’s-eye view is so grand and sweeping as to induce delusions of omnipotence.”

Rule Nine: Remember That Misinformation Can Be Beautiful Too

• 242: plotting linear data on a spiral may reveal cyclic patterns that otherwise get masked by non-periodic fluctuations

Rule Ten: Keep an Open Mind

• findings from Tetlock and Mellers on what makes a superforecaster:
  • 267: having a bit of training on forecasting helps
    • 267-268: one of the best ways to improve people’s forecasting ability is to prompt them to think about base rates
  • 269: keeping track of past predictions is important
  • superforecasters tend to continuously revise and update their forecasts as new information comes in
  • “Superforecasting is a matter of having an open minded personality. The superforecasters are what psychologists call ‘actively open minded thinkers’—people who don’t cling too tightly to a single approach, are comfortable abandoning an old view in the light of fresh evidence or new arguments, and embrace disagreements with others as an opportunity to learn.”

The Golden Rule: Be Curious

• 282: study by Dan Kahan on found that the best inoculation against toxic polarization is “scientific curiosity” (nb. not the same as scientific literacy)
  • “It’s important not to exaggerate the effect”
  • “There is little correlation between scientific curiosity and political affiliation”
• 287: beware of The Illusion of Explanatory Depth: people tend to believe they understand a thing better than they actually do. This tends to cause people to be less curious about things, and to ask fewer questions
• 288-289: narrative is very important in engaging people’s interest: “studies in which people were asked to read narratives and non-narrative texts found that they zipped through the narrative at twice the speed, and recalled twice as much information later.”