Sue Lani Madsen: How to talk back to a statistic

The oft quoted “lies, damned lies and statistics” is attributed to British politicians but most famously to Mark Twain, itinerant freelance writer and observer of human nature. Politics as well as journalism are full of opportunities to use and be abused by statistics. And so the title of the little orange book caught my eye a few years ago when packing my aunt’s apartment after she died, “How to Lie With Statistics” by Darrell Huff. It takes an entertainingly Twainian approach to the ways and means, then turns serious with advice on spotting statistical fraud.

Huff admits his book “may seem altogether too much like a manual for swindlers,” but promises it will be useful for the reader’s self-defense. The 1954 cartooned graphics are original. Sixty years after first publication it’s still a best-selling reference for the non-statistician who wants to know “how to talk back to a statistic.” Huff recommends five key questions.

First, who says so? Where did the numbers come from? It’s easy to spot the potential for conscious bias. Bill Gates isn’t likely to highlight numbers demonstrating the value of more cows to organically capture carbon at the same time he’s pitching patented high tech Impossible Burgers. Unconscious bias is more dangerous, says Huff, and points to the “charts and predictions of statisticians and economists in 1928” focusing on good times forever and totally overlooking the cracks in the economy leading to the 1929 crash.

Second, how is it known? Was it a large or small sample, a random or a self-selected sample? Sometimes bias creeps in from unexpected factors. Take the 1936 election, when the Literary Digest infamously predicted Alf Landon would beat Franklin D. Roosevelt in a landslide. Its sample was taken from the magazine’s subscribers and lists of people with their own telephones, not at all representative of the total electorate. Election predictions in 2016 were similarly confounded by poor sampling.

Context is critical and “what’s missing” is the third question. In an era when a woman attending a university was suspected of just husband-shopping, Huff reports “someone not particularly enamored of coeducation reported a real shocker: Thirty-three and one-third per cent of the women at Johns Hopkins had married faculty members!” What was missing? There were only three women enrolled at the university that year.

Or as Todd Myers of the Washington Policy Center pointed out at a March 8 legislative update, the Seattle Times used a chart of Snake River salmon on the path to extinction with a graph ending at 2019, when 2017-2019 was a down cycle, leaving out the 2020 statistics and the current projections of continued recovery.

Then there’s the correlation/causation confusion, or “did somebody change the subject?” If last year’s data on fruit consumption counted apples, and this year counted oranges, it tells you nothing about annual fruit consumption.

Raw data also has to be relevant to the conclusion. If the metric for judging importance is number of stories in the newspaper, it may or may not be a match. Huff recounts how Teddy Roosevelt as leader of a board on police reform put an end to a New York City crime wave by telling two newspapermen competing for attention to knock it off. They had been front-page highlighting crime with ever-bolder headlines in a “competition as to who could dig up the most burglaries … the official police record showed no increase at all.” Click bait headlines have been around longer than clicks. (And by the way, the writer does not control the headline, that’s all on the editors.)

Then lastly, the “does it make sense” gut check. “Many a statistic … gets by only because the magic of numbers brings about a suspension of common sense,” writes Huff. The most egregious examples are the “impressively precise figures” hiding reality and nuance. Which sounds more gut-wrenching, average student loan debt in the U.S. is $38,255.03 or it’s about $38,000? The median at $17,000 tells us more. Grad students carry an average of $82,200. Forty-two percent of public university students graduate with no debt, 78% with less than $30,000 and only 4% with more than $60,000. And rattling off a whole string of statistics is another strategy for distraction.

Studying statistics reminds me of my Great Uncle John, who proudly proclaimed he’d graduated third in his class from Curlew High School. Then he’d laugh and tell you he also graduated last in his class. It’s all about context and framing. Don’t be April-fooled, be well-armed.

Contact Sue Lani Madsen at rulingpen@gmail.com.