Randomized Testing Works

Andrew Leigh is Australia’s shadow assistant treasurer, an Australian MP and a former economics professor at the Australian National University. He discusses the methods and positive results of randomized trials. Randomized trials, he asserts, increase knowledge and improve outcomes. Big data can contribute more information more cheaply, which would simplify trials. As randomized trials revolutionized medicine in the 20th century, Leigh believes they can revolutionize social policy in the 21st.

His treatise drew wide-ranging praise. Steven Levitt, the author of Freakonomics, said of Leigh’s work, “Packed with tantalizing tales, Randomistas is essential reading for anyone interested in debunking myths and uncovering hidden truths.” Publishers Weekly wrote, “Leigh finds that randomized trials have challenged assumptions in many fields, from social welfare policy to retail marketing strategies.”

Randomized trials

Leigh opens by stating that people err in thinking randomized trials aren’t useful. Randomized trials, he argues, contribute to a growth mind-set because discovering what you don’t know makes the world better.

Leigh explains that governments, medicine, aid agencies, economists and advertising use them to measure impacts on quality of life, recovery times and access to services and social programs, policies for prison systems and voting habits.

There’s simply no better way to determine the counterfactual than to randomly allocate participants into two groups: one that gets the treatment and another that does not.Andrew Leigh

To conduct a randomized trial, Leigh details how developers choose a problem – such as high rates of absenteeism in schools – and a possible solution – such as offering an after-school program. They select a treatment group to receive the proposed remedy and a control group that receives no remedy. Leigh clarifies that comparing the treatment group with the counterfactuals allows developers to assess a program’s effectiveness. 

Randomized trials that address complex social issues take time, and Leigh takes heart that such trials helped developed policies for wage subsidies, health insurance coverage, job training, universal basic income, anti-smoking campaigns and support for psychiatric patients.

20th-Century Medicine

After initial safety testing on animals, Leigh illustrates how drugs undergo trials before going to market. Phase I trials involve fewer than 100 people, and Phase II increases that number. Phase III tests the drug on thousands of people and compares results with other drugs. If a drug passes all the tests and goes to market, Leigh recounts that a post-testing phase takes place in the general population. He points out that only one in ten drugs passes all trials.

Poverty and Homelessness

In 1997, Leigh tells, the US government ran a pilot called Moving to Opportunity to determine how geography affects people’s capacity to overcome poverty.​​​​​ Researchers assigned thousands of people in Los Angeles, Chicago, New York, Baltimore and Boston to one of three groups: one that received a voucher for subsidized housing in a low-poverty district, another that received a voucher with no usage restrictions, and a third that received no vouchers and stayed in public housing.

Leigh reveals that, over the years, testing showed that moving to a low-poverty neighborhood improved overall health but didn’t show discernible improvements for children. In 2015, a Harvard study linking taxation records found that, among kids who moved to a low-poverty neighborhood before age 13, their average income in adulthood was one-third higher than those who stayed in the high-poverty neighborhood.

The incremental approach won’t remake the world overnight, but it will over a generation.Andrew Leigh

Randomized trials that address complex social issues take time, Leigh notes, and success is uncertain. But such trials, he asserts, have proven crucial in developing policies for wage subsidies, health insurance coverage, job training, universal basic income, anti-smoking campaigns and support for psychiatric patients.

Social Pressure

Leigh tells how randomized trials show that voters responded more to nonpartisan literature that “shamed” them into being good citizens by threatening to publicize voter turnout identities than to television ads or partisan mailings.

People don’t vote because millions are voting; they vote because their closest friends are voting.Andrew Leigh

Shaming increased voter turnout, Leigh details, to a massive rate of one extra voter for every twelve letters sent. Partisan literature, Leigh notes, has a take-up rate of only one extra voter for every 10,000 letters – statistically zero.

Ethical Standards

Ethical safeguards, Leigh avows, protect test subjects. Ethical questions arise in tests that assure hardship for those who, for example, don’t receive treatment or aid.

Sometimes, Leigh laments, failing to undertake randomized tests proves unethical. He offers the example of how, from 1957 to 1961, doctors in the United Kingdom, Germany and Canada prescribed Thalidomide to pregnant women with morning sickness, which resulted in the births of more than 10,000 deformed babies. The drug didn’t pass US Food and Drug Administration regulations; pregnant American women didn’t take it.

Not a Policy Wonk

Leigh offers complex ideas with enthusiasm and economy of language. He loves his subject so he’s almost a nerd about it. What saves him from talking only to himself is how much he believes in random testing as a social good and a tool for positive changes. Leigh’s an optimist regarding the junction of science and human nature – an unusual position for a politician – and wants you to embrace random testing and its possibilities as he does. His passion, bemused common sense and bracing language combine to make this an especially engrossing and informative read.

Andrew Leigh also wrote What’s the Worst That Could Happen?; Leading the Way; Battlers and Billionaires; The Luck of Politics; and Reconnected, which he co-authored with Nick Terrell.