Mismeasuring Impact

The hidden dangers of randomized controlled trials

The need to demonstrate the effectiveness of nonprofit social programs has led to a rapid rise in the use of randomized controlled trials (RCTs), for evaluation. As a result, most nonprofit sector professionals can tell you why nonprofits should do an RCT. This book tells you why they probably shouldn't, and what to do instead.

Mismeasuring Impact explores why RCTs are being embraced as the "gold standard" for nonprofit evaluation, despite the high cost and time investment required and the serious problems with using RCTs in a nonprofit context. (Most RCTs conducted in nonprofits fail to meet required standards for rigor, undercutting their accuracy). The book describes what happens inside nonprofits when they take part in RCTs, the unintended equity issues that arise, and why nonprofits feel pressured participate in RCTs despite the problems.

University of Chicago professors Marwell and Mosley's research is based on extensive interviews with key players: nonprofit managers, professional program evaluators, and program officers in philanthropic foundations. The book argues that, ultimately, RCTs are used to poorly ground nonprofit legitimacy, not to foster nonprofit improvement. RCTs also privilege program and organizational standardization over the key strengths of nonprofit organizations: flexible innovation and responsiveness to community needs.

Nonprofits and funders need forms of evaluation that lift up these strengths. Mismeasuring Impact offers alternative approaches that build strong organizations, not just cookie-cutter programs, and which funders and nonprofits of all sizes can support.