How Three-Arm Random Assignment Within Sites can Improve Non-Experimental Cross-Site Estimates of the Relationship Between Program Characteristics and Impact

In: Evaluation review: a journal of applied social research, S. 0193841X2211406

Abstract

This article considers a new method, called Cross-Site Attributional Model Improved by Calibration to Within-Site Individual Randomization Findings (abbreviated as "CAMIC"), which seeks to reduce bias in analyses of the contributions of a program's various design, implementation, and contextual characteristics to its overall impacts. It requires a multi-site experiment where some (or all) sites randomize individuals to one of three arms: a standard treatment group, an enhanced treatment group (that receives the standard treatment plus a program "enhancement"), or a control group (that has no access to the program). A recent evaluation—that of the Health Profession Opportunity Grants (HPOG) program—provides a motivating example of this design and its potential for both methodological and substantive lessons; and the article considers other evaluations that would be fitting for deploying and learning from CAMIC. We conclude that the promise for CAMIC lies in situations where the correlations between the selected program enhancement and alternative program characteristics of interest are relatively high, implying that producing an experimental estimate of the enhancement can reduce bias in the estimation of other non-randomized program characteristics.