Nonresponse Bias in Randomized Controlled Experiments in Criminology: Putting the Queensland Community Engagement Trial (QCET) Under a Microscope

In: Evaluation review: a journal of applied social research, Band 37, Heft 3-4, S. 197-212

Abstract

Objectives: The goal of this article is to examine whether or not the results of the Queensland Community Engagement Trial (QCET)—a randomized controlled trial that tested the impact of procedural justice policing on citizen attitudes toward police—were affected by different types of nonresponse bias. Method: We use two methods (Cochrane and Elffers methods) to explore nonresponse bias: First, we assess the impact of the low response rate by examining the effects of nonresponse group differences between the experimental and control conditions and pooled variance under different scenarios. Second, we assess the degree to which item response rates are influenced by the control and experimental conditions. Results: Our analysis of the QCET data suggests that our substantive findings are not influenced by the low response rate in the trial. The results are robust even under extreme conditions, and statistical significance of the results would only be compromised in cases where the pooled variance was much larger for the nonresponse group and the difference between experimental and control conditions was greatly diminished. We also find that there were no biases in the item response rates across the experimental and control conditions. Conclusion: RCTs that involve field survey responses—like QCET—are potentially compromised by low response rates and how item response rates might be influenced by the control or experimental conditions. Our results show that the QCET results were not sensitive to the overall low response rate across the experimental and control conditions and the item response rates were not significantly different across the experimental and control groups. Overall, our analysis suggests that the results of QCET are robust and any biases in the survey responses do not significantly influence the main experimental findings.