Can experts judge elections? Testing the validity of expert judgments for measuring election integrity

Abstract

Abstract. Expert surveys have been used to measure a wide variety of phenomena in political science, ranging from party positions, to corruption, to the quality of democracy and elections. However, expert judgments raise important validity concerns, both about the object being measured as well as the experts. It is argued in this article that the context of evaluation is also important to consider when assessing the validity of expert surveys. This is even more important for expert surveys with a comprehensive, world- wide scope, such as democracy or corruption indices. This article tests the validity of expert judgments about election integrity – a topic of increasing concern to both the international community and aca- demics. Evaluating expert judgments of election integrity provides an important contribution to the lit- erature evaluating the validity of expert surveys as instruments of measurement as: (1) the object under study is particularly complex to define and multifaceted; and (2) election integrity is measured in widely varying institutional contexts, ranging from electoral autocracies to liberal democracies. Three potential sources of bias are analysed (the object, the experts and the context), using a unique new dataset on election integrity entitled the 'Perceptions of Electoral Integrity' dataset. The data include over 800 experts in 66 parliamentary and presidential elections worldwide. It is found that validity of expert judg- ments about election integrity is increased if experts are asked to provide factual information (rather than evaluative judgments), and if they are asked to evaluate election day (rather than pre-election) integrity. It is also found that ideologically polarised elections and elections of lower integrity increase expert disagreement about election integrity. The article concludes with suggestions for researchers using the expert survey data on election integrity on how to check the validity of their data and adjust their analyses accordingly, and outlines some remaining challenges for future data collection using expert surveys.