What do we Lose with Online-Only Surveys? Estimating the Bias in Selected Political Variables Due to Online Mode Restriction

In: Statistics, Politics, and Policy, Band 8, Heft 1

Abstract

AbstractThis paper adds to the discussion on the value of online surveys for political science research. Mainly because of the lower costs, collecting survey data over the web has become increasingly popular in recent years, despite the higher sampling and coverage error in web-only surveys, especially online access polls. Recruiting respondents for the actual panel surveys based on a representative sample using a different mode is regarded as a solution to the sampling problem. Two approaches have been used to tackle the problem of coverage error: Providing respondents with computers (e.g. LISS, ELIPPS, GIP) and offering a different mode than online to respondents, thereby adopting a mixed mode design (e.g. paper as in GALLUP, GESIS Panel). The literature suggests that offering participation in the respondent's preferred mode affects response rates positively but not much is known about respondents' reasoning to choose a specific mode. We argue that it is important to understand this decision to evaluate the selection into online surveys and the consequences this has for data quality. We investigate this question by drawing on data from the GESIS Panel face-to-face recruitment interview for building a mixed mode access panel (paper and web) in Germany that gave a mode choice to internet users. Our results suggest that web literacy, age and education alone do not explain the mode choice but that affinity towards the technology related to the online mode has an independent effect. In a second step, we analyse the effect of this selection mechanism on answers to questions on typical variables used in political participation research such as media competence, political interest and civic duty in the subsequent mixed mode survey. We assess the added value of adopting a mixed-mode strategy. The results inform the evaluation of biases in unimode online surveys.