Search results

Grid questions are frequently employed in web surveys due to their assumed response efficiency. In line with this, many previous studies have found shorter response times for grid questions compared to item-by-item formats. Our contribution to this literature is to investigate how altering the question format affects response behavior and the depth of cognitive processing when answering both grid question and item-by-item formats. To answer these questions, we implemented an experiment with three questions in an eye-tracking study. Each question consisted of a set of ten items which respondents answered either on a single page (large grid), on two pages with five items each (small grid), or on ten separate pages (item-by-item). We did not find substantial differences in cognitive processing overall, while the processing of the question stem and the response scale labels was significantly higher for the item-by-item design than for the large grid in all three questions. We, however, found that when answering an item in a grid question, respondents often refer to surrounding items when making a judgement. We discuss the findings and limitations of our study and provide suggestions for practical design decisions.

Abstract
More survey results are available today than ever before. This increase in survey data has been accompanied by growing concerns about their quality. With the present study, we aim to investigate to what extent the public draws on survey quality information when evaluating the trustworthiness of survey results. We implemented a vignette experiment in an online panel survey (N = 3,313), in which respondents each received four different survey descriptions with varying methodological information. Compared with respondent characteristics, survey quality information had only a minor effect on perceptions of trustworthiness. However, trust in the survey results was significantly influenced by sample size and sample balance. Finally, the relevance of survey quality information increased with the cognitive ability of the respondent.

Attention checks detect inattentiveness by instructing respondents to perform a specific task. However, while respondents may correctly process the task, they may choose to not comply with the instructions. We investigated the issue of noncompliance in attention checks in two web surveys. In Study 1, we measured respondents' attitudes toward attention checks and their self-reported compliance. In Study 2, we experimentally varied the reasons given to respondents for conducting the attention check. Our results showed that while most respondents understand why attention checks are conducted, a nonnegligible proportion of respondents evaluated them as controlling or annoying. Most respondents passed the attention check; however, among those who failed the test, 61% seem to have failed the task deliberately. These findings reinforce that noncompliance is a serious concern with attention check instruments. The results of our experiment showed that more respondents passed the attention check if a comprehensible reason was given.

A major challenge in web-based cross-cultural data collection is varying response rates, which can result in low data quality and non-response bias. Country-specific factors such as the political and demographic, economic, and technological factors as well as the socio-cultural environment may have an effect on the response rates to web surveys. This study evaluates web survey response rates using meta-analytical methods based on 110 experimental studies from seven countries. Three dependent variables, so-called effect sizes, are used: the web response rate, the response rate to the comparison survey mode, and the difference between the two response rates. The meta-analysis indicates that four country-specific factors (political and demographic, economic, technological, and socio-cultural) impact the magnitude of web survey response rates. Specifically, web surveys achieve high response rates in countries with high population growth, high internet coverage, and a high survey participation propensity. On the other hand, web surveys are at a disadvantage in countries with a high population age and high cell phone coverage. This study concludes that web surveys can be a reliable alternative to other survey modes due to their consistent response rates and are expected to be used more frequently in national and international settings.

Identifying inattentive respondents in self-administered surveys is a challenging goal for survey researchers. Instructed response items (IRIs) provide a measure for inattentiveness in grid questions that is easy to implement. The present article adds to the sparse research on the use and implementation of attention checks by addressing three research objectives. In a first study, we provide evidence that IRIs identify respondents who show an elevated use of straightlining, speeding, item nonresponse, inconsistent answers, and implausible statements throughout a survey. Excluding inattentive respondents, however, did not alter the results of substantive analyses. Our second study suggests that respondents' inattentiveness partially changes as the context in which they complete the survey changes. In a third study, we present experimental evidence that a mere exposure to an IRI does not negatively or positively affect response behavior within a survey. A critical discussion on using IRI attention checks concludes this article.

Satisficing often has been assumed to be a hazard to response quality in web surveys because interview supervision is limited in the absence of a human interviewer. Therefore, devising methods that help to mitigate satisficing poses an important challenge to survey methodology. The present article examines whether splitting up cognitively demanding grid questions into single items can be an effective means to reduce measurement error and nonresponse resulting from survey satisficing. Furthermore, we investigate whether modifying the question design decreases the adverse effects of low ability and motivation of the respondents on response quality. The statistical analyses in our study relied on data from two web-based experiments with respondents from an opt-in and a probability-based online panel. Our results showed that using single items increased the response times compared to the use of standard grid questions, which might indicate a greater response burden. However, the use of single items significantly reduced the amount of response nondifferentiation and nonsubstantive responses. Our results further showed that the impact of respondent ability and motivation on the likelihood of satisficing was moderated by the question design.

This study reveals the existence of a paradox in how the public views polling within the democratic process. Specifically, even though the public believes that it can influence poli­cymaking, it considers public opinion polls not as useful as other, less representative forms of public input, such as comments at town hall meetings. Analyzing data from multiple surveys conducted in the United States of America, we find no evidence for the demo­cratic representation hypothesis with respect to polling. Comparisons across stakeholders (public, journalists, and politicians) demonstrate that general perceptions of inputs into the democratic process are similar, which confirms the citizen-elite congruence hypothesis. However, unlike members of the public, experts are more likely to believe that public opin­ion polls are the optimal method by which the public can successfully inform policymak­ing, a finding consistent with the legitimization hypothesis. With respect to perceptions of politicians, we found substantial differences regarding party registration with Democrats and Independents favoring public opinion polling and Republicans preferring alternative methods (e.g., town hall meetings) of informing policymakers.

Open survey questions are often used to evaluate closed questions. However, they can fulfil this function only if there is a strong link between answers to open questions and answers to related closed questions. Using reasons for non-voting reported in the German Longitudinal Election Study 2013, we investigated this link by examining whether the reported reasons for non-voting may be substantive reasons or ex-post legitimations. We tested five theoretically derived hypotheses about respondents who gave, or did not give, a specific reason. Results showed that (a) answers to open questions were indeed related to answers to closed questions and could be used in explanatory turnout models to predict voting behavior, and (b) the relationship between answers to open and closed questions and the predictive power of reasons given in response to the open questions were stronger in the post-election survey (reported behavior) than in the pre-election survey (intended behavior).

In this paper, we investigate the context stability of questions on political issues in cross-national surveys. For this purpose, we conducted three replication studies (N1 = 213; N2 = 677; N3 = 1,489) based on eight split-ballot design experiments with undergraduate and graduate students to test for question order effects. The questions, which were taken from the Eurobarometer (2013), included questions on perceived performance and identification. Respondents were randomly assigned to one of two experimental groups which received the questions either in the original or the reversed order. In all three studies, respondents answered the questions about Germany and the European Union/Europe differently depending on whether the question was asked first or second in the question sequence. Specifically, when answering a subsequent question in a question sequence, the preceding question seems to have functioned as a standard of comparison. Our empirical findings also suggest that the likelihood of the occurrence of such context effects can be reduced by implementing informed questionnaire design strategies.

AbstractObjective:Anecdotal evidence suggests that the term "pollster" has, in recent years, become stigmatized in the United States. We explore this and a subsequent question as to whether negative perceptions of pollsters affect people's perceived trustworthiness of survey findings.Methods:Survey experiments were administered to national probability‐based samples after the 2016 and 2020 elections.Results:In each study, pollsters obtained significantly more negative ratings when compared to "survey researchers" and "public opinion researchers," suggesting that the general public views pollsters, who are more likely to be viewed as partisan, as being less honest/ethical. In line with social identity theory, interaction models revealed that those rating pollster critic Donald Trump most favorably had the most negative ratings of pollsters and public opinion researchers, compared to survey researchers. Yet, the vignette experiment showed that negative perceptions of pollsters did not affect the perceived trustworthiness of survey result reports.Conclusions:We conclude that while there appears to be a stigmatization of pollsters, those negative perceptions do not translate into less trust in the findings of public opinion.

Background: COVID-19 vaccine hesistancy is a serious policy issue in Germany as vaccinations have stagnated at low levels compared to most other European countries. In this context, we study whether and how health workers can be leveraged to promote the COVID-19 vaccination campaign. Methods: We employed an information experiment with health workers in Germany to quantify how access to information related to (i) AstraZeneca's vaccine safety, (ii) misinformation, (iii) individual health risks, and (iv) public health risks can sway health workers' recommendations for any of the following vaccines: AstraZeneca, Johnson & Johnson, Moderna, Pfizer/BioNTech, Sinopharm, and Sputnik-V. The information experiment was conducted as a randomized controlled trial with four treatment arms and was embedded in an online survey. Results: Health workers reduce their willingness to recommend four out of six vaccines once they learn about different statements of European and German health authorities with respect to the safety of the AstraZeneca vaccine. Consistent with the discussion on AstraZeneca's safety focusing on possible side effects among younger women, we find that especially female health workers become less likely to recommend the majority of COVID-19 vaccines. Lastly, we show that health workers vaccine recommendations are not affected by misinformation and appeals to individual or public health. Conclusion: In order to mobilize health workers in the fight against vaccine hesitancy, information campaigns need to be tailor-made for the target audience. In particular, health workers react to different types of information than the general public. As with the general public, we provide suggestive evidence that health workers require unambigious messages from drug authorities in order to support vaccination efforts. We believe that a more coordinated and coherent approach of public authorities can reduce the amount of mixed signals that health workers receive and therefore contribute to health workers engagement in the outroll of mass COVID-19 vaccination campaigns.

Filter questions are used to administer follow-up questions to eligible respondents while allowing respondents who are not eligible to skip those questions. Filter questions can be asked in either the interleafed or the grouped formats. In the interleafed format, the follow-ups are asked immediately after the filter question; in the grouped format, follow-ups are asked after the filter question block. Underreporting can occur in the interleafed format due to respondents' desire to reduce the burden of the survey. This phenomenon is called motivated misreporting. Because smartphone surveys are more burdensome than web surveys completed on a computer or laptop, due to the smaller screen size, longer page loading times, and more distraction, we expect that motivated misreporting is more pronounced on smartphones. Furthermore, we expect that misreporting occurs not only in the filter questions themselves but also extends to data quality in the follow-up questions. We randomly assigned 3,517 respondents of a German online access panel to either the PC or the smartphone. Our results show that while both PC and smartphone respondents trigger fewer filter questions in the interleafed format than the grouped format, we did not find differences between PC and smartphone respondents regarding the number of triggered filter questions. However, smartphone respondents provide lower data quality in the follow-up questions, especially in the grouped format. We conclude with recommendations for web survey designers who intend to incorporate smartphone respondents in their surveys.