Suchergebnisse

Pretesting survey questions via cognitive interviewing is based on the assumptions that the problems identified by the method truly exist in a later survey and that question revisions based on cognitive interviewing findings produce higher-quality data than the original questions. In this study, we empirically tested these assumptions in a web survey experiment (n = 2,200). Respondents received one of two versions of a question on self-reported financial knowledge: either the original draft version, which was pretested in ten cognitive interviews, or a revised version, which was modified based on the results of the cognitive interviews. We examined whether the cognitive interviewing findings predicted problems encountered in the web survey and whether the revised question version was associated with higher content-related and criterion-related validity than the draft version. The results show that cognitive interviewing is effective in identifying real question problems, but not necessarily in fixing survey questions and improving data quality. Overall, our findings point to the importance of using iterative pretesting designs, that is, carrying out multiple rounds of cognitive interviews and also testing the revisions to ensure that they are indeed of higher quality than the draft questions.

The literature on human-computer interaction consistently stresses the importance of reducing the cognitive effort required by users who interact with a computer in order to improve the experience and enhance usability and comprehension. Applying this perspective to web surveys, questionnaire designers are advised to strive for layouts that facilitate the response process and reduce the effort required to select an answer. In this article, we examine whether placing the answer boxes (i.e., radio buttons or check boxes) to the left or to the right of the answer options in closed questions with vertically arranged response categories enhances usability and facilitates responding. First, we discuss a set of opposing principles of how respondents may process these types of questions in web surveys, some suggesting placing the answer boxes to the left and others suggesting placing them to the right side of the answer options. Second, we report an eye-tracking experiment that examined whether web survey responding is best described by one or another of these principles, and consequently whether one of the three layouts is preferable in terms of usability: (1) answer boxes to the left of left-aligned answer options, (2) answer boxes to the right of left-aligned answer options, and (3) answer boxes to the right of right-aligned answer options. Our results indicate that the majority of respondents conform to a principle suggesting placing the answer boxes to the left of left-aligned answer options. Moreover, respondents require less cognitive effort (operationalized by response latencies, fixation times, fixation counts, and number of gaze switches between answer options and answer boxes) to select an answer in this layout.

An important objective in survey question design is to write clear questions that respondents find easy to understand and to answer. This contribution identifies the factors that influence question clarity. Theoretical and empirical evidence from psycholinguistics suggests that specific text features (e.g., low-frequency words, left-embedded syntax) cause comprehension difficulties and impose a high cognitive burden on respondents. To examine the effect of seven different text features on question clarity, an online experiment was conducted in which well-formulated questions were compared to suboptimal counterparts. The cognitive burden of the questions was assessed with response times. Data quality was compared in terms of drop-out rates and survey satisficing behavior. The results show that at least six of the text features are relevant for the clarity of a question. We provide a detailed explanation of these text features and advise survey designers to avoid them when crafting questions.

Abstract
Cognitive interviewing in the form of probing is key for developing methodologically sound survey questions. For a long time, probing was tied to the laboratory setting, making it difficult to achieve large sample sizes and creating a time-intensive undertaking for both researchers and participants. Web surveys paved the way for administering probing questions over the Internet in a time- and cost-efficient manner. In so-called web probing studies, respondents first answer a question and then they receive one or more open-ended questions about their response process, with requests for written answers. However, participants frequently provide very short or no answers at all to open-ended questions, in part because answering questions in writing is tedious. This is especially the case when the web survey is completed via a smartphone with a virtual on-screen keypad that shrinks the viewing space. In this study, we examine whether the problem of short and uninterpretable answers in web probing studies can be mitigated by asking respondents to complete the web survey on a smartphone and to record their answers via the built-in microphone. We conducted an experiment in a smartphone survey (N = 1,001), randomizing respondents to different communication modes (written or oral) for answering two comprehension probes about two questions on national identity and citizenship. The results indicate that probes with requests for oral answers produce four to five times more nonresponse than their written counterparts. However, oral answers contain about three times as many words, include about 0.3 more themes (first probing question only), and the proportion of clearly interpretable answers is about 6 percentage points higher (for the first probing question only). Nonetheless, both communication modes result in similar themes mentioned by respondents.

The use of agree/disagree (A/D) questions is a common technique to measure attitudes. For instance, this question format is employed frequently in the Eurobarometer and International Social Survey Programme (ISSP). Theoretical considerations, however, suggest that A/D questions require a complex processing. Therefore, many survey researchers have
recommended the use of item-specific (IS) questions, since they seem to be less burdensome. Parallel to this methodological discussion is the discussion around the use of mobile devices for responding to surveys. However, until now, evidence has been lacking as to whether the use of mobile devices for survey response affects the performance of established question
formats. In this study, implemented in the Netquest panel in Spain (N = 1,476), we investigated the cognitive effort and response quality associated with A/D and IS questions across PCs and smartphones. For this purpose, we applied a split-ballot design defined by device type and question format. Our analyses revealed longer response times for IS questions than A/D
questions, irrespective of the device type and scale length. Also, the IS questions produced better response quality than their A/D counterparts. All in all, the findings indicate a more conscientious response to IS questions compared to A/D questions.

Many test constructions show a lack of awareness concerning the comprehensibility of items, specifically in personality tests. However, this lack of awareness can have implications for psychometric quality. Frequently, it is simply assumed that respondents' understanding of a test item matches the meaning implied by the test developer. It is, however, rarely ever tested whether all respondents of the targeted population actually understand the test items correctly and in a similar way. As pointed out in the last editorial (Ziegler, 2014), item content should be precisely tuned to the needs of every potential respondent in a population targeted by a newly developed test. More specifically, person variables such as age, gender, and education (Rammstedt & Kemper, 2011) have to be taken into account when constructing test items in order to assure that each potential respondent fully understands the meaning and may respond accordingly. If these person variables are neglected in the process of test construction, the psychometric quality of a test may be substantially affected by fuzzy concepts. The goal of this editorial is to raise awareness for the detrimental effects fuzzy concepts can have in test development and possible remedies. (author's abstract)

Web surveys are a common self-administered mode of data collection using written language to convey information. This language is usually accompanied by visual design elements, such as numbers, symbols, and graphics. As shown by previous research, such elements of survey questions can affect response behavior because respondents sometimes use interpretive heuristics, such as the "middle means typical" and the "left and top means first" heuristics when answering survey questions. In this study, we adopted the designs and survey questions of two experiments reported in Tourangeau, Couper, and Conrad (2004). One experiment varied the position of nonsubstantive response options in relation to other substantive response options and the second experiment varied the order of the response options. We implemented both experiments in an eye-tracking study. By recording respondents' eye movements, we are able to observe how they read question stems and response options and we are able to draw conclusions about the survey response process the questions initiate. This enables us to investigate the mechanisms underlying the two interpretive heuristics and to test the assumptions of Tourangeau et al. (2004) about the ways in which interpretive heuristics influence survey responding. The eye-tracking data reveal mixed results for the two interpretive heuristics. For the middle means typical heuristic, it remains somewhat unclear whether respondents seize on the conceptual or visual midpoint of a response scale when answering survey questions. For the left and top means first heuristic, we found that violations of the heuristic increase response effort in terms of eye fixations. These results are discussed in the context of the findings of the original studies.

This contribution provides an overview of the basic rules that should be observed when formulating survey questions. The structure of the chapter is oriented towards the cognitive response process that respondents undergo when answering survey questions. This process involves a total of four tasks: understanding the question text, retrieving information from memory, forming a judgment, and formulating a response in the format provided. Using practical examples, the authors demonstrate how questions can be formulated in such a way that it makes it easier for respondents to handle these four tasks.

Dieser Beitrag gibt einen Überblick über grundsätzliche Regeln, die bei der Formulierung von Survey-Fragen berücksichtigt werden sollten. Die Gliederung des Kapitels orientiert sich dabei am kognitiven Antwortprozess, den Befragte bei der Beantwortung von Survey-Fragen durchlaufen und der insgesamt vier Aufgaben umfasst: Verstehen des Fragetextes, Informationsgewinnung, Urteilsbildung und Formatierung der Antwort. Anhand von praktischen Beispielen wird illustriert, wie man mit Hilfe der Frageformulierung Befragten die Bewältigung dieser vier Aufgaben erleichtern kann.

AbstractWeb surveys are a common self-administered mode of data collection using written language to convey information. This language is usually accompanied by visual design elements, such as numbers, symbols, and graphics. As shown by previous research, such elements of survey questions can affect response behavior because respondents sometimes use interpretive heuristics, such as the "middle means typical" and the "left and top means first" heuristics when answering survey questions. In this study, we adopted the designs and survey questions of two experiments reported in Tourangeau, Couper, and Conrad (2004). One experiment varied the position of nonsubstantive response options in relation to other substantive response options and the second experiment varied the order of the response options. We implemented both experiments in an eye-tracking study. By recording respondents' eye movements, we are able to observe how they read question stems and response options and we are able to draw conclusions about the survey response process the questions initiate. This enables us to investigate the mechanisms underlying the two interpretive heuristics and to test the assumptions of Tourangeau et al. (2004) about the ways in which interpretive heuristics influence survey responding. The eye-tracking data reveal mixed results for the two interpretive heuristics. For the middle means typical heuristic, it remains somewhat unclear whether respondents seize on the conceptual or visual midpoint of a response scale when answering survey questions. For the left and top means first heuristic, we found that violations of the heuristic increase response effort in terms of eye fixations. These results are discussed in the context of the findings of the original studies.