Search results

"Survey designers often ask about the best length for their questionnaires and the best format for their questions. Much research has already addressed these issues. However, the answers to these questions may vary with the population of interest, the mode of data collection used, and other factors. The goal of this paper is twofold: 1. To give an overview of the present situation in opt-in online panels, in terms of survey characteristics, participation, and evaluation, by reviewing 186 surveys managed by the panel company Netquest in Spain in 2016. This will be useful to determine areas where further research needs to focus. 2. To study how key characteristics of questionnaires impact survey evaluation and levels of survey break-off. This will allow us to highlight the characteristics that best reduce break-off and improve respondents' survey evaluation. Based on these results, we will propose practical recommendations for future survey design within the framework of opt-in online panels." (author's abstract)

"Measurement equivalence is a pre-requisite to be able to make comparisons across groups. In this paper the authoress is interested in testing measurement equivalence across respondents answering surveys done using different modes of data collection. Indeed, different modes of data collection have specific characteristics that may create measurement non-equivalence across modes. If this is so, data collected in different modes cannot be compared. This would be problematic since, in order to respond to new challenges, like costs and time pressure, more and more often researchers choose to use different modes to collect their data across time, across surveys, and across countries. Studying data about trust and attitudes towards immigration, this paper shows that measurement equivalence holds across a face-to-face and a web survey done in the Netherlands (2008-2009). Moreover, the quality estimates of the Composite Scores are quite high and pretty similar in the two surveys for the four concepts considered." (author's abstract)

This dissertation studies the impact of the mode of data collection on the quality of answers to survey questions, defined as the product of reliability and validity. Using data from the Netherlands about different topics (media, social and political trust, satisfaction, political orientation, left-right self-placement, attitudes toward immigration), it shows that the quality is similar in a computed assisted face-to-face survey using show cards (the European Social Survey, ESS) and a web survey based on a probability sample (the LISS panel). This is true both at the level of single items and composite scores. It suggests that standardised relationships across variables can be compared across these two modes. On the contrary, telephone interviews lead to some differences in quality. For complex concepts, measurement equivalence also holds, meaning that means and unstandardised relationships can be compared across the faceto- face and web surveys mentioned previously. ; Esta tesis estudia el impacto que el método de recolección de datos en encuestas tiene sobre la calidad de las respuestas, definida como el producto de la fiabilidad y la validez. Utilizando datos de Holanda sobre temas diversos (utilización de los medios de comunicación, confianza social y política, satisfacción, orientaciones políticas, autoubicación en la escala izquierda-derecha, actitudes hacia la inmigración), se muestra que la calidad es similar en una encuesta cara-a-cara asistida con ordenador y utilizando tarjetas (la Encuesta Social Europea) y una encuesta online basada en una muestra probabilística (el panel LISS). Esto se cumple tanto para los indicadores simples, como para indicadores complejos. Los resultados sugieren que las relaciones estandardizadas entre variables son comparables entre los dos métodos de recolección. Al contrario, las entrevistas telefónicas producen diferencias de calidad. Para conceptos complejos, la equivalencia de las mediciones también está garantizada: las medias y las relaciones no estandarizadas son ...

"So far, most surveys used face-to-face or telephone questionnaires in order to collect data. But the costs of achieving a survey using these traditional modes increase. At the same time, the response rates decrease, making the idea of switching mode very attractive. Because each mode has its own weaknesses and strengths, the idea of mixing modes of data collection is becoming more and more popular. Nevertheless, combining different modes of data collection may be problematic if people answer differently depending on the mode. Also, a switch from a unimode to a mixed-mode design may threaten the comparability of the data across time. This paper focuses first on the selection effect and shows that different kinds of respondents answer in different modes: therefore, mixing modes might make sense since it may improve the representativeness of the sample keeping the costs low. It is still necessary however to guarantee that mixing modes would not threaten the comparability. Then, the paper therefore compares the quality of questions asked in a unimode and two mixed-mode surveys. Using data of the European Social Survey (ESS) in the Netherlands, and following a multitrait-multimethod approach (MTMM), few differences are found between the unimode and mixed-mode designs in terms of quality. Looking at the differences across modes lead to slightly less similarities, but overall the quality does not change much." (author's abstract)

AbstractThe Satisfaction With Democracy (SWD) indicator is very often used in social sciences' research. However, while there is debate about which concept it measures, the discussion about the size of its measurement errors (how well it measures the underlying concept 'satisfaction with the way democracy works') is scarce. Nonetheless, measurement errors can affect the results and threaten comparisons across studies, countries and languages. Thus, in this paper, we estimated the measurement quality (complement of measurement errors) of the SWD indicator for 7 response scales across 38 country-language groups, using three multitrait-multimethod experiments from the European Social Survey. Results show that measurement errors explain from 16% (11-point scale) to 54% (4-point scale) of the variance in the observed responses. Additionally, we provide insights to improve questionnaire design and evaluate the indicator's comparability across scales, countries and languages.

Abstract
Time and cost pressures, the availability of alternative sources of data, and societal changes are leading to a move from traditional face-to-face surveys to web or mixed-mode data collection. While we know that there are mode differences between web and face-to-face (presence of an interviewer or not, type of stimuli, etc.), it is not clear to what extent these differences could threaten the comparability of data collected in face-to-face and web surveys. In this article, we investigate the differences in measurement quality between the European Social Survey (ESS) Round 8 and the CROss-National Online Survey (CRONOS) panel. We address three main research questions: (1) Do we observe differences in terms of measurement quality across face-to-face and web for the same people and questions? (2) Can we explain individual-level differences in data quality using respondents' characteristics? and (3) Does measurement equivalence (metric and scalar) hold across the ESS Round 8 and the CRONOS panel? The results suggest that: (1) in terms of data quality, the measurement mode effect between web and face-to-face as implemented in the ESS (i.e., using show cards) is not very large, (2) none of the variables considered consistently explain individual differences in mode effects, and (3) measurement equivalence often holds for the topics studied.

Little is known about the measurement quality of questions in web surveys, even if, this information is crucial to design better questionnaires and to correct for measurement errors in substantive analyses. This paper aims to cover this gap by answering the following four objectives. The first objective, is to evaluate the measurement quality of a set of survey questions from two Multitrait-Multimethod (MTMM) experiments implemented in the 5th wave of the Norwegian Citizen Panel ; one of the few probability-based online panels existing at this day. Each experiment is designed to evaluate three different formulations of the response scale for the topics: political satisfaction and trust in the institutions. The second objective is to predict the measurement quality of these questions by its design characteristics, using the software Survey Quality Predictor (SQP). The third, is to compare the quality of the different formulations of the response scale used. The fourth, is to compare both the MTMM and the SQP approaches to assess whether both can lead to similar results when evaluating web survey questions. Overall, measurements' quality is quite high (between 0.60 and 0.89), and similar between the estimates obtained from the MTMM experiments and the SQP predictions. On the one hand, we conclude that when comparing the different scales, the horizontal 11-point scale with 2 fixed reference points and ordered from negative to positive, usually, provides the highest quality. On the other hand, we conclude that SQP can provide as accurate quality predictions as MTMM can estimate the quality for web survey questions. Given that each approach has its advantages and limitations, when possible we recommend using both to correct for measurement errors, as kind of sensitivity analysis.

Little is known about the measurement quality of questions in web surveys, even if, this information is crucial to design better questionnaires and to correct for measurement errors in substantive analyses. This paper aims to cover this gap by answering the following four objectives. The first objective, is to evaluate the measurement quality of a set of survey questions from two Multitrait-Multimethod (MTMM) experiments implemented in the 5th wave of the Norwegian Citizen Panel ; one of the few probability-based online panels existing at this day. Each experiment is designed to evaluate three different formulations of the response scale for the topics: political satisfaction and trust in the institutions. The second objective is to predict the measurement quality of these questions by its design characteristics, using the software Survey Quality Predictor (SQP). The third, is to compare the quality of the different formulations of the response scale used. The fourth, is to compare both the MTMM and the SQP approaches to assess whether both can lead to similar results when evaluating web survey questions. Overall, measurements' quality is quite high (between 0.60 and 0.89), and similar between the estimates obtained from the MTMM experiments and the SQP predictions. On the one hand, we conclude that when comparing the different scales, the horizontal 11-point scale with 2 fixed reference points and ordered from negative to positive, usually, provides the highest quality. On the other hand, we conclude that SQP can provide as accurate quality predictions as MTMM can estimate the quality for web survey questions. Given that each approach has its advantages and limitations, when possible we recommend using both to correct for measurement errors, as kind of sensitivity analysis.

"More and more respondents use mobile devices to complete web surveys. These devices have different characteristics, if compared to PCs (e.g. smaller screen sizes and higher portability). These characteristics can affect the survey responses, mostly when a questionnaire includes sensitive questions. This topic was already studied by Mavletova and Couper (2013), through a two-wave experiment comparing PCs and mobile devices results for the same respondents in a Russian opt-in panel. We replicated this cross-over design, focusing on an opt-in panel for Spain, involving 1,800 panellists and comparing PCs and smartphones. Our results support most of Mavletova and Couper's (2013) findings (e.g. generally the used device does not significantly affect the reporting of sensitive information), confirming their robustness over the two studied countries. For other results (e.g. trust in data confidentiality), we found differences that can be justified by the diverse context/culture or by the quick changes that are still characterizing the mobile web survey participation." (author's abstract)

The formulation of theories and hypotheses is done at the level of concepts. These concepts are often tested by operationalizing them using survey questions. However, measurement errors make it impossible for survey questions to measure the concepts of interest perfectly. In order to correct for measurement errors, information is needed about their size, or the size of their complement, the quality. For the USA and Europe, a lot is already known about the quality of questions, but this has not yet been studied in some other parts of the world. In this paper, we use a multitrait-multimethod approach to estimate the quality of 27 questions in Mexico and Colombia. These initial results on quality for Central and Latin American countries show quality estimates that are relatively similar in terms of their relationships with the scale characteristics to what has been observed in the USA and Europe. (JPLA)

La mayoría de la investigación social estudia conceptos abstractos (p. ej., actitudes) mediante preguntas de encuestas. No obstante, las encuestas adolecen de errores de medición que afectan a las conclusiones sustantivas. Cuando dichos errores difieren entre países, comparar relaciones estadísticas estandarizadas entre países puede resultar en conclusiones incorrectas. Sin embargo, la calidad de medición de las preguntas de encuestas en España no ha sido investigada de forma comparada. Utilizando un experimento MultiRasgo-MultiMétodo, realizado en la Encuesta Social Europea (ESS), comparamos la calidad de las preguntas en España con la de otros países. En general, la calidad de medición en España es superior a la mayoría de países participantes. Además, si no se tienen en cuenta los errores de medición al comparar España con otros países, las conclusiones sustantivas pueden ser erróneas.

Abstract
It is common practice in behavioral and social sciences to repeat questions, such as in pretest–posttest designs. However, if respondents recall their first answer to a repeated question and use it to decide their second answer, this may cause memory effects, affecting empirical findings. So far, only a few studies have investigated respondents' recall ability and memory effects in surveys. Thus, we conducted an experiment in a probability-based online panel (N = 4,681) where we repeatedly asked a question on political interest. Our results reveal that respondents' recall ability is high and was not reduced by the implementation of memory interference tasks. Memory effects, in contrast, were low (about 7%). They were also not reduced by memory interference tasks.

Revilla and Saris compare a face-to-face and a web survey in terms of quality, defined as the strength of the relationships between the latent and the observed variables. The comparison is done between the European Social Survey (ESS), where the data is collected by face-to-face interviews at respondents' homes, and a study completed by the respondents of the Longitudinal Internet Studies for the Social Sciences (LISS) panel. Since the LISS is an online Dutch panel, whereas the ESS is conducted in many countries in order to avoid variations due to cultural or language differences, they focus only on the Netherlands. Adapted from the source document.