Search results

Near the end of a web survey respondents are often asked whether they have additional comments. Such final comments are usually ignored, partially because open-ended questions are more challenging to analyze. A random sample of final comments in the LISS panel and Dutch immigrant panel were categorized into one of nine categories (neutral, positive, multiple subcategories of negative). While few respondents chose to make a final comment, this is more common in the Immigrant panel (5.7%) than in the LISS panel (3.6%). In both panels there are slightly more neutral than negative comments, and very few positive comments. The number of final comments about unclear questions was 2.7 times larger in the immigrant panel than in the LISS panel. The number of final comments complaining about survey length on the other hand was 2.7 times larger in the LISS panel than in the immigrant panel. Researchers might want to consider additional pretesting of questions when fielding a questionnaire in the immigrant panel.

Abstract
Manual coding of text data from open-ended questions into different categories is time consuming and expensive. Automated coding uses statistical/machine learning to train on a small subset of manually-coded text answers. Recently, pretraining a general language model on vast amounts of unrelated data and then adapting the model to the specific application has proven effective in natural language processing. Using two data sets, we empirically investigate whether BERT, the currently dominant pretrained language model, is more effective at automated coding of answers to open-ended questions than other non-pretrained statistical learning approaches. We found fine-tuning the pretrained BERT parameters is essential as otherwise BERT is not competitive. Second, we found fine-tuned BERT barely beats the non-pretrained statistical learning approaches in terms of classification accuracy when trained on 100 manually coded observations. However, BERT's relative advantage increases rapidly when more manually coded observations (e.g., 200–400) are available for training. We conclude that for automatically coding answers to open-ended questions BERT is preferable to non-pretrained models such as support vector machines and boosting.

The data can be used to study the classification of open-ended questions. It is not suitable for subject matter analysis becase it is not linked to any other data.

The question asked: "Joe's doctor told him that he would need to return in two weeks to find out whether or not his condition had improved. But when Joe asked the receptionist for an appointment, he was told that it would be over a month before the next available appointment.
What should Joe do? "
The answer was categorized by two coders into one of 4 categories: destructive, passive, somewhat proactive, proactive. The coding book is supplied.
The variables include the categorization by the two coders, as well as the resolved category.

Abstract
Text answers to open-ended questions are typically manually coded into one of several codes. Usually, a random subset of text answers is double-coded to assess intercoder reliability, but most of the data remain single-coded. Any disagreement between the two coders points to an error by one of the coders. When the budget allows double coding additional text answers, we propose employing statistical learning models to predict which single-coded answers have a high risk of a coding error. Specifically, we train a model on the double-coded random subset and predict the probability that the single-coded codes are correct. Then, text answers with the highest risk are double-coded to verify. In experiments with three data sets, we found that this method identifies two to three times as many coding errors in the additional text answers as compared to random guessing, on average. We conclude that this method is preferred if the budget permits additional double-coding. When there are a lot of intercoder disagreements, the benefit can be substantial.

Text answers to open-ended questions are often manually coded into one of several predefined categories or classes. More recently, researchers have begun to employ statistical models to automatically classify such text responses. It is unclear whether such automated coders and human coders find the same type of observations difficult to code or whether humans and models might be able to compensate for each other's weaknesses. We analyze correlations between estimated error probabilities of human and automated coders and find: 1) Statistical models have higher error rates than human coders 2) Automated coders (models) and human coders tend to make similar coding mistakes. Specifically, the correlation between the estimated coding error of a statistical model and that of a human is comparable to that of two humans. 3) Two very different statistical models give highly correlated estimated coding errors. Therefore, a) the choice of statistical model does not matter, and b) having a second automated coder would be redundant.

Abstract
Asking people about their occupation is common practice in surveys and censuses around the world. The answers are typically recorded in textual form and subsequently assigned (coded) to categories, which have been defined in official occupational classifications. While this coding step is often done manually, substituting it with more automated workflows has been a longstanding goal, promising reduced data-processing costs and accelerated publication of key statistics. Although numerous researchers have developed different algorithms for automated occupation coding, the algorithms have rarely been compared with each other or tested on different data sets. We fill this gap by comparing some of the most promising algorithms found in the literature and testing them on five data sets from Germany. The first two algorithms we test exemplify a common practice in which answers are coded automatically according to a predefined list of job titles. Statistical learning algorithms—that is, regularized multinomial regression, tree boosting, or algorithms developed specifically for occupation coding (algorithms three to six)—can improve upon algorithms one and two, but only if a sufficient number of training observations from previous surveys is available. The best results are obtained by merging the list of job titles with coded answers from previous surveys before using this combined training data for statistical learning (algorithm 7). However, the differences between the algorithms are often small compared to the large variation found across different data sets, which we ascribe to systematic differences in the way the data were coded in the first place. Such differences complicate the application of statistical learning, which risks perpetuating questionable coding decisions from the training data to the future.

AbstractMany undergraduate degrees require the completion of an introductory statistics course, but it is unclear to what extent taking a statistics course improves statistical literacy. We conducted an online survey with a simple random sample of undergraduate students at the University of Waterloo, Canada. We then compared students who have completed at least one statistics course to those who have completed none and found that taking a statistics course did not improve statistical literacy on the questions asked (causation versus correlation, margin of error, and others). Introductory statistics courses may want to address statistical literacy as a learning outcome more explicitly for a better understanding of public policies.

"Near the end of a web survey respondents are often asked whether they have further comments. Such final comments are usually ignored, in part because open-ended questions are challenging to analyse. We explored whether final comments are associated with next-wave attrition in survey panels. We categorized a random sample of final comments in the Longitudinal Studies for the Social Sciences (LISS) panel and Dutch Immigrant panel into one of eight categories (neutral, positive, six subcategories of negative) and regressed the indicator of next-wave attrition on comment length, comment category and socio-demographic variables. In the Immigrant panel we found shorter final comments (55 words) with decreased next-wave attrition relative to making no comment. Comments about unclear survey questions quadruple the odds of attrition and 'other' (uncategorized) negative comments almost double the odds of attrition. In the LISS panel, making a comment (vs. not) and comment length are not associated with attrition. However, when specifying individual comment categories, neutral comments are associated with half the odds of attrition relative to not making a comment." (author's abstract)

"Straightlining, an indicator of satisficing, refers to giving the same answer in a series of questions arranged on a grid. We investigated whether straightlining changes with respondents' panel experience in two open-access Internet panels in the Netherlands: the LISS and Dutch Immigrant panels. Specifically, we considered straightlining on 10 grid questions in LISS core modules (7 waves) and on a grid of evaluation questions in both the LISS panel (150+ waves) and the Dutch immigrant panel (50+ waves). For both core modules and evaluation questions we found that straightlining increases with respondents' panel experience for at least three years. Straightlining is also associated with younger age and non-western 1st generation immigrants. Where straightlining was a plausible set of answers, prevalence of straightlining was much larger (15-40%) than where straightlining was implausible (<2% in wave 1)." (author's abstract)

"Text data from open-ended questions in surveys are difficult to analyze and are frequently ignored. Yet open-ended questions are important because they do not constrain respondents' answer choices. Where open-ended questions are necessary, sometimes multiple human coders hand-code answers into one of several categories. At the same time, computer scientists have made impressive advances in text mining that may allow automation of such coding. Automated algorithms do not achieve an overall accuracy high enough to entirely replace humans. We categorize open-ended questions soliciting narrative responses using text mining for easy-to-categorize answers and humans for the remainder using expected accuracies to guide the choice of the threshold delineating between 'easy' and 'hard'. Employing multinomial boosting avoids the common practice of converting machine learning 'confidence scores' into pseudo-probabilities. This approach is illustrated with examples from open-ended questions related to respondents' advice to a patient in a hypothetical dilemma, a follow-up probe related to respondents' perception of disclosure/privacy risk, and from a question on reasons for quitting smoking from a follow-up survey from the Ontario Smoker's Helpline. Targeting 80% combined accuracy, we found that 54%-80% of the data could be categorized automatically in research surveys." (author's abstract)

"In household panels, typically all household members are surveyed. Because household composition changes over time, so-called following rules are implemented to decide whether to continue surveying household members who leave the household (e.g. former spouses/ partners, grown children) in subsequent waves. Following rules have been largely ignored in the literature leaving panel designers unaware of the breadth of their options and forcing them to make ad hoc decisions. In particular, to what extent various following rules affect sample size over time is unknown. From an operational point of view such knowledge is important because sample size greatly affects costs. Moreover, the decision of whom to follow has irreversible consequences as finding household members who moved out years earlier is very difficult. The authors find that household survey panels implement a wide variety of following rules but their effect on sample size is relatively limited for a couple of decades. Even after 25 years, the rule 'follow only wave 1 respondents' still captures 85% of the respondents of the rule 'follow everyone who can be traced back to a wave 1 household through living arrangements' in the SOEP. Once children of permanent sample members start moving out, following such children greatly affects sample size. This effect is noticeable after 25 years in the PSID. Unless attrition is low, there is no danger of an ever expanding panel because even wide following rules do not typically exceed attrition. Grown children of permanent sample members with their own households have a significantly lower attrition rate than first wave respondents in the PSID. Presence of a spouse or a child in a household does not affect attrition; however, presence of other household members significantly increases attrition." (author's abstract)

"Inference from Web surveys may be affected by non-random selection of Web survey participants. One approach to reduce selection bias is to use propensity scores and a parallel phone survey. This approach uses demographic and additional so-called Webographic or lifestyle variables to balance observed differences between Web survey respondents and phone survey respondents. Here the authors investigate some of the Webographic questions used by Harris Interactive, a commercial company specializing in Web surveys. Their Webographic questions include choice of activities such as reading, sports and traveling and perceptions about what would constitute a violation of privacy. The authors use data from an existing probability sample of respondents over 40 who are interviewed over the phone, and a corresponding sample of respondents interviewed over the Web. They find that Webographic questions differentiate between on and offline populations differently than demographic questions. In general, propensity score adjustment of variables in the Web survey works quite well for a number of variables of interest (including home ownership and labor force participation). For two outcomes, (having emotional problems and often experiencing pain) the process of adjusting for demographic variables leads to the discovery of an instance of Simpson's paradox, implying a differential mode effect or differential selection. They interpret this mainly as the result of a mode effect, where sensitive questions are more likely to receive a positive response over the Internet than over the phone." (author's abstract)