Suchergebnisse

The use of web-based surveys to gather information from teachers has become increasingly common primarily based on the premise that they can reduce costs. Yet, relatively, little is known about the quality or cost effectiveness of web-based versus mail surveys for teachers. To study the efficacy of web-based teacher surveys, the author randomly assigned a nationally representative sample of 877 elementary school teachers to a paper or web survey mode. The cost savings from the web administration were not enough to offset the loss of sample due to missing/inaccurate email addresses and the lower response rates that resulted from the web survey.

AbstractWhen attempting to identify educational settings that are most effective in improving student achievement, classroom process (that is, the way in which a teacher interacts with his or her students) is a key feature of interest. Unfortunately, high‐quality assessment of the student‐teacher interaction occurs all too infrequently, despite the critical role that understanding and measuring such processes can play in school improvement.This article discusses the strengths and weaknesses of two common approaches to studying these processes—direct classroom observation and annual surveys of teachers—and then describes the ways in which instructional logs can be used to overcome some of the limitations of these two approaches when gathering data on curriculum content and coverage. Classroom observations are expensive, require extensive training of raters to ensure consistency in the observations, and because of their expense generally cannot be conducted frequently enough to enable the researcher to generalize observational findings to the entire school year or illuminate the patterns of instructional change that occur across the school year. Annual surveys are less expensive but often suffer from self‐report bias and the bias that occurs when teachers are asked to retrospectively report on their activities over the course of a single year. Instructional logs offer a valid, reliable, and relatively cost‐effective alternative for collecting detailed information about classroom practice and can overcome some of the limitations of both observations and annual surveys.

Objective: In this article, we examine whether a well-executed comparative interrupted time series (CITS) design can produce valid inferences about the effectiveness of a school-level intervention. This article also explores the trade-off between bias reduction and precision loss across different methods of selecting comparison groups for the CITS design and assesses whether choosing matched comparison schools based only on preintervention test scores is sufficient to produce internally valid impact estimates. Research Design: We conduct a validation study of the CITS design based on the federal Reading First program as implemented in one state using results from a regression discontinuity design as a causal benchmark. Results: Our results contribute to the growing base of evidence regarding the validity of nonexperimental designs. We demonstrate that the CITS design can, in our example, produce internally valid estimates of program impacts when multiple years of preintervention outcome data (test scores in the present case) are available and when a set of reasonable criteria are used to select comparison organizations (schools in the present case).

The Making Pre-K Count and High 5s studies represent a recent application of a phased two-stage, multi-level design, which was used to examine the effects of two aligned math programs implemented in early childhood settings. The purpose of this paper is to describe the challenges encountered in implementing this two-stage design and to describe approaches to resolving them. We then present a set of sensitivity analyses the study team used to examine the robustness of the findings. During the pre-K year, pre-K centers were randomly assigned either to receive an evidence-based early math curriculum and associated professional development (Making Pre-K Count) or to a pre-K-as-usual control condition. In the kindergarten year, students who had been in Making Pre-K Count program classrooms in pre-K were then individually randomly assigned within schools to small-group supplemental math clubs that were designed to sustain the gains from the pre-K program, or to a business-as-usual kindergarten experience. Making Pre-K Count took place in 69 pre-K sites, comprising 173 classrooms across New York City. High 5s took place in the 24 sites that were part of the public school treatment arm of the Making Pre-K Count study and included 613 students. The study focuses on the effect of the Making Pre-K Count and High 5s programs on children's math skills at the end of kindergarten as measured by two instruments, the Research-Based Early Math Assessment-Kindergarten (REMA-K) and the Woodcock–Johnson Applied Problems test. The multi-armed design, while logistically and analytically challenging, balanced multiple considerations of power, the number of research questions that could be answered, and efficiency of resources. Robustness checks suggest that the design created groups that were both meaningfully and statistically equivalent. Decisions to use a phased multi-armed design should consider both its strengths and weaknesses. While the design allows for a more flexible, expansive research study, it also introduces complexities that need to be addressed both logistically and analytically.