SEDL Researchers Contribute to an Article Proposing a New Method of Sample Selection in Randomized Experiments

April 22, 2014
Austin, Texas

Laura Shankland
Communications Associate
Phone: 512-391-6556

Since 2009, with support from the Institute of Education Sciences, researchers at SEDL and the University of Wisconsin-Madison have been conducting two large scale-up randomized controlled trial studies of SRA Imagine It! Today’s Open Court reading program and Everyday Mathematics® to determine the programs’ impacts on teacher practices and student academic achievement. Both programs have prior evidence demonstrating their effectiveness, but past studies have been limited in scope and generalizability of findings. The SEDL study is national and will provide findings that researchers intend to apply to a much larger population.

The article published in Journal of Research on Educational Effectiveness, with the title “Sample Selection in Randomized Experiments: A New Method Using Propensity Score Stratified Sampling,” presents an innovative method of sample selection in randomized experiments to empirically increase the generalizability of study findings. The lead author, Elizabeth Tipton of Columbia University, and the co-author, Larry Hedges of Northwestern University, have been working with SEDL researchers on these projects as part of the technical working group. Three SEDL researchers—Michael Vaden-Kiernan, Kate Sullivan, and Sarah Caverly—contributed to the article. Geoffrey Borman of University of Wisconsin-Madison, who is a partner with SEDL on these projects, was also a co-author.

One of the challenges of a scale-up study is ensuring that the study sample is representative of the target population you wish to generalize the findings of the study to. Researchers often take a sample and then determine if the study sample is representative of the target population. Tipton et al. present a new method for sample selection, which makes it easier to ensure that the sample can be generalized to the target population from the onset of the study. The authors propose that researchers very explicitly define the target population and how this population relates to the definition of eligibility for the experiment. This process is a combination of stratified sampling—dividing the target population (i.e., districts and schools likely to adopt Imagine It! and Everyday Mathematics®) into different categories relevant to the research variables of interest, such as school size, location, and socio-economic status of the student population—and propensity score matching methods—matching eligible members of the sample population with districts and schools that are already using Imagine It! and Everyday Mathematics®. This methodology enables researchers to identify and conduct sample selection and recruitment with the target population in mind from the beginning of the study.

The method presented by the authors is flexible and practical, and it helps researchers determine which areas of the population may be the most critical and most difficult to recruit from, enabling recruitment resources to be deployed strategically throughout the recruitment period.

Read the original article (subscription required).

About SEDL

SEDL (formerly Southwest Educational Development Laboratory) is a nonprofit corporation based in Austin, Texas. SEDL is dedicated to solving significant education problems and improving teaching and learning through research, research-based resources, and professional development. For more information about SEDL, visit