Readers are often – and understandably – frustrated when it comes to reports of educational experiments and effect sizes. Let’s say a given program had an effect size of +0.30 (or 30% of a standard deviation). Is that large? Small? Is the program worth doing or worth forgetting? There is no simple answer, because it depends on the quality of the study.
The Institute of Education Sciences recently issued a report by Mark Lipsey and colleaguesfocusing on how to interpret effect sizes. It is very useful for researchers, as intended, but still not so useful for readers of research. I wanted to point out some key conclusions of this report and a few additional thoughts.
First, the Lipsey et al. report dismisses the old Cohen characterization of effect sizes of +0.20 as “small,” +0.50 as “moderate,” and +0.80 as “large.” Those numbers applied to small, highly controlled lab studies. In real-life educational experiments with broad measures of achievement and random assignment to treatments, effect sizes as large as +0.50, much less +0.80, are hardly ever seen, except on occasion in studies of one-to-one tutoring.
The larger issue is that studies vary in quality, and many features of studies give hugely inflated estimates of effect sizes. In order of likely importance, here are some factors to watch for:
- Use of measures made by the researchers.
- Very brief studies (often, one hour or less)
- Studies with small sample sizes
Studies that incorporate any of these elements can easily produce effect sizes of +1.00 or more. Such studies should be disregarded by readers serious about knowing what works in real classrooms and what does not.
- Slavin: Overcoming Four Barriers to Evidence-Based Education - May 6, 2013
- Sputnik: Taking the guesswork out of policy - January 15, 2013
- Sputnik: Effect size matters in educational research - January 9, 2013
- Sputnik: Technology without supports – like cotton candy for breakfast - January 4, 2013
- Sputnik: Transforming Low-Performing Schools - December 13, 2012
- Sputnik: - November 12, 2012
- Sputnik: It’s the Right Time to do the Right Thing - November 2, 2012
- Sputnik: Sunset for Textbook Adoption in California - November 2, 2012
- Sputnik: What the Presidential Election means for Evidenced-based reform - October 25, 2012
- Sputnik: Won’t Back Down – Union-bashing goes Hollywood - October 12, 2012