How Much Academic Growth?

Recently, a colleague in England very politely called me on the carpet for something I wrote in 1981 that did not mercifully die in the interim. Here is his question.

Dear Dr. Glass

As you may know, the Education Endowment Foundation in England, in order to help with interpreting effect sizes in education, has suggested to teachers that an effect size of one standard deviation is approximately equal to one year’s progress, and they cite the following from page 103 of your 1981 book ‘Meta-analysis in education’ in support:

It is also known, as an empirical—not definitional—fact that the standard deviation of most achievement tests in elementary school is 1.0 grade-equivalent units; hence the effect size of one year’s instruction at the elementary school level is about +1, for example,

∆ = (4.0 - 3.0) / 1.0 = +1.

I asked one of your co-authors, Barry McGaw (whom I know well) if he could recall where this figure came from and he could not, hence this email to you.

The reason I ask is that for many of the tests currently in use in elementary schools in the US, one year’s progress is considerably less than one standard deviation, and for students in the upper elementary school (say grades 4 and 5) the available evidence, for example from Howard Bloom and his colleagues, suggest that one year’s typical progress is around 0.4 standard deviations.

Any light you can shed on this would be appreciated.

Sincerely, Dylan Wiliam
Dylan Wiliam BSc, BA, MSc, PhD, FRSA, AcSS, MBPsS
Emeritus Professor of Educational Assessment
University College London
20 Bedford Way
London WC1H 0AL

It took me a day or two to dig up some data pertinent to the question that Dylan asked. Thanks to the power of the world wide web, What was needed was near at hand.

Hi, Dylan
Sorry to be slow.

Yes, I was flying a bit blind in 1981 when I put that stuff in the book. I hope it was not ever used to anyone's disadvantage.

When I came up with "one year's growth is about 1.0 effect size," I was looking at a bunch of achievement test data in the early elementary grades that were easily available. What is pretty clear now is that "effect-size growth" for one year of schooling is about 1.0 in early grades (1-3) and steadily declines as you go up the grades. Because the st-dev of Reading is usually a bit greater than that for Math (for several reasons, some obvious), the effect-size growth is a little greater in Math and in Reading for one year's instruction for most grades.

Here are a couple of excerpts from available reports that report sufficient information to calculate effect-size growth for one year for several grades.
The first table is from a report by Michael Russell at Boston College. Here's the URL: https://pareonline.net/htm/v7n6.htm

So, it looks like effect-size growth (ESG) is in the vicinity of 1 in the early grades and declines as the grades increase. And, Math ESG is a little bigger than Reading.

Here are more data showing the same trends.These data are from the norming of the Iowa Test of Basic Skills: https://itp.education.uiowa.edu/ia/documents/Measuring-Growth-with-the-Iowa-Assessments.pdf

Let's look at one example. Math growth from Grade 6 to 7.

∆ = (250 - 232)/28.3 = .63

So, the average gain in Math achievement from the 6th to the 7th grade is about two-thirds of a standard deviation. In other words, the average 6th grade student moves from the 50th percentile to the 75th percentile of the 6th grade norms in one year. Also, the table shows the same trends that ones sees in other data: Smaller than 1.0 ESG in the intermediate grades; ESGs a little bigger in Math than in Reading.

I should have been more circumspect in 1980 when I was tossing off opinions without sufficient serious study. I was guilty of the same loose talk that I have so abhorred in the writings of people like Cohen who wrote that ES = .25 is "Small" and ES = .5 is "Medium," etc. It all depends!

Hope this helps.

Gene

I'm more sympathetic to those writers who leave their descendants instructions to burn everything they have written.

Gene V Glass San José State University

Arizona State University

University of Colorado Boulder, National Education Policy Center

The opinions expressed here are those of the author and do not represent the official position of San José State University, the National Education Policy Center, Arizona State University, or the University of Colorado Boulder.