More Things Wrong with International Assessments Like PISA

On February 17th, I posted to this blog some thoughts on the ubiquitous "international assessments" of achivement. From the access logs I can tell that such topics command wide attention. Unfortunately, the attention paid to criticisms of the international assessments is merely the other side of the coin: far too much is made of these virtually meaningless exercises. Why does anyone think that comparing how many kids in the U.S. vs. Slovenia or Singapore solve a math problem will inform the movement toward better schools? Outsiders always think that what they are observing is simpler than it is in reality, whether they are watching someone play a trombone, or hit a tennis ball, or teach a child to read.

The point I was making last February about international assessments had to do with the gullibility of not just the general public but of professional educators themselves. People swallow without a second thought the comparison of a few dozen nations on a test of reading comprehension! Seemingly no one questions the claim that a test can be written in two different languages while controlling for its difficulty. But it is a claim that is patently false. I wrote in February:

There is something far more wrong with these international assessments and comparisons than anyone seems interested in talking about. Think! ... The obvious question is “In what language is the test written?” And the obvious answer is “In the language of that nation.” But who is drawing the obvious conclusion? How in heaven’s name can you construct a reading test in dozens of different languages (English, Hungarian, Norwegian, and yes, Finnish) and be confident that the test is equally difficult in all of these languages? Well, the answer is that you can’t. It should be perfectly obvious to anyone who thinks about it for more than five minutes that it is impossible.

... A recent article in the Scandanavian Journal of Educational Research by Inga Arffman carries the title “Equivalence of translations in international reading literacy studies,” (Vol. 54, No. 1, 37-59). The paper summarizes a study that examined the problems encountered in translating texts in international reading assessments. And in spite of the fact that Arffman is a faculty member of the University of Jyväskylä in Finland—which has every motive possible to believe that PISA Reading assessments are the most valid tests in the history of psychometrics—the conclusion of the research is that '...it will probably never be possible to attain full equivalence of difficulty in international reading literacy studies....' Amen.

And now a few months later comes a great reference from the work of Jerry Bracey — our now departed and forever respected colleague to whom we all owe a continuing debt of gratitude for his marvelous critiques of scientistic buffoonery in education. (I thank David Berliner for bringing this reference to my attention.)

Posted by Susan Williams on February 24, 2012, to a Teachers College Record blog.

Gerald Bracey's research in the 1990s ... exposed the hidden "competitive edge" of Finland's schools: the Finnish language is eentsy weentsy compared to the English language. We have far, far more vocabulary words than they do.

So when a standardized test is internationalized — translated into Finnish, in this case — it gets really easy for the Finnish kids. ... But they're really not outdoing us.

Bracey used the example of an international test question in which students were to tell whether these two words are synonyms or antonyms: pessimistic and sanguine. Only 50% of Americans got that right. But 98% of Finnish kids did. Oh! Are they that much smarter? Are their schools that much better? Noooooo. Finnish has no equivalent for the word "sanguine," so the word substituted for it was a dead giveaway — optimistic. Duhhhh!

Gene V Glass
University of Colorado Boulder
Arizona State University

A Loss for Cyber-Schooling or Just a Regrouping?

On Monday, May 14, Arizona Republican Governor Jan Brewer vetoed SB 1259, the bill written by ALEC and K12 Inc. lobbyists that would have greatly expanded the give-away of Arizona taxpayer dollars to outside profit making corporations. In short, SB 1259 would have required every school district in the state of Arizona to provide a minimum of two online courses per year to any grade 7 – 12 pupil requesting such. The company supplying the cyber-courses would be reimbursed at 100% of a pro-rated per pupil expenditure. Full-time online students in Arizona currently number nearly 40,000, most of the money flowing through online charter schools to K12 Inc. SB 1259 would have opened up a market for the corporations many times larger than the full-time charter market.

What made SB 1259 fly through the Arizona Legislature and onto the Governor’s desk was an accountability provision promising to guarantee mastery of course content. Each online course would have to be accompanied by a final exam that was matched in difficulty to the state-developed AIMS test. Presumably, after sufficient psychometric magic had been performed, the new course-level final exam would “pass” no one who would not “pass” some associated AIMS exam. But this accountability provision was nothing but window dressing. A standardized test is a test that is administered under standardized conditions. Those taking it are allowed the same amount of time to finish and are treated equally in terms of accessories (i.e., one student may not be administered the exam without the aid of a smart phone while another is allowed to consult the internet for help in answering a question.). This obvious condition to impose on any test that deserves the name “accountability” could be met by sending all online students to a testing center to have their exam proctored. A private company could even set up testing centers all around the state and make itself a handsome profit (off of money that might otherwise have been sent to K12 Inc in Herndon, Virginia). Instead, modifications to SB 1259 before it reached the Governor’s desk stipulated that the final exam in the online course would have to be taken by the student in the presence of a “non-family member.” This feeble stab at instituting an accountability measure is simply laughable.

So why did Brewer veto SB 1259? The ostensible reason was that the Governor considered it inappropriate for the state "or an entity on behalf of the state [to approve] online courses or curriculum." This refers to a “master list” of courses that would have been created by the Arizona Department of Education for delivery to students through the online program. Courses currently offered by online providers (read “K12 Inc.”) would be “grandfathered” onto the master list.

What might be the real reasons that Brewer vetoed the bill?

The development costs for creating all those final exams and linking them to the state’s AIMS test could have proved too costly for either the state or the outside companies. It is unlikely that the accountability provisions in the final drafting of the bill originated with its sponsors; they probably were added by Democrats in committee. K12 Inc. might have wanted the Governor to kill the bill so that ALEC and their lobbyists could take another run at it in a future session.

Another reason for scuttling SB 1259 could involve the new State Superintendent of Public Instruction. A technophile politician by the name of John Huppenthal recently replaced a non-educator political opportunist who ran successfully for State Attorney General. Huppenthal, a former systems analyst for one of the state’s major utility companies, may have ambitions of his own for developing a state-owned cyber-schooling capability. K12 Inc. has attempted to kill off state-operated cyber-coursework in other states (Arkansas, Tennessee) to protect and expand its market. Brewer’s veto could signal a battle between states and corporations for control of the cyber-schooling market nationwide. If so, it will be a battle where students and taxpayers lose, no matter what the outcome.

Gene V Glass
University of Colorado Boulder
Arizona State University

Houston, You Have a Problem!

Education Policy Analysis Archives recently published an article by Audrey Amrein-Beardsley and Clarin Collins that effectively exposes the Houston Independent School District use of a value-added teacher evaluation system as a disaster. The Educational Value-Added Assessment System (EVAAS) is alleged by its creators, the software giant SAS, to be the “the most robust and reliable” system of teacher evaluation ever invented. Amrein-Beardsley and Collins demonstrate to the contrary that EVAAS is a psychometric bad joke and a nightmare to teachers.

EVAAS produces “value-added” measures for the same teachers that jump around willy-nilly from large and negative to large and positive from year-to-year when neither the general nature of the students nor the nature of the teaching differs across time. In defense of the EVAAS one could note that this is common to all such systems of attributing students’ test scores to teachers’ actions so that EVAAS might still lay claim to being “most robust and reliable”—since they are all unreliable and who knows what “robust” means?

Unlike many school districts which have the good sense to use these value-added systems for symbolic purposes only (“Look at us; we are getting tough about quality.”), Houston actually fired four teachers (three African-American, one Latina) based on their EVAAS scores. Houston fired Teacher A partly on the basis of EVAAS scores that looked like this:

EVAAS Scores for Teacher A by Year & Subject
Subject	2006-2007	2007-2008
Math	–2.0	+0.7
Science	+2.4	–3.5

The above scores are just a representative sample of the wildly unreliable scores that Teacher A accumulated over four years in several subjects.

As if this pattern did not alone exonerate Teacher A, her supervisor’s rating of her performance based on classroom observations were highly negatively correlated with the EVAAS scores. Amrein-Beardsley & Collins report that 1) teachers insisted that their teaching methods changed little from year-to-year while their EVAAS scores jumped around wildly, and that 2) principals reported having been pressured to adjust their supervisory ratings of teachers so that they were in agreement with the EVAAS scores. After all, did the administration want to admit that they had spent a half-million dollars of an elaborate mistake?

The whole Houston story as reported by Amrein-Beardsley & Collins is gruesome in the extreme, and I recommend that you read it in its entirety. For me, the story sparked recollections of disasters from thirty years ago that I chronicled in a chapter in a book edited by Jason Millman and Linda Darling-Hammond. (Glass, Gene V. (1990). Using student test scores to evaluate teachers. Pp. 229-240 in Jason Millman & Linda Darling-Hammond (Eds.), The new handbook of teacher evaluation: Assessing elementary and secondary school teachers. Newbury Park, CA: SAGE Publications.) You can read the entire chapter here.

In the mid-1980s, I was able to find six school districts in the entire country that claimed to have based teacher compensation on the test-score performance of their teachers. Each of the six showed the same pattern of behaviors that I summarized thus:

Using student achievement data to evaluate teachers...
1. ...is nearly always undertaken at the level of a school (either all or none of the teachers in a school are rewarded equally) rather than at the level of individual teachers since a) no authoritative tests exist in most areas of the secondary school curriculum, nor for most special roles played by elementary teachers; and b) teachers reject the notion that they should compete with their colleagues for raises, privileges and perquisites;
2. ...is always combined with other criteria (such as absenteeism or extra work) which prove to be the real discriminators between who is rewarded and who is not;
3. ...is too susceptible to intentional distortion and manipulation to engender any confidence in the data; moreover teachers and others believe that no type of test nor any manner of statistical analysis can equate the difficulty of the teacher's task in the wide variety of circumstances in which they work;
4. ...elevates tests themselves to the level of curriculum goals, obscuring the distinction between learning and performing on tests;
5. ...is often a symbolic administrative act undertaken to reassure the lay public that student learning is valued and assiduously sought after.

Most of what I saw in the mid-1980s is true today and is true of the present-day EVAAS system in Houston. Regrettably, point #5 is not so true. Not content to use these systems as mere symbolic window dressing, Houston has actually fired teachers based on their students’ test scores. Is HISD the bellwether of a dawning scientific age? Is it the district with the courage of its convictions? Should the nation look to Houston for leadership in insuring that teacher evaluation must be hard-headed and results-based?

Well, coincidentally, Houston was one of the six school districts that I investigated for my chapter in the Millman & Darling-Hammond handbook. And here is what became of Houston’s early-day effort to reward teachers for their students’ test-score gains.

Rod Paige, who eventually became Secretary of Education for Bush 43, became superintendent of HISD in 1994. But Houston had had a system of teacher incentive pay based on student test scores before Paige ever arrived. Since Paige was also an officer of the HISD from 1989 to 1994 and had co-authored the districts “Declaration of Beliefs and Visions,” his influence may have been responsible for the teacher incentive pay system that antedated his superintendency.

Teachers in Houston elementary schools were being given monetary bonuses—not base salary increases—when the entire school reproduced the previous year’s average test plus an additional increment, say, last year’s growth + 2 months grade-equivalents. The bonuses amounted to a few hundred dollars for each teacher. After a couple years in which the bonuses effectively replaced cost-of-living increases in the salary schedule, the test score gains were bumping up against the ceiling of the test. One or two schools missed their bonuses and tensions were rising.

In the meantime, the flow of money did not go unnoticed by the building administrators. Now principals are “instructional leaders,” or so the story goes. How they find time (or the expertise) to lead teaching while insuring the safety of students and staff, enforce discipline, direct traffic, and field complaints from angry parents is a mystery to me; but perhaps I don’t know what an instructional leader really is. So the principals banded together and approached the HISD administration and asked for their reward for making the test score gains. Only this time, the rewards were $10,000, $12,000, and sometimes $15,000—we’re talking 1985 dollars here too. Within a year or two, a couple of building principals were discovered having taken the test answer sheets into their offices and engaged in some erasing and marking. The entire system blew up and status quo ante was reinstituted.

Are things different now? Has some genius or some software company come up with a new system that is truly “robust and reliable”? And has a system been found that teachers and administrators acknowledge is legitimate and fair so that they will not be tempted to take whatever steps might be necessary not to become victims? And when will we see the value-added system that can be applied to politicians and school board members or even to researchers who invent value-added measuring systems? Or, as such persons regularly argue, is the value of their work so much more complex than that of a teacher of young children that it could not possibly be captured by a clumsy quantitative index?

Gene V Glass
University of Colorado Boulder
Arizona State University

New Orleans and Old Libertarians

Washington Post Op-Ed page staff writer Jo-Ann Armao enthused at length on Friday about the miraculous things happening in the erst-while very public schools of New Orleans in this post-Katrina era. In between gulps of Kool-Aid, Armao wrote about wonderful "turn-arounds" of schools once imprisoned in the grasp of evil teachers unions and inept traditional administrators.

Most of the buzz about the city’s reforms focuses on the banishment of organized labor and the proliferation of charter schools, which enroll nearly 80 percent of public school students, up from 1.5 percent pre-Katrina. But what really distinguishes New Orleans is how government has re­defined its role in education: stepping back from directly running schools and empowering educators to make the decisions about hours, curriculum and school culture that best drive student learning. Now, state and school-district officials mostly regulate and monitor — setting standards, ensuring equity and closing failing schools. Instead of a traditional school system, there is a system of schools in what officials liken to a fenced-in free market. Families have more choice about where their children can best succeed, they say, and educators have more opportunity to choose a school that best aligns with their approach.

OK. The stage is set. The free market in public education has come to post-Katrina New Orleans. And what has resulted? Armao goes on:

Since 2006, New Orleans students have halved the achievement gap with their state counterparts. They are on track to, in the next five years, make this the first urban city in the country to exceed its state’s average test scores. The share of students proficient on state tests rose from 35 percent in 2005 to 56 percent in 2011; 40 percent of students attended schools identified by the state as “academically unacceptable” in 2011, down from 78 percent in 2005.

Nothing short of a miracle, if you believe it; but only a fool would.

Milton Friedman, in one of his last public utterances before he died, called Katrina an "opportunity." "Most New Orleans schools are in ruins as are the homes of the children who have attended them. The children are now scattered all over the country. This is a tragedy. It is also an opportunity.” What he meant, of course, was that the tragedy presented the opportunity to privatize and "charterize" the New Orleans public school system. Friedman was being perfectly consistent with what he had written in 2002 about the use of "crises" to open the door to favored reforms. "Only a crisis—actual or perceived—produces real change. When that crisis occurs, the actions that are taken depend on the ideas that are lying around. That … is our basic function: to develop alternatives to existing policies, to keep them alive … until the politically impossible becomes politically inevitable.”

There is much that is shocking in Friedman's insensitivity and cynicism, not the least of which is that his comments imply that mere "perceived crises" (not necessarily "real"?) can present the opportunity to implement a radically different political philosophy. We have had enough of "manufactured crises" in the business of education reform.

But back to Armao. Yes, things might be different in the New Orleans public schools, but attributing those differences to free-market reforms is ludicrously naive. What happened from pre- to post-Katrina was a dramatic shift in the city's demographics. Between the 2000 Census and the 2010 Census, the population of New Orleans changed like so:

New Orleans Population Change, 2000—2010, by Race
Race	2000	2010	% Change
Non-Black	156,000	137,000	–12%
Black	329,000	207,000	–37%

More than one in every three Black persons in New Orleans was displaced from the city by Katrina, because they lacked the resources to return after the hurricane or because they lived in the wards most severely affected by the flood. About one in ten non-Black persons (overwhelmingly non-Hispanic White) was similarly displaced.

Any attempt to draw a conclusion about charter schooling or free-market reforms from pre-Katrina vs post-Katrina test score comparisons would be a good exercise for a Freshman course in logic. That the Op-Ed page of the Washington Post fails at such an exercise should be both surprising to readers and embarrassing to the editorial staff.

Gene V Glass
University of Colorado Boulder
Arizona State University

When Will the Cyberschooling Giants Start Acquiring EMOs?

For the past few weeks, I have been corresponding with some film producers who—encouraged perhaps by the commercial success of "Waiting for Superman"—have an inkling that some very important things are happening with public education in America...some things like crony capitalism, and an economy shifting to public risk and private profit. Recently they asked me whether the virtual schools trend was important. I offered the reply copied below.

Is the advent of online schools for elementary and secondary school students one of the biggest stories in public education today?

The emergence and growth of cyberschools at the k-12 level is a great story because it tugs at the heart strings: little kids—6, 7, 8 years old—staring at the PC screen for hours, alone or with a sibling, their intellectual development entrusted to a parent whose moral outrage is not matched by their intellectual curiosity or mastery of even the simplest school subjects? Yes, that is truly awful. My daughter once had neighbors who were homeschooling their children, and she could see that the kids spent most of the day in the backyard on a trampoline. And homeschoolers are among the biggest source of customers for the cyberschool companies—homeschoolers and potential dropouts.

But cyberstudents still constitute a very small slice of the total k-12 population. One could argue that they are currently getting more attention than they deserve because their situation is so striking. If their numbers reach 300,000 students, that’s just 0.6% of the public school population (300,000/50,000,000). Nonetheless, the brazenness of the commercial companies—primarily K12 Inc.—does make cyberschools a compelling story.

Incidentally, you may want to look into whether the military recognizes an online school diploma. Last I heard, it did not.

Although k-12 cyberschools are still a tiny slice of the public school population, charter schools are not. In some states—Arizona for example with 540 charter schools enrolling 135,000 students; Colorado with 185 schools and 80,000 students—charter schools are enrolling about 10% of the public school population. And although that movement started out with idealistic visions of brilliant young teachers and highly motivated students, it is devolving into a shoddy business with increasing numbers of charter schools operated by profit-making Education Management Organizations (EMOs).

Now the cyberschool companies use the charter school laws to make their money. It’s obvious if you think about it. So can alliances and mergers between the biggies—K12 Inc and Pearson (Connections)—and the EMOs be far in the future? For a report on the state of the EMOs, see http://nepc.colorado.edu/publication/EMO-profiles-10-11 .

Currently in the U.S., there are approximately 5,500 charter schools enrolling more than 1.7 Million students. That works out to about 1,700,000/50,000,000 = 3.4% of all the k-12 school children in the nation. Now if we strike a “price” of about $10,000 per student—the current average expenditure in the U.S. for a k-12 student—we can see that the charter school business is a $17 Billion market (“B”, not “M”). In 2010-2011, EMOs managed about 1,900 charter schools enrolling approximately 750,000 students in Kindergarten through grade 12. These three-quarter million EMO-managed charter school students represent a $7.5 Billion market for the cyberschool industry. K12 Inc., the big shot in the cyberschool business, has annual revenues just more than a half billion dollars.

What should we expect then? Obviously, the big players—K12 Inc. and Connections—must already be looking to acquire some of the small EMOs. Stock-holders expect such things, particularly when a major player like K12 Inc. disappointed them mightily in November 2011 when its stock price dived 50% from $37/sh to $18/sh, not coincidentally after the October release of a critical policy brief by the National Education Policy Center focused on cyberschools generally and an article in the New York Times focused specifically on K12 Inc.

Gene V Glass
University of Colorado Boulder
Arizona State University

Conservatives, ALEC, and Vouchers Lose—Temporarily—in Arizona

Governor Brewer (R-AZ) caused a bill to be killed that would have greatly expanded the Arizona voucher program. Enacted a few years ago, the program gave vouchers, pegged at 90% of the average per pupil expenditure, to handicapped children’s parents that could be redeemed at any private school. The Arizona Supreme Court routinely approves of such legislation. In Fall 2011, only 75 vouchers were issued statewide. Perhaps most private schools would rather not deal with handicapped students, and those that do deal with them well cost considerably more than $8,000 per year. So legislators came back this session with a bill to expand the program to included “gifted” children. Since all God’s children are gifted in one way or another, the bill killed by Brewer would have enormously expanded the program and created a market niche for private schools: the “gifted” children of middle-class families.

Giftedness has such an absurd history in American education that basing any legislation on it is a joke. The pressure exerted by parents on administrators to label their child “gifted” results in multiple definitions–soon “gifted” is expanded to “gifted and talented”–so that ultimately the wave of giftedness has to be controlled by placing a statistical limit on the use of the label. (Many years ago while studying the distribution of “emotionally disturbed” students across states, I noticed that Utah had the highest rate of EMD pupils in the nation. That did not comport with my own stereotype of the Beehive State, so I investigated. I found out that the vigorous efforts of professionals in the state had prompted the Utah Legislature to place a cap of 3.5% on the percentage of students who could be labeled EMD, and lo and behold, 3.5% of the students in the Utah public schools were diagnosed as emotionally disturbed.)

Bills like the attempted expansion of the Arizona voucher program are probably lifted directly from the model legislation disseminated by ALEC, the American Legislative Exchange Council. ALEC is the source of most of the business-friendly legislation making its way around the country these days. Wikipedia says quite accurately that “The American Legislative Exchange Council (ALEC) is a politically conservative 501(c)(3) nonprofit policy organization, consisting of both state legislators and members of the private sector, mostly representing corporations. ALEC's mission statement describes the organization's purpose as the advancement of free-market principles, limited government,federalism, and individual liberty.” ALEC’s budget amounts to roughly $7 Million a year, but only 1% of that total comes from state legislator members; the rest comes from large corporations. Among ALEC’s recent pushes has been the attempt to get legislatures to pass voter identification laws in preparation for the 2012 general election. For example, potential voters in Texas will be able to cast their ballot in November if they present identification in the form of a gun permit but not a college student ID card. Coca-Cola and Pepsi withdrew their financial support of ALEC this week.

Governor Brewer has promised her conservative backers that she will revisit the expansion of the voucher program question this summer after the state budget is set.

Gene V Glass
University of Colorado Boulder
Arizona State University

High Button Shoes and Education Reform

As a very young child, I was fascinated by my grandmother’s collection of button hooks. It was the mid-1940s and high button shoes had been out of style for decades. She collected the hooks that were used to pull the tiny buttons through the holes that ran up the sides of the ladies’ shoes back in the early 20th century.

Often made of silver with decorative handles of porcelain or glass, the hooks made for an attractive display.

Warren Buffett and Charlie Munger, his long-time partner at Berkshire-Hathaway, also had a connection to shoe buttons. Munger’s grandfather had managed to corner the market on shoe buttons back around 1900. The grandfather exercised a virtual monopoly over their production and sale. Emboldened by his business acumen, the old man grew to believe that he not only knew more than anyone about shoe buttons but that he knew more than anyone about anything—and he preached and proclaimed at length on such. Munger and Buffett named the syndrome the Shoe Button Complex, and they encountered it frequently in their dealings with successful business practitioners.

Now Buffett struggled assiduously to avoid developing the Shoe Button Complex. As one of the richest persons in the world, the temptation to succumb would surely have been great. He was careful to restrict his actions and speaking to what he called his Circle of Competence. He recognized that there were a limited number of things he could know well, and he did not presume to act as though he was expert of those things lying outside the Circle.

Buffett’s recognition of his Circle of Competence led to some surprising decisions as he approached and passed his 80th birthday. He did not understand the emerging digital technology, and while the masses were losing their shirts trading high tech companies, Buffett was buying stocks in railroads and underwear manufacturers. Poseurs with enormous Circles of Competence scorned the old man when he reminded them that in the past 120 years in the U.S. more than 1,000 automobile manufacturers came and went, leaving just four. Buffett’s rise to the top of the list of the world’s billionaires left him with a problem as he saw the end of his career approaching. How does one with a small Circle of Competence give away tens of billions of dollars to some good end?

Bill Gates and the Shoe Button Complex

Buffett startled the world when he announced a few years ago that he was leaving his great wealth to the Bill & Melinda Gates Foundation. Buffett spoke thus in addressing the question of why he turned over his philanthropy to the Gateses:

Bill Gates is the most rational guy around in terms of his foundation. He and Melinda are saving more lives in terms of dollars spent than anyone else. They’ve worked enormously hard on it. He thinks extremely well. He reads thousands of pages a year on philanthropy and health care. You couldn’t have two better people running things. They have done incredible work, they’ve thought it through, their values are right, their logic is right. (p. 768 in The Snowball: Warren Buffett and the Business of Life)

Well, Buffett’s decision presents us with a conundrum. If he is not competent to determine the worthwhile recipients of his beneficence, then how is it that he knows that the Gates Foundation is dispensing beneficences in a worthwhile way?

Education is an arena particularly prone to attracting Shoe Button Complexes. Everyone has been to school; everybody thinks they know what is wrong with schools.

And so we come to the question, How is the Gates Foundation doing these days? This question is of more than passing interest on account of the fact that the Foundation is dispensing roughly $400 Million a year to education related causes. Moreover, some utterances by the benefactors have raised eyebrows among groups that have long made the search for understanding education their preoccupation. For example, Melinda Gates startled her interviewer on an NPR program in 2007 when she seemed to suggest that 100% of high school students should continue their education into college.

(Interviewer) I just want to ask you about a statement on the Bill and Melinda Gates Foundation's Web site that I read, which is that, "All students in the United States can and must graduate from high school, and they must leave with the skills necessary for college, work, and citizenship." I think everyone would agree that they better leave with the skills for citizenship because everyone can vote at age 18, and we urge them to. College. Can we reasonably expect 100 percent of high school students to become college students?

(Melinda Gates) Yes, I think we can. And, in fact, I'm here today in the Chicago school district visiting with students – huge number of Latinos and African-American populations, and guess what? I'm in schools where 95 to 98 percent of these kids are going on to college, and it's because they started freshman year with teachers who believe in them and said, 'These kids can do it.' …

(Interviewer) That would be a dramatic increase of the share of high school students, if 100 percent went on to college. I mean, you would be effecting an enormous social change if you could reach – (Melinda Gates) Correct, and that is the idea.

(Interviewer) How many years do you think it would take to achieve that particular – (Melinda Gates) I think it is going to take us quite a while. I think that this is a long-term effort and I think it's one that the foundation is going to be at for a very long time. ... (See the entire transcript here.)

We may be looking at a Circle of Competence problem here. And the situation may not be much better with her husband, who teeters dangerously close to the edge of a Shoe Button Complex. Bill Gates has referred to Diane Ravitch as “public enemy #1” of effective education. Whether either Diane Ravitch or the nation’s schools fall within his Circle of Competence is questionable. A few years ago when Gates testified to Congress on the current state and future of American education, he spent most of his time complaining about difficulties in obtaining visas and green cards for young tech employees of Microsoft. That his interest might be stronger in promoting the health of this business than in promoting the development of the nation’s children may be understandable. After all, what does the richest man in America really know about the needs of the nation’s children—the vast majority of whom will hold a half dozen low-level jobs during their lifetimes in industries like recreation, food services, child care, health care, and the like?

Now my own Circle of Competence does not extend much past some understanding of what is happening to K-12 public education in America. Thanks to Ken Libby of the National Education Policy Center and his analysis of the Gates Foundation grants to U.S. education, we have at hand new information about what Bill & Melinda Gates consider to be efforts to improve schooling that are worthy of their own and Warren Buffett’s support. The following table shows Libby’s breakdown of where Gates Foundation money for education went in the three years from 2008 through 2010.

Category	Total $ 2008-2010	%
Charters	$73.1M	7
Alternative Public Schools	$97.8M	9
Private Schools	$47.4M	4
Small Schools	$30.1M	3
School Reforms	$61.4M	6
Government Agencies	$9.0M	1
Advocacy	$116.8M	11
Think Tanks	$10.0M	1
Research	$78.6M	7
Development	$112.6M	10
College/Career Ready	$52.4M	5
College Completion	$145.4M	13
Common Core	$18.5M	2
Early Learning	$46.8M	4
Conferences	$10.0M	1
STEM	$26.0M	2
Human Capital	$104.0M	9
Media	$17.0M	2
RTTT & i3	$4.7M	0
Other	$40.2M	4
Total	$1,101.8M	100%

I leave it to the readers to make their own interpretations of the mind-set that lies behind these kinds of allocations. As for me, that mind-set shows little faith in the development of better education for the vast majority of America’s children, particularly children in poverty. It is a mind-set that comes about from drinking the Kool-Aid that the “market” will lead America’s schools to the promise land.

Consider the following example of how money from the Gates Foundation is being spent. A proposed law in Florida named Parent Empowerment was pushed this legislative season by a California-based group called Parent Revolution. Parent Revolution is funded by Gates, the Eli Broad Foundation, and the Walton Foundation. If Parent Empowerment became law in Florida, then 51 percent of the parents in a public school could sign a petition that would give them control of a school and give them the power to decide whether to close it or turn it over to a charter management organization. It is not difficult to see who the true beneficiaries of this bill would be. In fact, the parent Empowerment bill could have been written by lobbyists for the education management industry. (See Diane Ravitch's report on the true parents uprising that defeated this bill that was nonetheless backed by former governor Jeb Bush and current governor Rick Scott.) (Also, see the NEPC report on the EMOs for a revealing look at the breadth of this industry.)

The Gates Foundation support of dubious enterprises didn't start with the Parent Empowerment bill. As Diane Ravitch recently remarked, "The Bill & Melinda Gates Foundation puts up the money to ensure that ["Waiting for Superman"] this morality tale of good reformers and bad teachers is shown to state legislatures, to civic groups, to people living in housing projects. The movie itself is financed in part by an evangelical billionaire (Philip Anschutz) who contributes heavily to libertarian and ultra-conservative causes."

The Shoe Button Complex in Arizona

Jan Brewer, Republican governor of Arizona and famous for issuing a tongue wagging to President Obama, appointed Intel ex-CEO Craig Barrett to chair a council—Ready Arizona--to study and recommend public education reform for the state. It is unclear what Barrett knows about education. One suspects that we are encountering another case of the Shoe Button Complex. Barrett is urging businesses to push school reform. His public utterances strike familiar chords: the future of the entire state rests on the test scores of little kids; more science and math majors will attract businesses to the state; it’s a global economy. After all, the public schools are “suppliers” of labor for businesses. And at Intel, “if a supplier didn’t meet our specifications, we would call the supplier and say, ‘Meet our specifications or we will fire you.’” Apparently, Barrett shares his fellow Republican Mitt Romney’s pleasure in firing people.

Of course, what Barrett is actually and unknowingly talking about is crony capitalism: Linking government and business in relationships that favor the economy. Whether the intellectual, moral, physical, and aesthetic well-being of young people is benefited by their education probably never occasions to Barrett and his ilk. Or perhaps "well-being" to Barrett means having acquired a taste for consumerism and a job to support it. In fact, most industry leaders would like to see specialized training pushed down as early in the curriculum as possible so that high school graduates appear in their HR departments job-ready, trained at public expense. And if training kids for Intel just happens to involve piping a bunch of online courses into Arizona public schools, well so much the better since Barrett also serves on the board of K-12 Inc., the nation’s #1 supplier of cyber-courses. Whether the former CEO of Intel knows everything there is to know about selling microprocessors AND education, or whether this is merely another manifestation of the Shoe Button Complex remains to be seen.

Gene V Glass
University of Colorado Boulder
Arizona State University

How the Rich Get Richer—Arizona’s Tuition Tax Credits

The Arizona Tuition Tax Credit has been the subject at this blog on a couple of occasions. In its early form it drew the incredulous attention of scholars who questioned both its legality and its fairness (See Welner & Moses in References).

In 1998, the law allowed tax payers to direct up to $200 of their state income tax indebtedness to a public school to be used for extra-curricular activities. (The $200 extra-curricular thing was really just a sop to opponents of the main section of the bill, which now allows a $2,000 tax credit contribution to a private and/or religious school for tuition.) Glen Wilson calculated that in the first year of the program approximately $6 Million were contributed to a total of about 900 public schools. The per student donation was $8.80--nothing much to get excited about. But the program has grown amazingly large. Modifications of the law have raised the contribution limit to $400 for a couple filing their tax return jointly and permitted the money to be spent on character education programs.

Tens of millions of dollars of state tax indebtedness are now directed to public schools in Arizona. As one might expect, the money is hardly distributed equitably. Let’s take a look at a half-dozen schools in a single school district (Scottsdale Unified School District) that differ greatly in the socio-economic level of their attendance areas, and, as a result, differ greatly in how they are benefiting from the tax credit program. In the table below, four K-5 elementary schools and two 9-12 secondary schools are shown along with their enrollments, total contributions, per pupil contribution and typical housing prices of their attendance areas.

School	Total $	# Pupils	$/Pupils	Neighborhood House Prices
Tonalea (K-5)	$15,150	550	$28	$25,000 to $200,000
Tavan (K-5)	$23,800	770	$31	$200,000 to $300,000
Hopi (K-5)	$70,300	770	$91	$1,000,000 to $2,000,000
Kiva (K-5)	$87,350	770	$113	$500,000 to $5,000,000
Coronado (9-12)	$121,500	1,300	$93	$100,000 to $200,000
Chaparral (9-12)	$404,950	1,900	$213	$500,000 to $5,000,000

Tonalea and Tovan draw students from lower middle-class attendance areas and collect roughly $30 per student, while Hopi and Kiva are collecting 3 and 4 times that much. The former state superintendent of schools sent her children to Kiva, where one took a trip to Catalina Island on tax credit funds in the earliest days of the program. At the secondary school level the inequity is huge—more than a $100 per student difference in contributions. Coronado is a low-income high school receiving less than $100 per pupil under the tax credit program while Chaparral enjoys contributions topping $200 per student.

And so once again our legislators and their constituents have brought the Matthew Principle to the service of their children and their neighborhoods. “For unto every one that hath shall be given, and he shall have abundance: but from him that hath not shall be taken away even that which he hath .” (Matthew 25:29)

References

Moses, Michele S. (2000) Arizona Education Tax Credit and Hidden Considerations of Justice. Education Policy Analysis Archives, 8(37). Retrieved March 19, 2012 from http://epaa.asu.edu/ojs/article/view/428/551.

Welner, Kevin G. Taxing the Establishment Clause: The Revolutionary Decision of the Arizona Supreme Court in Kotterman v. Killian. Education Policy Analysis Archives, 8(36). Retrieved March 19, 2012 from http://epaa.asu.edu/ojs/article/view/427/550.

Wilson, Glen Y. (2000) Effects on Funding Equity of the Arizona Tax Credit Law. Education Policy Analysis Archives, 8(38). Retrieved March 19, 2012 from http://epaa.asu.edu/ojs/article/view/429/552.

Gene V Glass
University of Colorado Boulder
Arizona State University

Eric Hanushek Testifies in School Finance Cases

Eric Hanushek testifies in school finance cases. Again, and again, and again. Thirty-some years ago in the Maryland (Hornbeck) case and most recently in the Colorado (Lobato) case. And each time, Hanushek, an economist at the Hoover Institution, testifies to the same position: increased funding for K-12 schools will not improve their effectiveness; court-ordered remedies that cost money will not improve the lot of poor students or English Language Learners or anyone else for that matter.

Hanushek is nothing if not a believer in the unconditional truth emanating from his regression equations. But of course, those equations have not always been as clear cut in their implications as some might believe. In 1997, Hanushek published an article in which he argued that a summary of dozens and dozens of correlation studies proved that teacher experience is unrelated to their students’ achievement—the financial implications being obvious. He presented the following summary of studies that investigated the relationship (in terms of regression coefficients) between student achievement and their teachers’ “years of experience.”

Although a statistically significant regression coefficient for “teacher experience” was six times more likely to be positive than negative, Hanushek nonetheless read the results as negative for the effects of teacher experience on achievement. “Importantly, … 71% [of the regression coefficients] still indicate worsening performance with experience or less confidence in any positive effect,” he wrote.

The logic of this conclusion is illusive—no; it makes no sense. Of results that reach statistical significance, 85% (60/70) are positive; students of more experienced teachers achieve at higher levels. Of the statistically non-significant results that can be determined, 55% are positive, but fail to reach conventional levels of significance. Hanushek creates an impression of no effect of teacher experience by lumping together the category (1) “indicative of worsening performance or less confidence of beneficial performance” all significant but negative coefficients (5%), (2) all nonsignificant coefficients whether positive or negative (30% + 24%) and, (3) remarkably, the 12% of the coefficients that were so incompletely reported that it could not be determined whether they were positive or negative. The treatment of these data is hardly even handed. By such logic, ten “positive studies,” “no negative studies” and 100 studies so poorly reported that the results could not be discerned would lead Hanushek to a conclusion of no confidence in a positive result. My reading of these results is much different from Hanushek's. Regression studies have generally shown a positive relationship between teacher experience and student achievement. Period.

So if Hanushek’s performance in reviewing the research on teacher experience and student achievement is as shaky as this, how does he perform in court on matters closely related? Aaron Pallas, in his blog "A Sociological Eye on Education" recently wrote on that question.

I was reminded of a school finance court case in Maryland some 30 years ago [Hornbeck; I testified in that case on the role of class size and student achievement ~GVG] for which I served as a consultant. … The poorest districts in the state, including Baltimore City, were suing the state to force it to equalize school funding. The state … hired noted economist Eric Hanushek to testify that money doesn’t make a difference in student outcomes. I was hired to prepare questions for cross-examination that might discredit Hanushek’s testimony.

A friend … suggested a novel line: If, as Hanushek argued, spending more money wouldn’t increase achievement, wouldn’t spending less money have no effect on achievement either? “Brilliant!” I thought.

The time came for the cross-examination, and, among many other questions, the plaintiffs’ attorney asked this question.

“That—that almost follows,” Hanushek replied.
…
It wasn’t the Perry Mason moment I was hoping for, but it was enlightening nevertheless. The reality is that Hanushek’s claim that money doesn’t matter was based on natural variations among districts in their spending patterns, and wealthy districts spent more and had better educational outcomes than poorer districts that spent less. There were no studies showing what would happen if spending were to increase or decrease precipitously over time ….

Now thirty years later, Hanushek is still telling courts that money doesn’t matter. (Linda Darling-Hammond probably made the most incisive comment on the “money doesn’t matter” position when she said, “If money doesn’t matter, why are the rich trying so hard to hold onto it?”) He testified in the recent Colorado school finance case (Lobato vs State of Colorado) once again that money doesn’t matter. Only this time, a Colorado judge had little sympathy for counter-intuitive social science stuff. In her 189-page ruling deciding in favor of the plaintiffs, Judge Sheila Rappaport commented thusly on Hanushek’s testimony:

Dr. Hanushek’s analysis that there is not much relationship in Colorado between spending and achievement contradicts testimony and documentary evidence from dozens of well-respected educators in the State, defies logic, and is statistically flawed.

[Ouch!]

For decades, judges at many levels in the judicial system have been cowered by complicated looking statistical testimony. One hopes that those days may be coming to an end.

References

Hanushek, Eric. (1997) Assessing the effects of school resources on student performance: An update. Educational Evaluation and Policy Analysis 19 (2), 141-164.

Glass, Gene V. (2002). Teacher characteristics. Chapter 8 (Pp. 155-174) in Molnar, Alex. (Ed.) School Reform Proposals: The Research Evidence. Greenwich, CT: Information Age Publishing, Inc.

Gene V Glass
University of Colorado Boulder
Arizona State University

The Cost of Online (Cybercharter) Schooling ... or Shooting Your Friend in the Foot

Today, Jennifer King Rice's critique of a Thomas B. Fordham Foundation report on the cost of full-time online K-12 schooling was published by the National Education Policy Center. Dr. Rice pointed out the many complications of attempting to put a dollar figure on ordinary brick-and-mortar schooling, let alone the price of a year of cyberschool. But the gist of both the Fordham Foundation article and Rice's critique as well as several other shots that have been taken at pricing cyberschooling is this: the latter costs about two-thirds to three-quarters of the former; roughly $10,000 per pupil per year vs $7,000. And it must be recalled that even this cost differential is clouded in the haze of dicey reporting by the for-profit cyberschooling companies (read "K12 Inc." and "Connections"). For example, the huge run-up in the past three years of K12 Inc. stock (LRN on the NYSE) includes profits on their half-billion dollars a year revenues, which profits must be sizeable to justify the price of the stock. In Arkansas, a state board of education member raised questions about the administrative costs claimed by the Arkansas Virtual Academy (a K12 Inc. company). Arkansas Virtual Academy was charging 15% administrative costs compared to 5% for brick-and-mortar schools in the state. (Perhaps K12 Inc. writes off lobbying expenses to "administrative costs." After all, in Mississippi where a pitched battle is being waged by K12 Inc. to have cyberschools included in the state's charter school system, former Governor Haley Barbour's nephew Henry has been hired as a lobbyist.)

The interesting sidebar to all this research and politicking is that one traveler in the free-market schooling campaign is shooting another fellow traveler in the foot. At the same time that the Thomas B. Fordham Foundation is putting out research claiming that cyberschools cost 30% less than brick-and-mortar schools, companies like K12 Inc. are trying to convince legislators around the country that the difference is zero, i.e., that the charter school should be reimbursed at 100% the cost of educating a pupil in a brick-and-mortar school.

In 2010, the Georgia State Commission on Charter Schools established a funding level of $3,500 per full-time K-12 pupil in online charters, plus a 3% administrative fee. Testifying in favor of 100% funding, one Allison Powell, vice president of the International Association for K-12 Online Learning (a lobbying front for the big cyberschooling companies) claimed, “The costs are pretty equal to a lot of brick and mortar schools. You don’t have the transportation or building costs, but you still have to provide Internet and computers. You have personnel. A lot of states will fund them at the same level.”

In truth, there are hardly any states that fund cybercharters at the level of funding for brick-and-mortar schools. But that is not to say that the companies are not trying. A bill currently making its way through the Arizona legislature would increase cybercharter per pupil funding to 100% state allocation provided a few easy hurdles are cleared: 50% for all students enrolled on October 1, 35% more for all who finish the year, and the final 15% for all those who pass an achievement test. The bill looks like it was written by lobbyists, who reluctantly, perhaps, acceded to pressure to add some symbolic "accountability" features.

Gene V Glass
University of Colorado Boulder
Arizona State University