For all the fuss about multiplication tables , algebra and calculus what’s more alarming is the depth of our layman understanding of statistics–including students with high test scores and elite educational backgrounds. Including, of course, journalists and politicians and newspaper editors.
Once again Richard Rothstein of EPI comes to the rescue. Along with Martin Canoy of Stamford they undertook a close examination of the numbers around international test score comparisons. Unsurprisingly they found that the differences were (1) usually a reflection of the percentage of low-income children in the sampled population, 2) the choice of what kind of information to cover (number theory vs algebra–where we do better on the latter and the Finns on the former, but we just happened to be on the unlucky draw of the dice).
It makes for good reading. For more go to Economic Policy Institute (EPI)
I think I turned off of math in elementary and high school (although I did fairy well because I was dutiful). I thought it was a dull subject because it was just a question of right and wrong answers–which didn’t seem to me to offer room for intellectual curiosity. Except for statistics, which I took in college and again when I needed a certificate in eduction. (It’s too bad, because later on I discovered that all math can be fascinating.)
It was statistics that first clicked for me–because by that time I had become well aware of how easy it was to use data to “prove” one’s case–regardless of which case you had in mind. (My father was an expert at it!) By the time I began teaching, and above all by the time I was putting together reports to the DOE and friendly Foundations, not to mention serving on my local school board, I discovered a world in which data was the most freely floating of phenomena. I became a fa of true hard data–direct narrative. At a certain point I ceased to believe any data on education that rested on any kind of “high stakes” data collecting–that is data that teachers and/or principals and/or school boards/ and or politicians needed for one self-interested purpose or another. (It took me a while to suspect that I needed to do the same about economic data, financial data, and on and on.)
Nothing could be taken for granted. Not even the definition of being “absent” from school. In California, at one time, only students who were playing truant and had no legitimate excuse note from home were marked “absent.” In NYC it depended on when attendance was taken. Too raise attendace rates in NYC one enterprising chancellor just changed the time of day for taking attendance–from first to 4th period.
Drop out rates? They remain a complete fraud. Partly because we define them differently and partly because it’s so easy for a principal to cheat and partly because it’s not easy for them to actually know whether a student simply transferred or dropped out.
And on and on. I laugh when anyone gives me data from countries like China – where they are no doubt experts at using data for propaganda purposes. Are those poor children in Shanghai–who were considered illegal immigrants (from the countryside) counted? Probably not because they couldn’t attend schools in Shanghai. And few Chinese citizens are likely to take up the FairTests role in China: questioning the State’s official truth.
Yes yes yes. I trust the judgment of almost every teacher more than the judgment that rests only on so-called “hard” data. The hardest data of all is the work of the kids themselves, their voices and language, their defense of their work, their projects and products. If we cared enough–that’s what would be judged.
Imagine the increase in road accidents if we eliminated the actual driving test and only used the multiple-choice section!!! The scorers who drive with the testee may have biases, but I’ll accept their judgment before the paper-and-pencil test score, with all of its “objectivity”. Of course, we could improve those driving tests…..
.
Filed under: 2013 posts |
Deborah Meier on Education 
This is a brilliant commentary. Thanks!
It strikes at an example of the horizon effect in the thinking of most and of everyone when time to explore is tight. Namely that the superficial acceptance of information is too easy as most have not developed their critical faculties. By way of example, it sounds superficially obvious that better teachers should receive more pay than weaker teachers. But the implementation costs of performance pay, and the corrosive effect on teacher collaboration would outweigh what are debatable gains in the first place. What sounds intuitively correct – that superficial thinking – is often wrong.
Stepping back further still, human judgement is assumed to be rational. But it is often not (read Sutherland’s book “Irrationality” if you are interested in this subject).
And who are the kinds of people who respond superficially to matters – who do not explore nuances and consequences? Those in power who act fast to make education decisions.
As the article says, data is superficially accepted, yet is flawed in many ways. Stats is one of the most exploited aspects of mathematics. And as an aside, economic theories that combine stats and a presumption of rational human behaviour are some of the most dangerous of all.
Deb, about a year out of college, I joined a “reliability” briefing on the system we were building–a $2 billion avionics system, then the most complex electronics system in development.
So the reliability guys show chart after mind-numbing chart adding up the reliability estimates of all the components. The ring-laser gyros have a .99844 reliability. The navigation computer processor a .99723. The air-data sensor package a .99379. Add them all up (according to the statistics rules) and you can expect that the aircraft will be flying 99.xxx% of the time its supposed to.
So I ask the briefer, “Did you include the system-wide redesign we pushed last quarter? “No.”
Perhaps you have heard of Ohio’s attendance scandal this fall. ODE seems to have been encouraging schools to backdate the dis-enrollment of students and erase the recorded absences along the way. Maybe the process even made sense, who knows.
The lesson from these two stories and yours above is not “don’t look at the data”. That’s totally the wrong lesson. The data–even when fuzzy–can help you to look at things you overlooked. Real things.
For instance, no matter how big the discrepancies in drop-out data, the numbers tell us that in two car-and-steel towns, Pittsburgh has something right going on, and Detroit is dysfunctional beyond belief.
I did some forensic accounting for an insurance company. In the end, the numbers told us that everything was as it seemed before the “whistle-blower” pointed the finger. However, along the way we learned many things about relations with customers, how their needs were being met, and where we had mis-tepped with individuals.
And those aircraft I mentioned still fly the skies.
The numbers tell us many other things. Like that teachers have half a trillion dollars earning profits with big corporations! Evil 10+% profits are funding teacher retirements!
And that government in the US spends more of the nation’s income than at any time save the global war years of 42-44.
And that it has always and still does take a two-parent family to escape poverty, governments expensive efforts over 40 years having changed that eternal truth not a whit.
Poets like to shrug off numbers. Poets make bad laws. Bad laws make bad situations worse.
Ed,
It’s when high stakes get attached that we begin to game the data. Otherwise data is simply that–interesting. If it seems counter-intuitive it’s worth exploring before believing, but maybe it’s counter-intuitivity will lead to a real breakthrough in ones understanding. If I’ve been in two schools and the data on them runs counter to everything I witnessed, it’s worth wondering. Often it’s “Campbell’s Law” that’s at fault.
Then too, one has to compare the damage done by bad data vs the positive outcomes that good data can do.
etc etc.
Thanks!
Speaking of data, have you watched/read Charles Murray’s “”Coming Apart: The State of White America, 1960-2010″”?
We’ve not discussed it, but it seems to cast a huge light on many issues that we associate with poverty/racism/schools/government.
http://www.booktv.org/Program/13161/quotComing+Apart+The+State+of+White+America+19602010quot.aspx
I find Murray’s history and the title disturbing. At the moment I/m putting reading it off. But, eventually I guess I must. What is your take? deb
Watched, but haven’t read it. (My technical/etc reading leaves me time for one good fiction book and perusal of something like Pakistan: A Hard Country.)
The BookTV presentation made the case for why he limits it. His research instantly takes racism out of the question. And what he finds is that the inequalities many people blame on racism or historical treatment of Blacks are pretty much the experiences of poor whites.
Experiences both good and bad.
Middle-class people are more likely to be part of a faith community than the poor. That didn’t used to be the case.
Middle-class people are more likely to be married than the poor. Again, that didn’t used to be the case.
And a number of other disturbing trends that make achievement of the American dream to look much less likely for the poor than in the pre-TV days.
I need to look at it again, myself.
Than why the title?