Friday, July 29, 2016

Misunderstanding standardized tests and quality

Someone recently shared an article by a very thoughtful writer who acknowledged that while standardized testing has shortcomings it is nevertheless necessary for understanding something about the quality of our schools. This once again shows that even very smart people have bought into the idea that standardized test scores are an indicator of quality. That is a pervasive fallacy we've been fighting since the invention of a test capable of rank ordering people. The damn psychologists all those years ago were interested in rank ordering because they saw it as a way to confirm their social and racial biases as being based in the notion that they were better than their non-white, non-male peers. They failed to see that the consistency in their results had to do with the fact that society didn't change much while they were doing their work as opposed to repeatedly confirming their stupid, racist, sexist biases.

Standardized testing continues to play into the larger narratives of competitiveness and survival of the fittest, and it is supported (or tolerated) by those in power since like those early psychologists it helps confirm the rightness of a status quo that is stacked in their favor. Arguing against standardized testing as the basis for judging schools often looks like an argument against basic American tenets, and if those in power actually get around to agreeing they risk undoing the very status quo that helps prop them (us) up.

That's why I go all the way back to the moment when standardized testing was invented. When you can get people to understand it was invented as a tool to analyze something that could not be directly observed and for which no measuring tape existed, it begins to create an aha moment. When you show that even those inventors admitted to themselves that they weren't actually measuring the amount of anything it starts to get people thinking. And when you show that the stability in such measures is because of their foundation in statistical averages, you can start to show the limitations in such measures. If you can get people that far into the argument.

What we still lack, however, is an easily understood, easily repeatable counter-narrative, one that can say, "it is imperative that we replace the paradigm of rank ordering students with xxx." We can start to describe the criteria for it: one in which all students can succeed (not that they necessarily will), one which acknowledges the massive differences in opportunity shown in a rank ordering need not doom those at the bottom to always being at the bottom, etc., but the thing itself is elusive.

My most recent attempt at a new narrative--one I'm still formulating--is this: America is not a zero-sum game with the requirement to have both winners and losers, so why do we depend so heavily or rank order testing that does just that? America will be ever so much better when we realize that not only can the economy support a world where the vast majority of students achieve excellence, but that is in fact a necessary thing if we are to preserve our standing in a world that is quickly gaining on us.

Non-Scientific Polls



Dear CNN and every other lover of non-scientific polls:

I beg you to please realize the degree to which you undermine your mission as organizations committed to objective reporting with such meaningless and un-interpretable data.

The sampling required for a scientific poll requires a remarkable art. When someone wants to take the pulse of a group, be it eligible voters, likely voters, neo-Nazis, or parents of two year olds, sampling statisticians go to great lengths to deploy their umpteen degrees and massive intellects to ensure that their results are as valid and reliable as possible.

What that means is in layman’s terms is that when a scientific poll is performed two days in a row, the two results would not vary much and can therefore be trusted to reflect the views of the targeted group. That requires the sample to include a mix of people fully representative of the target. Sampling statisticians deploy a wealth of tools, databases, statistical weighting processes, and sampling frames to ensure that their estimates are valid.

A non-scientific poll will produce wildly different results when it is re-administered a few hours or a few days later. It provides the opinion of the people asked, but that opinion cannot be generalized any further. You know nothing from such a poll about the larger population or the general opinion on the matter. When you stare at the results from such a poll what you see is a Rorschach blot, a nonsensical conversation starter that enables the viewer to state their opinion on something. The inferential ability from such a sample is zero, and yet presenting it to an audience and talking about it makes it seem otherwise.

When a network says, "we conducted a non-scientific survey" the only rationale responses are:
1. Ignore the response as a vapid attempt to lend credence to one person’s opinion or discredit another’s and give the talking heads a launching pad for whatever opinions they already had.
2. Switch to PBS or NPR, which make an erstwhile attempt to avoid such malpractice.
4. Cancel the network's Survey Monkey account and demand all staff who will ever comment and/or utter the word “survey” attend and pass college courses in basic statistics, sampling techniques, and common sense.

Monday, July 25, 2016

What accountability means

An organization consists of people, processes, and systems that come together to provide a benefit, without which the organization would have no reason to exist. A huge component of accountability is the measure of how well the organization provided that benefit. When the benefit isn't met then all the parts of the organization may need to change, which means that the accountability measure needs to be useful in that regard.

Schools are organizations that exist to produce a benefit for the students that attend them. It would be wonderful to think that accountability measures that have something to do with that benefit might one day be at the heart of accountability discussions. At the moment, the only benefit being rewarded are high or rising standardized test scores, which do not represent the rich benefit I think we all want for our students.