Yet one more doomed attempt to link standardized testing and instruction
The recent acquisition by HMH of NWEA seems to have been driven by a phenomenally flawed logic: that normative (i.e., standardized, predictive) testing can be tied to instruction. This error that apparently seems so rational on the surface is so fraught with absurdities that it’s hard to imagine—perhaps think of using a fish as a hammer on a ship since they both occupy a spot in the ocean. I wish I were exaggerating, but I can’t think of anything more absurd at the moment—it's the wrong tool for the job by as far as the eye can see.
Standardized testing is a highly technical research methodology that sorts students onto a scale for the purpose of analyzing the relative differences between them. That’s a mouthful. Think of it like this: pretend you wanted to analyze height, but the measuring tape hadn’t yet been invented yet. You could still do an analysis by lining everyone up from the shortest to the tallest. From there you can find the average, identify how many steps above or below average a person is, and more importantly, start to look for patterns. Some we might expect, such as men being generally taller than women, and jockeys being shorter than basketball players.
But some we would not expect. Perhaps people from one country tend to be shorter than another. Or one country has both the tallest and the shortest. Those represent opportunities for further exploration that might result in some useful findings—but only after additional explorations. On their own, the ordering is valueless—only with additional data and evidence are any sort of interpretations possible, because until we have a clear sense of cause, we can’t know what anything means.
That’s the problem with any scale—you can put people on it, but you can’t ever know why they are where they are absent that additional exploration. When you put students on a scale based on, say, literacy, you can see that they are above or below average, or that their position shifted over time, but the why is always somewhere else. Maybe a great teacher did a great job. Maybe a student matured and grew a foot over a year. Maybe a teacher screwed up. Maybe what happens outside of school is mostly responsible for the score, be it high or low.
The point is you can’t know unless you go look. Until you do you are at risk of assuming the same causes for every observed test score when nothing of the sort has ever existed. The causes for any effect—especially something as complex as numeracy or literacy—are always multiple, and to make a good interpretation an investigation is always necessary.
This isn’t a condemnation of this sort of testing—just a fact. The content may come from a states content, but still, the cause for it getting into a student’s brain can’t be known until you go look. Assuming otherwise means you’ll mostly be wrong.
As long as I have been in education (much of it in the world of testing) publishers been seeking a magical connection between standardized testing and curriculum and haven’t found it for the simple reason that it doesn’t and never will exist. Curriculum and teaching are designed to be highly recursive functions that require constant interpretation on the part of the teacher based on that specific moment in time. Standardized testing is designed to allow for an analysis at a moment in time but has no ability to comment on the quality of the decisions made (or not made) to arrive at that point. One is a fish, one is a hammer, and no amount of wishful thinking (or investment of money) is enough to transmogrify them both into something that by design they can never be.
Standardized testing is a highly technical research methodology that sorts students onto a scale for the purpose of analyzing the relative differences between them. That’s a mouthful. Think of it like this: pretend you wanted to analyze height, but the measuring tape hadn’t yet been invented yet. You could still do an analysis by lining everyone up from the shortest to the tallest. From there you can find the average, identify how many steps above or below average a person is, and more importantly, start to look for patterns. Some we might expect, such as men being generally taller than women, and jockeys being shorter than basketball players.
But some we would not expect. Perhaps people from one country tend to be shorter than another. Or one country has both the tallest and the shortest. Those represent opportunities for further exploration that might result in some useful findings—but only after additional explorations. On their own, the ordering is valueless—only with additional data and evidence are any sort of interpretations possible, because until we have a clear sense of cause, we can’t know what anything means.
That’s the problem with any scale—you can put people on it, but you can’t ever know why they are where they are absent that additional exploration. When you put students on a scale based on, say, literacy, you can see that they are above or below average, or that their position shifted over time, but the why is always somewhere else. Maybe a great teacher did a great job. Maybe a student matured and grew a foot over a year. Maybe a teacher screwed up. Maybe what happens outside of school is mostly responsible for the score, be it high or low.
The point is you can’t know unless you go look. Until you do you are at risk of assuming the same causes for every observed test score when nothing of the sort has ever existed. The causes for any effect—especially something as complex as numeracy or literacy—are always multiple, and to make a good interpretation an investigation is always necessary.
This isn’t a condemnation of this sort of testing—just a fact. The content may come from a states content, but still, the cause for it getting into a student’s brain can’t be known until you go look. Assuming otherwise means you’ll mostly be wrong.
As long as I have been in education (much of it in the world of testing) publishers been seeking a magical connection between standardized testing and curriculum and haven’t found it for the simple reason that it doesn’t and never will exist. Curriculum and teaching are designed to be highly recursive functions that require constant interpretation on the part of the teacher based on that specific moment in time. Standardized testing is designed to allow for an analysis at a moment in time but has no ability to comment on the quality of the decisions made (or not made) to arrive at that point. One is a fish, one is a hammer, and no amount of wishful thinking (or investment of money) is enough to transmogrify them both into something that by design they can never be.
Comments
Post a Comment