Monday, September 9, 2013

Skills assessment

Lots of attention is now being given to the notion that a skills-based education is a good thing and in our test-obsessed culture many are starting to look for assessments that indicate the presence or absence of such skills. That’s likely to lead to a whole bunch of inauthentic behaviors if we aren’t careful.

Consider that researchers are quite adept at finding ways to identify the presence or absence of such things under the guise of research, but in order to do so must take a fairly circuitous route. Questions, observations, and a host of other data gathering efforts that distill information into usable chunks are extremely valuable in allowing a researcher to make statements regarding skill attainment in schools, but the data elements almost always represent a correlation that in turn enables the inference.

In order for the inference to be valid, however, the correlation must be to the desired behavior. The instant any sort of accountability is tied to the correlation the correlate will substitute for the desired behavior—the system will follow the formal definition of success, even if it differs from what is actually desired.

If our history with standardized testing offers any lessons, policy makers—who have a long history in education of confusing correlations with cause and effect—will at some point legislate success on the substitutes. They then run the risk of presuming that as the substitute measure climbs and falls it represents something real and meaningful, when odds are, like a test score in the current system, it is as much an indicator of the degree of manipulation the system has undergone as opposed to a measure of the presence or absence of a desired behavior.

The fact that systems sway when accountability is placed on some component of them should come as no surprise—that is the intent. Educational policy makers have up to now shown not one bit of hesitancy in placing accountability on the thing that was used by researchers as the correlate to something larger and more important rather than the thing itself and then delude themselves into thinking that the correlate and the thing are one and the same.

The hope for the skills movement is that the correlates often don’t really even resemble the desired skill. A simple commonsense look should suggest that the patterns of responses in a researcher’s survey or the presence of certain traits in an observer’s checklist that may correlate to the presence or absence of a desired skill are not themselves a representation of the skill. Such things are easily manipulated by anyone who realizes that a right answer exists, and even those looking to answer honestly will skew towards any right answers they know to exist. Just knowing that a right answer exists will have that effect when the results will be used as a basis for judgment.

Accounting for a skills-based environment is critical and doable, but we are going to have to do so using a very different set of tools than anything in the current educational toolbox.

No comments:

Post a Comment