Tuesday, October 6, 2015

A response regarding an assay

I responded to a colleague's email this morning and thought it worthy of a post:

Hi Ted--

Our accountability tests are necessarily uni-dimensional. That is, in order to generate consistent results over time we pick a single "construct," find "average" within the target population on that construct, and then measure out from that point to determine how far above or below average a student is (of course we don't use those terms, replacing them with numbers that sound less judgmental). That construct has to be uni-dimensional or "average" as the basis of consistent scores or the scores won't be stable over time. Thus a standardized test could be viewed as a sort of assay since it attempts to detect the degree to which a "thing" varies within a population.

However, a standardized test--contrary to much popular belief--is only designed to measure the distance from average for a tested subject, which means it can never and has never provided an indicator regarding the amount of actual learning or achievement that took place. It is not a graduated cylinder that measures volumes, but rather, a narrow measuring device that estimates relative distances from a stable central point. Those distances identify differences in a population without ever identifying what is actually in those distances (if only I could help policy makers understand this).

As far as the notion of a single target entity such as a standardized test being the point of an assay, the idea is spot on.

As to "analysis" you point out its most relevant meaning and how far gone it is in today's schools. This occurs exactly as you point out when schools fail to realize that they aren't paying attention to all the elements of a student's learning in both quantity and quality. In fact, when it comes to both quantity and quality our accountability tests are absolutely silent, by design, which then leads to the massive inefficiencies that plague our systems today when we treat tests as saying something light years beyond their design. (It is nearly as bad as if we'd asked a thermometer to tell time by dumping out the mercury and trying to read it like tea leaves.)

The assay effect (a fine essay title) is greatly exacerbated by uni-dimensional tests that schools and policy makers pretend have something to do with what was actually learned. This misunderstanding hurts the opportunity for actual analysis by replacing it with an assay and then pretending analysis is actually occurring. That leaves schools thinking that they have an analytical environment that can support learning, when very little of that sort of thing actually exists given the lack of understanding regarding the choice of instrumentation.