Thursday, July 5, 2018

On confirmation bias and test scores

It is a natural thing to seek out messages that confirm our preexisting beliefs or ideas about the world and other people. It is also fairly common to interpret unclear messages in a manner favorable to the beliefs or ideas we hold. However, just because it is natural or common does not mean it is also good or right. It is not—in fact, just the opposite.

The tendency to seek out confirming messages is called confirmation bias. The trouble with confirmation bias is that it always risks replacing the truth with what we might want to hear. Any resulting action then risks being the wrong action when the truth is considered, while appearing to be the right action given the bias. It allows any judgment and subsequent action to appear and feel appropriate, while it may be entirely wrong given the underlying realities.

Our preexisting beliefs or ideas can occur from a dizzying array of possibilities, can be subtle or blatant, and can be sexist or racist. They may resonate from surface resentments, or deep within the subconscious mind. Regardless, if all we do is seek out messages that match those biases, we risk living in a fictional bubble that assumes the supremacy and validity of our beliefs, while presuming, quite illogically, that it is everyone else that is illogical.

Consider the bias that exists generally in American society that schools in wealthy neighborhoods are better than those in poorer neighborhoods. It is fairly simple to see that as a bias through a simple example: consider a school in a wealthier neighborhood is full of wealthy students whose families support and contribute to their children’s’ education while the school leaders coast on demographics and offer little by way of support. If so, the school deserves to be called out. Or consider that school leaders in a poor part of the community can be shown to spend each day making up for the effects of poverty and that because of the excellent decisions being made each of their students now has a meaningful chance at a good life. If so, those school leaders deserve our applause and our support.

Or it could be the opposite. Or anything in between. The point is, you have to go look. Unless you do, you risk an erroneous judgment and actions that are themselves illogical. It would be illogical and harmful to tell the poor school above it is failing, and it must change everything when that is blatantly untrue, just as it would be to tell the wealthy school it is a wonderful success and to keep up the good work. Both are lies that help neither.

But assume that you really believe your bias that schools in poorer neighborhoods are bad, and your goal is to find evidence to that end. You might consider lining students up as of a particular day each year according to their literacy or numeracy attainment. Because numeracy and literacy attainment are heavily influenced by the quality of non-school experiences, and because students from poorer environments tend to have fewer quality experiences than their wealthier peers, we would expect to see that reflected in such any ordering performed on a given day. In other words, students who are poorer will tend to be in the lower portion of the ordering, while students who are wealthier will tend to be more towards the top.

From an objective perspective such an ordering offers nothing to judge, and certainly nothing to act on. It reveals patterns we can analyze and attempt to disrupt, but until we go look, no judgments can be made, or actions assigned. If we were ignorant enough to assign a judgment and subsequent actions to the schools mentioned above based on such an ordering, we risk judging both wrongly at the direct expense of the students.

Just for the record, state test scores are based upon a testing methodology that orders students from the student furthest below to the student furthest above average as of a date certain each year on reading and math. Their design limits their interpretive range to revealing patterns for exploration, but because they match a preconceived bias they have been ignorantly presumed to mean so much more.

A person who falls victim to their bias that a rich school is just better than a poor school risks seeing the test scores as a confirmation of that bias, elevating that bias in their mind to a truth: they believed something and now they have evidence for their belief (even though their belief was wrong, and they have no real evidence confirming it). Policy makers and society at large have long held just this sort of bias, and when they saw standardized tests scores inappropriately declared, “aha! We told you so. We now must hold those terrible schools accountable and reading and math test scores will be just the thing.”

Here’s how you can know once and for all this is just a stupid bias: we could just as easily have rank-ordered students on creative output, only that would not confirm the bias that schools in wealthy neighborhoods are better that schools in poor neighborhoods because the ordering would not follow socioeconomic lines. Like any ordering that ordering might reveal patterns for us to explore but that’s it.

And here’s the danger of a confirmation bias against poor schools: if you confirm that bias through a tool that will always put students with fewer quality numeracy and literacy experiences outside of schooling at the bottom, you condemn those schools to perpetual failure without ever considering the actual evidence. And the longer you do that, the more you insist that those schools change everything, the more inefficient you make those schools, which risks converting your bias to the truth.

In short, what was a bias grounded in fiction risks becoming a truth because you forced bad judgments and inappropriate decisions on an at-risk population that in turn serves to keep them in their marginalized place. Shame on any of us that ever thought that was an acceptable thing to do.

No comments:

Post a Comment