Friday, June 8, 2018

Why judgments based on a rank position are stupid (technical term)

Several people forwarded a Seth Godin blog on forced rankings. See it here. I’ll add two cents to the conversation.

A forced ranking in education is: 1) is useful for some types of limited analysis; 2) has no capacity to offer a valid judgment, and 3) if used as a judgment tool almost always falls into the trap of a confirming an existing bias rather than reflecting the truth. Ironically, they are rarely used for analysis. Their most common usage in education is to confirm what people already (often wrongly) believe.

First, its limited usefulness: a ranking (or ordering) can be useful for detecting patterns in single traits where very little information is actually available (when lots of information is available it is far better to use other, more nuanced tools). Once a researcher forces a ranking he/she can search for patterns within that single trait. Some of those patterns may need to be undone. A salary differential between men and women, for example, would be one such pattern. At a later point in time, the ranking can be performed again to see if the pattern was successfully disrupted.

Note that I said single trait. You can order people on the relative differences in height or hair length, but not both at the same time. This represents an extraordinary limitation, one that must be considered, or misinterpretations will occur. You have to stick to a single trait or your analysis will be useless. Any forced ranking that first combines unlike things (which the rankings of colleges do ad nauseum, for example) makes the ordering meaningless. I could use some elegant mathematics to combine hair length and height, force it into a ranking, and then hide behind the math and declare it meaningful, but that doesn’t make it so.

Second, forced rankings are stupid judgment tools (stupid should be a technical term—I can think of no other word that captures the absurdity of judgments forced on a place in a ranking). A place in a ranking can be high or low or somewhere in the middle. To place a judgment at any point requires the irrational by assigning the same judgment to every person/event/college at that point prior to knowing the cause, which is the only thing that should be judged. In the case of colleges, one college may be at a point in a ranking due to great leadership and a committed faculty, while another may be at that same point due to its ability to attract a certain student demographic. Placing the same judgment on both would be silly—they are profoundly different institutions and their differences deserve to be seen accurately.

Third, those in any sort of position of privilege love forced rankings because of a thing called confirmation bias, which is another way of declaring an addiction to messages that confirm what people in a position of privilege want to believe: that they truly are the best. That bias is powerful. Would they accept a forced ranking that placed Harvard dead last? Or that didn’t list the attractive schools in wealthy neighborhoods ahead of the disheveled looking schools in the poor part of town? They would not.

I say “they” to make a point—the bias confirmed through a ranking occurs from a viewpoint. This is simple to see given all the ways to rank that aren’t selected that would show a very different order to things.

For example, student musicality, creative output, success at 4H competitions, and fluency in more than one language, to name but a few, would offer rankings that would not automatically discriminate against poorer communities, nor automatically put wealthier ones on top. If the preferred viewpoint is that schools in wealthier neighborhoods are the best schools, an ordering based on numeracy and literacy will help confirm that bias, while an ordering based on 4H programs would not. The two rankings would be profoundly different.

In both cases an analysis could be performed, and meaningful patterns identified, but any judgments made at any point in either ranking is a judgment made absent the underlying facts. But because those living in wealthier neighborhoods got to pick the ordering that confirmed their bias, they went with the ranking with the best fit.

That isn’t to say that numeracy and literacy are unimportant, or that serious discrepancies don’t exist that need to be addressed, but rather, that judgments based on a place in an ordering are highly likely to be false, specious, and dead wrong.

The biased selection of the ranking that fits your limited world view and its use to judge others does not make that view correct, but merely biased. It sacrifices the truth for a good feeling at the expense of others. It is always wrong to do.

No comments:

Post a Comment