Tuesday, July 10, 2018

On standards standardizing

Formal standards standardize something. It is useful to standardize some things, such as electrical outlets, allowable car emissions, and the minimum requirements to become a doctor, lawyer, or nurse. Standardizing the outlet means that any electrical device with a compliant plug will fit, regardless of who manufactured either of them. Standardizing allowable car emissions helps keep the air clean. Standardizing minimum requirements to enter certain professions is intended to ensure a basic level of quality and protect citizens from those selling snake oil.

It should be noted that the standards that exist in the world have a profound impact on each of our daily lives. I can go to any gas station and know that the gas nozzle will fit in my car, rather than having to find a Honda nozzle for my Honda car. I can buy a car and trust it will meet federal standards regarding emissions. I can go to the doctor and know that at the very least they met the compliance requirements to be a doctor, and because I know those requirements to be fairly robust, have some confidence in that person never having met them.

Standards are met through compliance. When the plug fits the socket, the level of emissions is below a threshold, or the prospective doctor completes the last of the requirements to be a doctor, it can be said that in each case compliance with the standard was accomplished.

Standards are about control. An ideal standard is one that controls only what it must to achieve some end. We control the dimensions of a plug, which creates an infinite number of possibilities for whatever is powered at the end of that plug. We eliminate the opportunity for car manufacturers to make a profit off of cars that are bad for the environment. We control who can and cannot become a doctor for the sake of protecting citizens from harm.

A terrible standard standardizes the wrong things, which in turn creates inefficiencies and/or frustration. Standardizing the length of the cord attached to the plug to one or fifty feet would be silly as it would not match with the needs of both consumers and their selected product. A standard that required every car to have a gas tank in a safe place (think Ford Pinto), an internal combustion engine, and a quality exhaust system would eliminate the creativity required to build electric cars. A “desire to help” as the standard for entering the medical profession would doom a great many patients.

In schools we have standardized the length of the cord and the internal parts to schooling, as well as removed any standard for who can make educational decisions or even run a school. In short, the wrong things.

We standardize who is in what grade by age, regardless of need, life experiences, or the distance to some set of meaningful goals. We standardize the inputs through lists of controlled content (appropriately called standards given that they control what is to be taught) aligned to standardized tests that will be administered as the output, with the goal that all students will leave standardized around a particular test score (which for those of you who read my writings know is not actually possible), not to mention the bureaucratic requirements to standardize teacher actions throughout a school day.

And if you can fog a mirror but have never set foot in a school since you were a student you are all but qualified to apply for and run a charter school in most states.

We have to standardize some things in a school, just as any organization does. But what if we had approached schools like those who standardized the ordinary three-pronged plug did and standardized only what was absolutely necessary? What would those things be?

It isn’t a simple question. Outcome-based education was one attempt to standardize outcomes and let the inputs vary according to need, but it ran into a political buzz saw (and for legit reasons—there are a million ways to get to a successful adulthood and standardizing one or two of those belies that fact). The standards movement tried to standardize teaching and learning, and now we have lots of bored kids not learning to the depths they should, and teachers tired of not being able to focus on what students actually need. The test-based accountability movement tried to standardize an outcome each year on a date certain, by which all kids would have learned the years’ worth of material and demonstrate that learning via a test score. The silliness in that is just embarrassing (read my book, or anything I’ve ever written if you want the full argument as to why).

We should think about that question: what could we standardize in education that would help us maximize the educational benefit we can provide to each and every student within the limited resources available to us? We need to identify those things. And then we should figure out how to get there.

Quick note: I posted something similar to this before and someone wrote for sources. See chapter four in Pitfalls of Reform, for what is still one of the more detailed descriptions of the problem I've written. I'll tackle this issue at length when I get around to finishing the book I've been working on since Pitfalls was published.

Thursday, July 5, 2018

On confirmation bias and test scores

It is a natural thing to seek out messages that confirm our preexisting beliefs or ideas about the world and other people. It is also fairly common to interpret unclear messages in a manner favorable to the beliefs or ideas we hold. However, just because it is natural or common does not mean it is also good or right. It is not—in fact, just the opposite.

The tendency to seek out confirming messages is called confirmation bias. The trouble with confirmation bias is that it always risks replacing the truth with what we might want to hear. Any resulting action then risks being the wrong action when the truth is considered, while appearing to be the right action given the bias. It allows any judgment and subsequent action to appear and feel appropriate, while it may be entirely wrong given the underlying realities.

Our preexisting beliefs or ideas can occur from a dizzying array of possibilities, can be subtle or blatant, and can be sexist or racist. They may resonate from surface resentments, or deep within the subconscious mind. Regardless, if all we do is seek out messages that match those biases, we risk living in a fictional bubble that assumes the supremacy and validity of our beliefs, while presuming, quite illogically, that it is everyone else that is illogical.

Consider the bias that exists generally in American society that schools in wealthy neighborhoods are better than those in poorer neighborhoods. It is fairly simple to see that as a bias through a simple example: consider a school in a wealthier neighborhood is full of wealthy students whose families support and contribute to their children’s’ education while the school leaders coast on demographics and offer little by way of support. If so, the school deserves to be called out. Or consider that school leaders in a poor part of the community can be shown to spend each day making up for the effects of poverty and that because of the excellent decisions being made each of their students now has a meaningful chance at a good life. If so, those school leaders deserve our applause and our support.

Or it could be the opposite. Or anything in between. The point is, you have to go look. Unless you do, you risk an erroneous judgment and actions that are themselves illogical. It would be illogical and harmful to tell the poor school above it is failing, and it must change everything when that is blatantly untrue, just as it would be to tell the wealthy school it is a wonderful success and to keep up the good work. Both are lies that help neither.

But assume that you really believe your bias that schools in poorer neighborhoods are bad, and your goal is to find evidence to that end. You might consider lining students up as of a particular day each year according to their literacy or numeracy attainment. Because numeracy and literacy attainment are heavily influenced by the quality of non-school experiences, and because students from poorer environments tend to have fewer quality experiences than their wealthier peers, we would expect to see that reflected in such any ordering performed on a given day. In other words, students who are poorer will tend to be in the lower portion of the ordering, while students who are wealthier will tend to be more towards the top.

From an objective perspective such an ordering offers nothing to judge, and certainly nothing to act on. It reveals patterns we can analyze and attempt to disrupt, but until we go look, no judgments can be made, or actions assigned. If we were ignorant enough to assign a judgment and subsequent actions to the schools mentioned above based on such an ordering, we risk judging both wrongly at the direct expense of the students.

Just for the record, state test scores are based upon a testing methodology that orders students from the student furthest below to the student furthest above average as of a date certain each year on reading and math. Their design limits their interpretive range to revealing patterns for exploration, but because they match a preconceived bias they have been ignorantly presumed to mean so much more.

A person who falls victim to their bias that a rich school is just better than a poor school risks seeing the test scores as a confirmation of that bias, elevating that bias in their mind to a truth: they believed something and now they have evidence for their belief (even though their belief was wrong, and they have no real evidence confirming it). Policy makers and society at large have long held just this sort of bias, and when they saw standardized tests scores inappropriately declared, “aha! We told you so. We now must hold those terrible schools accountable and reading and math test scores will be just the thing.”

Here’s how you can know once and for all this is just a stupid bias: we could just as easily have rank-ordered students on creative output, only that would not confirm the bias that schools in wealthy neighborhoods are better that schools in poor neighborhoods because the ordering would not follow socioeconomic lines. Like any ordering that ordering might reveal patterns for us to explore but that’s it.

And here’s the danger of a confirmation bias against poor schools: if you confirm that bias through a tool that will always put students with fewer quality numeracy and literacy experiences outside of schooling at the bottom, you condemn those schools to perpetual failure without ever considering the actual evidence. And the longer you do that, the more you insist that those schools change everything, the more inefficient you make those schools, which risks converting your bias to the truth.

In short, what was a bias grounded in fiction risks becoming a truth because you forced bad judgments and inappropriate decisions on an at-risk population that in turn serves to keep them in their marginalized place. Shame on any of us that ever thought that was an acceptable thing to do.