Friday, September 27, 2013

Data-less decisions


(Reading the previous post first may help—this one follows from it)

A data-less decision in education is just that: a decision made absent supporting data. Data-less decisions are bad for the simple reason that whatever decisions are made tend to be in support of an existing bias. Such bias can be positive or negative, very fair and objective or extremely unfair and subjective. Sometimes the bias is based in what is actually true, but just as often it is based on an untruth or a stereotype. All this is why the mantra of data-driven decision-making has been established as a proper goal for educators.

The problem is that if I look at a student in a particular situation and I possess no meaningful data I am highly likely to let any number of my biases enter in to my view of the student. This can include but is certainly not limited to my views on gender, race, socioeconomic status, whether the school is in an urban, suburban, or rural setting, and perhaps the quality of the football team or the student’s status as a star athlete.

For example, the data are quite clear that suburban schools tend to outperform urban schools by a large margin for any number of reasons. Thus if the only thing I know about a student is the that the school of record is in an urban setting it would be a fairly natural thing to presume that from an achievement perspective the student might be expected to under-perform against suburban peers. If I acted upon such a supposition when assigning that student to classes I would be making a data-less decision heavily colored by that bias.

Having done so I may be guilty for promulgating a status quo I dislike. It may be that the student has the capacity to be the next Einstein and yet having assigned the student to remedial classes I helped preserve a stereotype rather than shut it down. Most of the time data-less decisions are made against a huge combination of different types of bias that are manifest in far subtler ways, but the pattern almost always seems to be toward the preservation of the status quo and not the other way around.

This isn’t to suggest that people are by nature racist or evil or mean. Many data-less decisions are made with the best of intentions. The point is that we are each products of a history that is anything but neutral, and the simple truth is that much of our bias has some basis in fact—e.g., urban schools as of this particular moment in time do in fact under-perform against their suburban counterparts—or stems from a historical precedent that can be tough to shake. The promise of data-driven decisions is that such bias can be removed from our decisions.

The most dangerous data-less decisions are those that appear to be supported by data. Those decisions risk reinforcing whatever societal norms exist under the false pretense that the data suggested that as the proper thing to do. The data, in that situation, act as a Rorschach blot, allowing you to think you see something that puts forth an argument for your approach when the empirical reality may be otherwise. Data-less decisions that appear to have the support of data risk justifying a bias that need not in fact exist. Such decisions help solidify such bias rather than disrupt it.

Nowhere is the data-less decision more prevalent than in the use of test data in schools. Standardized test data have a very limited range of potential uses by design. Included in that design is the ability to compare schools and students to each other as an aid to identifying which schools have solutions that are working that can then be applied elsewhere.

Not included in that design is pretty much everything else. Those comparisons are silent as to their cause, so any assumptions about the practices that produced the comparisons have to come from a place outside of the test data. So too with judgments regarding the quality of the school, the nature of the curriculum, and whether or not a teacher did or did not do his or her job properly. (Policy makers continue to assume otherwise much to the detriment of our students and schools, but bad policy cannot make test scores magically perform an act for which they were never designed.)

The majority of judgments being made about schools and teachers from test data are then data-less judgments, and any decisions made from such judgments are themselves data-less.

Due to the nature of test data, however, the data-less-ness is almost impossible to see. Remember that a set of standardized test data offers a statistical representation for how things are at the moment, and the test itself is designed to show where each student and school falls within that overall representation at that moment in time. Test data, then, actually reflect whatever biases happen to exist as of a given moment in time. Test data are neutral when it comes to what those biases are, in that they don’t care what biases exist. Test data will reflect them regardless.

That means it is a very tempting thing to look at the rank ordering of schools and conclude that those that rank near the bottom are lousy schools and those at the top are great schools, because in many cases that may well be true. However, any such judgment from the test data alone is a data-less judgment, since test data are silent as to their cause and are not designed to make judgments regarding quality.

That is so hard to see through. If we take a slice in time measure it will show the effect of whatever bias exists in the world. If at that same moment we add several additional slice in time measures designed to answer the additional questions we have we may be able to make some fairly accurate statements regarding the quality of a school and those working inside it.

But having taken those slice-in-time measures our goal must be to take a set of actions that remedy the shortcomings and advance the cause of education. If those remedies are successful, then a new set of slice in time measures should provide evidence of our progress.

Instead, what we now do is make one of those original slice-in-time measures—a test, which by definition is very limited and incapable of comments regarding quality—the basis for our remedies. We take an instrument selected and created for its capabilities to show us what a rank ordering at looked like yesterday and use it as the basis for defining tomorrow.

Here is where we need to really pause and ask ourselves a very tricky question: if we are basing tomorrow upon a measure designed to show yesterdays rank ordering, might we in fact be guilty of preserving the bias that existed when that original rank ordering took place? Rather than allowing education to progress and designing a new instrument that showed us the results of that progress, by using the original instrument over and over and placing the quality determination squarely within it might we in fact be guilty of further entrenching ourselves in an old status quo when all we really want to do is escape it?

As we heap data-less judgments regarding the quality of the teacher and the school on to the system two very clear and very contradictory messages emerge. The first is the altruistic insistence that teachers and schools advance the cause of education for their students and serve them well. The second is the very pragmatic demand that success is about teaching to a test designed to reveal the biases of yesterday. Accountability will be measured not by the altruistic message, but by how well you perform against a definition of reality that should now be out of date.

Lots of metaphors apply here: running on ice, running in circles, shooting yourself in the foot, you name it.

Having operated under such a scenario should we really be surprised that our test-based culture has failed to produce the transformations it promised? Or should we finally realize that believing the false promise of a test-based culture to magically transform the status quo is perhaps one of the greatest barriers to seeing that actually happen.

No comments:

Post a Comment