What reliability doesn’t say

A standardized test is at its simplest a data collection tool. It only works if the data collected meet a certain standard in terms of statistical reliability. Reliability is all about the consistency of the measure or observation, and generating a sufficient level of reliability to allow for reasonable inferences to be made requires both skill and planning. In the process of achieving that reliability, however, you impose a whole series of limitations on what the resulting data can say in the name of allowing it to say a few things well.

To make the idea of reliability more concrete, imagine that you have two observers of a rat moving through a maze and you ask each observer to record their observations by writing down what the rat is doing during the experiment, with no other tools than a pen and a piece of paper. Odds are the two observers will offer a related but very different narrative, which means that from a research perspective the observations would be of limited use.

The reason for the limited use is that some of the inferences drawn from one observation risk being refuted or not supported by the other, and so a researcher making any inference risks that inference not being supported by data.

Statistical reliability can be obtained if a certain amount of discipline is introduced to the observations. Adding a stopwatch would of course help because both observers would be likely to agree upon the amount of time it took. So too would limiting the observations being recorded to a list that included only the salient points of the hypothesis being studied.

Under this second scenario the agreement would offer an indication of the reliability of the judgments, and with a high degree of reliability a researcher has an increased level of confidence that their inferences can be based on good data.

What must be remembered, however, is that while such reliability enables a great deal in terms of making valid inferences, it also conceals a great deal in what goes unobserved or unrecorded. If, for example, our focus is on the number of wrong turns the rat takes under a number of different scenarios we would make note of those particular phenomena, and not on whether the rat was brown or white or tended to make its wrong turns more often to the right or to the left. It isn’t that those things are unimportant or irrelevant in a broader context, but rather, they are not a part of the question being asked and thus are not included as part of the observational milieu.

No set of reliable observations is therefore ever complete—the fact that they can be made to be reliable is an artifact of the manner in which the observations are controlled. They aren’t controlled for some nefarious purpose, but in order to create a limited number of powerful observations that can lead to increased understanding. Once those controls are put in place, however, a researcher, by definition, draws a very firm line in the sand that limits the range of possible inferences. Having done so, the vast majority of the universe of inferences is removed as a possibility, leaving only the few that are the focus of the research.

The price of reliability, then, is that it must pick and chose, and in doing so always leaves most of the universe outside of its gaze.

Standardized tests are a type of observation, with the data being collected in the form of a student’s responses. What that means is that the test itself is necessarily limited in its scope and the vast majority of the universe in which the tested content is contained is external to the tested material.

For the purposes behind a standardized test those limitations don’t pose a problem, because the purpose is pretty straightforward: show the rank ordering of students in a manner that students can be compared to each other, and one group of students can be compared to another group of students. To do that you need only test items that behave in a very narrow way: roughly half the students need to answer each of them correctly, and half incorrectly. Those are the ideal items for answering such a question since 25-30 of such items are generally enough to provide enough data to answer the question regarding where each student and group of students rank. Items within that narrow range do a very nice job of spreading students across every possible number of correct responses.

Within the universe of a domain such as reading the items on a reading test needed to show the rank ordering of students represent a tiny sliver of the domain.

Reliability is built into the instruments so that they allow for a high level of confidence in the inference regarding where a student ranks against the tested material. What they don’t allow for are for any additional inferences that were not a part of what was observed. That’s just plain logic: if you choose to narrow the focus of your observation to achieve a reliable set of data what would make you think that inferences outside and beyond what was observed are suddenly available?

So now lets think about what is outside the “observations” being made by a standardized test. That would include everything that isn’t in those items. It includes the most challenging of the material that was taught and the most basic, since none of that is on the test. That content consists of material that by the end of the year every student would have answered correctly, or perhaps the vast majority would still answer incorrectly, and thus that material fails to help answer the question regarding rank ordering. You could include it, but it wouldn’t contribute to the reliability of the measure—in fact, it hurts the reliability because it doesn’t contribute anything to the purpose of the measure.

Such tests don’t include any observations as to the quality of the teacher or the school. This often comes as a surprise to most people since the entire world now seems content to assign a quality judgment to a school based on test scores. But where within the limits of the observational lens is any question as to school or teacher quality? Rank orderings are good for the purpose of comparisons—in fact, for the purpose of offering up meaningful comparisons, they are ideal—but the placement of a student or a school within a ranking says absolutely nothing as to what caused a student or school to land at that point.

Filling in the silence beyond the student or school ranking with statements as to the quality of the school or the teachers may seem on the surface to be justified—and such judgments may in fact correlate somewhat with the reality regarding quality schools—but such statements are themselves entirely unsupported by the data. Inferences about quality made from any standardized test are in the same category as speculating that the faster mice in the maze ate a better breakfast than their counterparts without having a speck of data as to whether or not that is true.

Finally, the answers that students provide offer no advice as to what should be changed in the curriculum to support better instruction the following year, and yet policy requires that state test scores be returned in order for schools to use them for just this purpose. This is perhaps the greatest crime we commit with test scores, and takes us so far beyond the observational lens offered by a test score that it really is both shameful and laughable at the same time.

The best way I can show the paucity of instructional value from such a test is to point out the types of observations that would be needed in order to identify the candidates for improvement from one year to the next. Consider the following as a representative yet incomplete list of the kinds of questions that need to be answered, and compare that to the only question a standardized test is designed to answer regarding the rank ordering of students:

• Did a teacher teach a rich curriculum and teach it well? 
• Did a teacher differentiate instruction according to the needs of individual students?
• Were students under prepared coming in to school this year and thus in need of extra support to catch up to their peers?
• When learning failed to occur, what was the cause? Discipline issues? Personality conflicts? Novice teachers?
• Did teacher re-teach concepts and ideas that were not understood in ways that represented a “once more, but louder” approach, or did they attempt to teach the same thing but through a different lens or approach?

What should be obvious is that the data-gathering efforts that would help provide answers to these and other questions like them would require an entirely different set of observations than those required to identify the rank ordering of students. What should be just as obvious is that no matter how well we answer the question regarding how students and schools rank we cannot suddenly ask the observations to extend their reach into areas that the test doesn’t cover.

We need reliability in social science research and certainly in testing, but we need to understand that imposing it upon our observations—whatever their form—means that our opportunity to make inferences narrows to the observational target. If we think otherwise we are quite likely making inferences that lack any actual support.

If I am right in this last point—and I would argue I am—then much of what currently passes for data-driven decision-making is actually data-less decision making that we pretend is valid.