Q&A Regarding Texas Testing and Accountability

Texas just released more of its school grading nonsense. I imagined myself in a Q&A situation. This was done with my home state in mind, but this applies much more broadly. It is way too long for a blog, but it needs to be somewhere.

1. What is a standardized test like the ones used by Texas to create its ratings designed to show?

Surprisingly little. Standardized testing is a methodology that allows a researcher to observe the patterns in a population of students relative to a defined domain. So, for example, if students in some neighborhoods have more of the domain than in others. Its genius is that it does this without ever asking or knowing how much of the domain any student possesses. No one can measure the amount of literacy any student possesses—just as you can’t measure the amount of humor, cleverness, or grit any student possesses. But it is possible to observe when a student has more or less of a trait than their peers. That is a tiny bit of information, but it is enough to line students up from the student with the least to the student with the most. It isn’t a perfect methodology, but at scale, any patterns the methodology reveals likely exist in real life, enabling a researcher to explore ways to do something about them. That is what standardized tests are designed to show.

2. Can a standardized test score be used to judge teaching and learning?

On its own, absolutely not. The continuum in a set of standardized test scores is based on the total amount of the trait (relative to other students) regardless of how or where the student got to that point. Where a student is along the continuum is always due to multiple factors, some of which come from school, and some of which come from outside school. Those that come from outside school often have a far greater effect than those that happen in school.

A standardized test score, which can only reflect domain totals, cannot say what parts came from the school and what came from elsewhere. You could have a student who scores slightly above average that does so entirely because of what happened outside school. You could have a student who scores slightly below average due entirely to the efforts of a school. Drawing a compliance line between them and assuming it signaled effective schooling would be irrational. It would assign a failing mark to the school that did a great deal, and an effective mark to the school that did nothing.

3. So why are standardized tests used to judge schools when you just said they can’t do that?

In a word: bias. Most of us drive by a school and immediately develop a sense of whether it is a good or bad school. That’s human nature. Most people in positions of privilege live in neighborhoods where the sense is that the schools are good. Those schools tend to have relatively high standardized test scores, which are then misunderstood as indicators of effective schools, especially when those schools send more kids to college, end up with better jobs, etc.

But a great deal of that is an artifact based on lots of students for whom that would happen regardless gathering in the same neighborhood, not because the school in that neighborhood is causing it. That school may well be a wonderful school, but if so, it deserves to be judged on its merits, not its zip code.

Nevertheless, logic rarely prevails. Instead, because schools that are perceived as bad have lower test scores, and schools that are perceived as good have higher test scores, standardized test scores are taken as a signal of quality when that was never the case. That occurs prior to anyone ever checking as to the underlying truths.

This thinking is pervasive among journalists, real estate agents, policy makers, the business community, and most of the American public. It is always wrong.

Worse, if you draw a compliance line at any point along the continuum, you can look to the right and for the most part see schools that are from wealthier neighborhoods than the schools seen to the left. How could anyone mistake that for an accountability that tells us anything about school effectiveness?

4. But wait. We can see test score growth from one year to the next. Can’t we judge schools based on that?

That seems more reasonable, I agree, but it still requires research to know why movement from one year to the next occurred. Even if we didn’t have the embarrassingly bad math in most growth calculations, what gets learned over time has as many causes as the total amount of what a student knows at a moment in time. To assign all the cause to a school is again to make a judgment prior to having any evidence on which to base that judgment.

5. Has there ever been an instance where compliance could signal effectiveness?

Not that I have found and I can't imagine I ever will. Compliance is an accountability tool that offers broad protections from organizations or people doing dumb things, but it never signals if the organization or a person is also doing great things. A driver’s license makes sure people know what a stop sign is and who has the right of way, but compliance with those requirements offers no signal of great driving. FERPA, professional licensure, and emissions limits are the same.

There are accountabilities that are entirely about effectiveness, and becoming more effective, but none of them draw a line in the sand and pretend that they’ve identified effectiveness and ineffectiveness. Only in schools has this error been committed.

6. Wait. You’re saying that school accountability in its current form cannot identify effectiveness?

Yes. And even worse, it prevents the improvement it claims to want.

This has to do with what happens when you draw lines across metrics and pretend that effectiveness is on one side and failure on the other. Think about graduation rates. If a school regularly graduates all its students, and all those students would likely graduate no matter the high school attended, that school is not an example of how to graduate students. If a school regularly graduates 85% of its students and most of those students are at risk of graduating no matter the high school attended, that school may be a terrific example of how to graduate at risk students. In fact, if the demographic changes in the 100% school, they may need to visit the 85% school to see if they can pick up some pointers.

Should you draw a line at, say 92% and declare it the signal of effectiveness, you just made the 100% school the example for the 85% school, and labeled the effective practices in the 85% school as a failure. That would make the entire system less, not more effective, all but eliminating the chance to learn and improve.

The exact same thing is true with test scores but that can be a bit challenging to see. But what it boils down to is that we spend roughly $2 billion a year on a school effectiveness system that was never designed to tell us anything about school effectiveness.

7. Should schools be using the items answered incorrectly to make detailed curricular decisions?

Never. This seems on the surface logical, but it will have the effect of freezing test scores. One reason for this is simple. A standardized test is designed to reflect an entire domain. When more of that domain knowledge can be made to exist, the standardized test scores should reflect that.

Which begs the question: how can more domain knowledge be made to exist if teaching is artificially constrained to the tested pieces of the domain? The answer is it can’t. If the test doesn't change much from one year to the next you may see a slight boost in scores, but at the expense of the student and their learning.

But there is also a more complex answer, and that has to do with how the questions are selected that get included. Only questions that are answered according to pre-conceived statistical patterns will produce results that can be compared from one year to the next. To achieve that, the questions are selected for their ability to observe differences in what students know, not, where, how or what they know came to be. In other words, the item selection process all but eliminates any chance to glean detailed curricular information from the test, no matter how tempting it may be to do it.

8. Did not having the results harm student performance as the Commissioner claimed?

If he is referring to schools having access to missed questions as the basis for instruction, see #8. I can see no basis for that claim.

9. Why are so many more schools receiving bad marks than before?

If it were only a few more schools, that might be due to a change in test scores. But it is a 240% increase. That could only be accomplished by changing where or how the lines are drawn.

10. The number of questions that students need to answer correctly to pass changed. What does that mean?

In the big scheme of things in standardized test land, pretty much nothing.

Underlying any standardized test is a scale. Think of it as a set of stairs that cover 15-feet of elevation. Each of those stair steps represents a position along the scale. That means that if a student is at say step 16, they likely have a bit more of the trait than the students who are at step 15, and a bit less than the students who are at 17. The trick is to figure out where students are along that scale, which is what the test questions, if they are carefully selected according to the patterns in how students answer them, will do.

Now, think of each question on a standardized test as a stair step. That would put the easiest question on the lowest step, and the most difficult question on the highest, and order the rest in between. The difference between the highest step and the lowest step will likely only be noticed by the statistician or psychometrician building the test, but the differences will be there.

Now, with a 30-item test each step will be six inches tall. But if it were a 15-item test each step would be a foot tall. On a 60-item test each step would be about 3-inches tall. The scale is still 15-feet tall, but the number of positions along it will increase or decrease based on the number of items on the test.

That is anything but a perfect picture of a standardized test, but it makes the point.

But it also shows how confusing this can all get (and I’m still wildly oversimplifying this). For example, you could have a 30-item test in which the 16th step was marked as passing. That same step would equate to the 32nd step on a 60-item test. But if you move the bar, it can really get confusing: 30 correct on a 60-item test would land you at a lower step than getting 16 correct on a 30-item test, while a 33 on the larger test would place just a hair higher than the 16 on the shorter. (If you followed that, congratulations, because it is about as confusing as a thing can get. If you want to make this even more confusing, realize that even if both tests are the same length, that tests over time differ. That could mean that to get to the 16th step one year required 16/30 correct, but the next year it was 17/30, and the next it was 15/30. Then, when you switch to a longer test…)

What appears to have happened on some of the tests in Texas is that additional steps were added at the top of the stairs (that can be done by adding some questions that are significantly more difficult than in prior tests), the total number of questions increased, and the bar got moved higher. What that looks like is that the test has a lot more questions on it, but students can answer a lower percentage of those questions and still pass, and yet fewer are passing. The optics of that are terrible: it risks looking like schools are doing worse than before on a lowered bar when that is not the case.

All this is why professional test makers use scale scores to refer to the results of their work. Those scores can be said to refer to the stair steps, no matter how many questions are on the test.

11. Is there a better test that would make a difference?

Not as an accountability tool if we’re just going to draw more compliance lines and then pretend the lines signal effectiveness. You could subject the most perfect test in the history of the world to that treatment and it would corrupt the results.

12. Can you talk about standardized testing and bell curves for a minute?

One of the claims made repeatedly by the Commissioner in Texas is something to the effect that the system is valid because all students can pass the test if they all do well enough on it, and that they can all pass because they do not impose a bell curve on the results. This requires its own mini-Q&A:

1. Do the Texas test results follow a bell-shaped curve? No. If they did, we would likely have quotas so that regardless of the scores a similar number of students would be in each proficiency bucket each year. We don’t have that.
2. Do the Texas test results follow a curve? Of course. All well-constructed standardized tests do, and the Texas tests, like most state tests, are well-constructed tests (that, by the way, does not mean you can use them however you want). Their whole point is to position students along a continuum from the student with the least of the trait to the student with the most so you can investigate patterns. In any given year you can expect there will be a few students out at the extremes and lots of students towards the middle.
3. Because it isn’t a bell-shaped curve, does that mean all students can pass the test as the Commissioner has argued in the past? It is entirely disingenuous to make that claim. While it is theoretically possible to imagine that today’s average could one day be the lowest score at some point in the future, that is a philosophical pipe dream because it has never happened. Furthermore, the results at that hypothetical moment will still show a continuum from the student with the least of the trait to the student with the most. Should that happen fifty years out, do you really think the state would have left its old passing score alone for all those years, and at that point declare that all students had finally passed the test?
4. What would happen if all students scored at the top end of a standardized test? The test would be declared invalid. It exists to show the differences between students, not their similarities. If all students looked similar in the results, the test maker would not have done their job properly.
5. Do tests exist that all students could pass? Of course. Those would be tests of learning created by teachers and used in a classroom. If all students learned the taught material well, and all students aced the exam, that would be evidence of a good thing happening. But if that happened in standardized test land, we would go back to the drawing board and start over. All students learning something to a high level is not what standardized testing cares about. Its design limits it to only being about differences in what was learned, whatever that is.

12. Why do we continue down this path year after year?

Great question. You cannot name any situation where a theory of action was tried for three+ decades and did not have the desired effect and the answer each year is, “let’s do that again.”

But the fact that we are still on this path is a signal that we can’t seem to imagine something different. But it is possible to imagine a post-testing accountability universe, and all are now capable of getting there. See “A Complete, Coherent, & Practical Theory on Trust, School Accountability & What Can be Done to Right a Great Wrong in Our Lifetimes” at https://www.brave-ed.com/papers as but one example.