The need for a new kind of accountability, not a new test
Metrics-driven accountability systems absent context will always fail. Big statement, I know, but there’s lots of evidence for this fact, including the history of how school accountability has been done for the past 20-30 years. The fact is that metrics are technical tools that require expert interpreters and context, and that crude interpretations absent expertise is a recipe for invalid conclusions.
Let me show you why.
Take one of the simplest metrics I know: graduation rates. Which school is better at graduating kids: a high school that graduates 99% of its kids, or one that graduates 65%? The prevailing wisdom says that goes to the school that graduates 99% of its students. But is that actually true? The answer: you can’t know absent expertise and context. And if you think you can you are flat out wrong.
Take the 99% school. Imagine what that school probably looks like: middle or upper middle-class incomes, parents who are highly educated, minimal violence on the streets, most kids have healthcare, etc. I realize I’m generalizing here, but just go with it for a minute. Assuming that’s the school, what are the odds those students would graduate regardless of the school they attended? For that matter, what are the odds they’ll head off to college, again, regardless of the school?
The answer: pretty darn good. The simple fact is that in regard to whether or not the student will graduate, the school is adding very little value, maybe even none. Consider that if 100% of the students who graduated would graduate regardless of the school it would in fact be none. The school may well be adding tons of value in other ways, and if so, deserves lots of credit for doing so, but not in this one—the metric of graduation rates for them is virtually meaningless.
Now take the 65% school: lots of parents with incomes at or below the poverty line, regular violence on the streets, parents who want to support their kids but struggle to know how given their own educational experiences, minimal access to healthcare, or even books, etc. What would be the odds that those students would graduate without that school? In a different school? Or would they even be in school? It isn’t difficult to imagine that we could locate some considerable number of students in that school where their graduation was directly caused by those in the school.
In that case, the lower graduation rate contains evidence that the school is adding considerable value in this regard, and that absent the school the outcomes for some students would be decidedly different.
So, which is better at graduating kids? It may well be the school graduating fewer, not more of its students. The 99% school may not even know how to graduate students from the 65% school, meaning a student at risk of graduating would have a higher probability of graduating from the 65% school than the 99% school. If we ignore expert interpretations and context, we risk presuming just the opposite. If we then tell the 99% school that they’re good at something and they aren’t, that isn’t helpful. And if we tell the 65% school it’s a failure and to change everything, they risk getting rid of what’s working, which is in fact harmful.
Which is the more effective school: one with high predictive (most people call these standardized) test scores, or one with low predictive test scores? Again, common wisdom—and our formal policy—says the school with high scores, which again is an interpretation made absent expertise and context and thus is highly likely to be invalid or wrong. That expertise and context may well show that the lower scoring school is highly effective at teaching and learning in a challenging environment with a huge benefit for its students, while the high scoring school offers very little in the way of that sort of value.
Telling the high scoring they’re great at teaching when they may not be, or the low scoring school to change everything because they are failures improves nothing. In fact, just the opposite. What matters is the truth--maybe the high scoring school is the more effective school, but maybe it isn't. To jump to a convienient conclusion puts you further from the truth, not closer.
Let me show you why.
Take one of the simplest metrics I know: graduation rates. Which school is better at graduating kids: a high school that graduates 99% of its kids, or one that graduates 65%? The prevailing wisdom says that goes to the school that graduates 99% of its students. But is that actually true? The answer: you can’t know absent expertise and context. And if you think you can you are flat out wrong.
Take the 99% school. Imagine what that school probably looks like: middle or upper middle-class incomes, parents who are highly educated, minimal violence on the streets, most kids have healthcare, etc. I realize I’m generalizing here, but just go with it for a minute. Assuming that’s the school, what are the odds those students would graduate regardless of the school they attended? For that matter, what are the odds they’ll head off to college, again, regardless of the school?
The answer: pretty darn good. The simple fact is that in regard to whether or not the student will graduate, the school is adding very little value, maybe even none. Consider that if 100% of the students who graduated would graduate regardless of the school it would in fact be none. The school may well be adding tons of value in other ways, and if so, deserves lots of credit for doing so, but not in this one—the metric of graduation rates for them is virtually meaningless.
Now take the 65% school: lots of parents with incomes at or below the poverty line, regular violence on the streets, parents who want to support their kids but struggle to know how given their own educational experiences, minimal access to healthcare, or even books, etc. What would be the odds that those students would graduate without that school? In a different school? Or would they even be in school? It isn’t difficult to imagine that we could locate some considerable number of students in that school where their graduation was directly caused by those in the school.
In that case, the lower graduation rate contains evidence that the school is adding considerable value in this regard, and that absent the school the outcomes for some students would be decidedly different.
So, which is better at graduating kids? It may well be the school graduating fewer, not more of its students. The 99% school may not even know how to graduate students from the 65% school, meaning a student at risk of graduating would have a higher probability of graduating from the 65% school than the 99% school. If we ignore expert interpretations and context, we risk presuming just the opposite. If we then tell the 99% school that they’re good at something and they aren’t, that isn’t helpful. And if we tell the 65% school it’s a failure and to change everything, they risk getting rid of what’s working, which is in fact harmful.
Which is the more effective school: one with high predictive (most people call these standardized) test scores, or one with low predictive test scores? Again, common wisdom—and our formal policy—says the school with high scores, which again is an interpretation made absent expertise and context and thus is highly likely to be invalid or wrong. That expertise and context may well show that the lower scoring school is highly effective at teaching and learning in a challenging environment with a huge benefit for its students, while the high scoring school offers very little in the way of that sort of value.
Telling the high scoring they’re great at teaching when they may not be, or the low scoring school to change everything because they are failures improves nothing. In fact, just the opposite. What matters is the truth--maybe the high scoring school is the more effective school, but maybe it isn't. To jump to a convienient conclusion puts you further from the truth, not closer.
It doesn’t matter how you cut a test score: growth, value add, etc., or any metric for that matter, the opportunity for understanding where a school is or is not effective is only possible with expertise and context.
We need to finally let this fact, that blunt metrics-driven accountabilities don’t tell the truth about any organization or school, drive our actions going forward. Evidence is always a part of any good accountability, but so are expert interpretations of that evidence within the context in which the organization operates. Thinking that evidence tells the truth entirely on its own is just nonsense. If you happen to be from the business world and don’t believe this, look at the notes to a financial statement of any publicly traded company and all you will see are professional interpretations and contextualization, without which you lack the ability to make an accurate judgment of that company. Expert interpretations and context are what enable valid judgments and decisions to be made.
My primary concern at the moment is that while conversations are cropping up around the country about how to do a better educational accountability, they are frequently about a better metric, a better test, a better way to do value-add, etc. If you want to repeat the past regarding educational accountability, replacing what we have with one of these will guarantee that happens. If you finally want to understand where schools are or not effective—which is the only way to get every child a great education in schools that are always and constantly improving—we need to learn to do accountability in a very different way.
If you are interested go to bravedway.com, sign on, and you’ll start to learn how districts across the country are doing just that. What's there is free for the time being, so I hope you will. We have a chance to rethink educational accountability we may never have again, and we really need to take advantage of it.
We need to finally let this fact, that blunt metrics-driven accountabilities don’t tell the truth about any organization or school, drive our actions going forward. Evidence is always a part of any good accountability, but so are expert interpretations of that evidence within the context in which the organization operates. Thinking that evidence tells the truth entirely on its own is just nonsense. If you happen to be from the business world and don’t believe this, look at the notes to a financial statement of any publicly traded company and all you will see are professional interpretations and contextualization, without which you lack the ability to make an accurate judgment of that company. Expert interpretations and context are what enable valid judgments and decisions to be made.
My primary concern at the moment is that while conversations are cropping up around the country about how to do a better educational accountability, they are frequently about a better metric, a better test, a better way to do value-add, etc. If you want to repeat the past regarding educational accountability, replacing what we have with one of these will guarantee that happens. If you finally want to understand where schools are or not effective—which is the only way to get every child a great education in schools that are always and constantly improving—we need to learn to do accountability in a very different way.
If you are interested go to bravedway.com, sign on, and you’ll start to learn how districts across the country are doing just that. What's there is free for the time being, so I hope you will. We have a chance to rethink educational accountability we may never have again, and we really need to take advantage of it.
Comments
Post a Comment