Friday, August 24, 2018

Do you support sending our best teachers to our most challenging school environments?

The question in the title to this blog has been posed to me twice in the last week, which I think is due to states releasing their accountability judgments of schools just before the kids all come back.

Common sense might suggest that only an idiot would say no. I'm not an idiot (according to most people I meet) but I'm here to say that we'll do more harm than good if we oversimplify our responses and just say, "sure."

First, we need to identify what we mean by a best teacher. I can do that easily. A best teacher is one who can maximize an educational benefit for the children in his or her classroom. We could get even more specific if we wanted and say that the best teacher for a child is the one who can maximize the benefit for that specific child, but for the sake of the argument here, lets keep it general: a best teacher is one who can maximize the educational benefit for children.

Second, we need to identify what we mean by our most challenging school environments. I can also do that easily. Those would include places that have historically had poor school leadership, or teachers not committed to their profession, or that serve communities of children who through no fault of their own find themselves in environments that make learning a real challenge and the students would benefit from additional help and support.

To take those best teachers and ask them to serve in our most challenging environments as described above would, by any stretch of the imagination, be a good use of a valuable resource.

However, there are huge issues for how we identify best teachers and our most challenging environments given that state test scores tend to be the main mechanism for doing both. Schools with high test scores are presumed to be quality schools, while schools with low scores are presumed to be bad schools, and so too with the teachers in the building. Thus a simplistic approach to selecting the best teachers and placing them into our most challenging environments would be to take teachers from high scoring places and put them into low scoring places.

This would likely do more harm than good for two basic reasons: first, state test scores on their own (contrary to popular belief) were never designed to identify the quality of a teacher. Test scores of the type that produce consistent results over time are useful to researchers in that they can signal where a researcher ought to take a deeper look, but whether that researcher finds something worthy of a positive or negative judgment is a different issue entirely.

For example, a group of students from a low-scoring high school classroom may all still be in school due to a dedicated teacher who prevented them from dropping out—that is not failure on the part of the teacher, but rather, evidence that we might want that teacher in our most challenging environment.

Or consider a group of students in a high scoring school who are where they are entirely because of stable homes, highly educated parents, and a desire for higher education, and their performance can be shown to be the same regardless of which teacher was placed before them. That is not evidence of success on the part of those teachers, but of something else. And since we have no real evidence of their effectiveness, we have no way of knowing if placing them in our most challenging environment will have the desired effect.

If we assume that state test scores on their own identify the best teachers, we risk sending the wrong teacher to our most challenging environments, and the entire system of education would be less, not more efficient as a result.

The exact same is true when trying to define our most challenging environments. If we rely on test scores as the signal we risk disrupting schools in which effective work is taking place and not sending the best teachers to where they can be most effective. Replacing the teacher mentioned above who has proven capable of keeping at risk students in school risks putting a less capable teacher in his or her place. Replacing an effective school leader with strong ties to the community that can be leveraged to support at risk students with a less effective leader may occur if you try and judge leadership through test scores without looking at underlying effects.

The risk is this: by relying on test scores to identify the best teachers and our most challenging environments, we risk sending unqualified teachers to replace qualified teachers, and we risk sending them to places where they are not needed as opposed to places where they are.

You do that and you'll hurt children far more than you'll help them--and mostly those who really need us. I'm all for sending the best teachers to our most challenging environments, but only once we have a valid means for identifying both.


Monday, August 13, 2018

School grades as snake oil that is good for no one

Just because someone offers you a snake oil cure for how to improve the quality of public schools doesn’t mean you have to swallow it. In fact, you shouldn’t.

The latest snake oil cure in Texas is school accountability via school grades. I know a good bit about school accountability—I make a living from the topic and have a deep-seated belief that true accountability is both necessary and achievable. And that snake oil isn’t the answer.

It’s easy to see the snake oil for what it is if you back up and ask a simple question: how does accountability work in successful organizations? I’ve explored the answer for years, written a book and a bunch of articles on the topic, and now work with schools to put in place what I’ve discovered. The answers to the question reveal the difference between a false accountability that will miss every policy goal it claims to support, and a true accountability that can move an organization in a desired direction.

I’ll mention three principles of true accountability to make my point.

Principle number one is that true accountability requires a complete, not a partial accounting. You wouldn’t invest in a company that provided one month’s worth of records and insisted it represented the entire year. You wouldn’t know the meaning behind a set of financial records without the shareholder’s report that explained the company’s performance and its plans for the future. Nor would you trust a non-profit that claimed to help people but refused to disclose how your donations were being spent.

Partial accountings that attempt to substitute for a full accounting must always be considered invalid. They can tell a story, but it will never be a full story, and risks being a wrong story. Any action against a partial accounting risks being a wrong action that makes things worse.

Principle number two is that true accountability must account for the mission of the organization, not just what is convenient to see or measure. The example I use all the time is the mission of the light bulb verses its measures. I can measure a great deal about a bulb—in fact, lights bulbs have measurable standards down to the tiniest detail so that any light bulb will fit into any socket and be as bright as any other 60-watt bulb no matter who manufactured it.

But if I focus only on what can be measured I miss the mission. The Louvre at night, the lit stage at the Metropolitan Opera, or a city skyline just after dusk represent the mission of the light bulb. If you only care about what can be measured, you risk that mission never being realized. And most of what matters in life and in organizations is at the mission, not the measurement level.

Principle number three is that true accountability demands contextualization to be accurately understood. Raw data that shows one company’s profits at 30%, and another’s at 1% aren’t comparable absent a context. Grocery store chains build hugely successful business models at very low margins, while a tech company needs a much greater margin to keep up with a constantly changing world. Some companies considered successful haven’t yet turned a profit and don’t plan to for years.

Absent the context for each, no judgments can be made. The grocery business cannot be judged as more or less successful the others absent the context. The business that has not yet turned a profit may in fact be the most successful of them all. It is the context that reveals the truth.

Note that context never equals excuse.

Make no mistake about carrying out these principles: some leaders do so far better than others, which in a quality organization necessitates changes. The most productive change occurs when individuals learn and improve, and while most can do just that, some either cannot or will not and for the good of the organization are asked to work elsewhere.

The snake oil of school grades violates each of these principles (and, just for the record, they violate each of the other principles not mentioned here as well) to the point that to call it accountability is to misname it.

First, school grades are by definition a partial accounting. They combine reading and math scores from end of a year tests (which given the design of the tests don’t mean what most people think) with several other annualized variables to produce a grade. This is in fact a partial accounting of the few things being analyzed, which in turn are a partial accounting of what happens in schools. Therefore the stories that results ask being wrong.

Second, school grades occur at the moment of the measure and never account for the mission of schooling. The mission of schooling should be to maximize the educational benefit for each child in the finite amount of time we have them in the educational system, so they are well-prepared to tackle life. By stopping at the measure, we put that mission in jeopardy, which harms kids.

And third, school grades are presented absent a contextualization. What was the focus of the school for the year? What were the issues unique to the student population and what was done to properly serve those needs? What are the hopes and dreams of the parents for their children, and to what degree is the school making those a reality? To what degree was the school effective? Understanding the context may well reveal that a school with fairly low test scores is serving its students, their parents, and the community effectively, while a school with fairly high test scores is not. That truth would be useful and actionable. The potential falsehood presented through decontextualized test scores and their resulting decontextualized grades would not.

A better way is possible. Asking the main accountability question: for what and to whom? offers anyone willing to ask it an insight into the mission of schooling and what it must attempt in order to properly serve students. A true accountability to that mission is a higher accountability than anything represented in the snake oil of school grades, and far more demanding of educators. A true accountability incents the truth, demands continuous improvement, and puts benefitting students front and center. It will reveal that some leaders do all of this better than others and insist that that those who lag behind their peers learn and grow themselves, with consequences when they do not or cannot.

The school grading system in Texas needs to be recognized for the snake oil it is. Don’t buy into its false promises of being clear and meaningful or offering a true path to improvement. Its failure to align with even one of the accountability principles reveals it for the charlatan that it is, and if you pretend otherwise it is the children of Texas who will bear the brunt.

We can and must do better.

Wednesday, August 8, 2018

The most bogus claim I've heard in months: that school grades are fair

Education Commissioner Morath in the great state of Texas is about to release grades for schools. His quote:

"The idea that the design of the system was meant to highlight both high levels of student achievement and high levels of educator impact makes this essentially the fairest system in the history of the state of Texas." (Article by Julie Chang in the Austin American Statesman, August 7, 2018--see it here--italics are mine.)

The claim in italics is bogus. And it is easy for anyone to see why.

Think of what it means to assess a student. You can do that by imagining the full range of assessment done to create understanding regarding a student or a school as a large sphere, with many layers to it. Trying to understand all the complexity to properly assess student or school needs and assign appropriate judgments is a constant, on-going thing. It requires trained teachers, lots of effort and energy, and proximity to the students being assessed.
Tests are, by design and definition, a focused, limited form of assessment. You can think of each test as focused on a small portion of the surface of the sphere. But that's it. No test exists that can provide a complete assessment, standardized tests aren't even designed go below the surface, and even all the tests in the world can't do the full job of assessment.

When doctors use tests to guide their assessment of a patient, they generally do lots of tests--why? Because tests frequently produce contradictory or inconclusive results. It is then the doctor's job as the chief assessor to interpret the results from the various tests to the benefit of their patient. It is the job of educators to do the same. Should either not do that their conclusions risk being dead wrong for either the patient or the student, with serious consequences either way.

Doctors understand the complete lack of validity in making a complex prognosis from a single test--it would be unethical to do so. Educators understand the complete lack of validity in extending the results from a single, narrow test to a broader judgment that ignores the more critical assessment sphere--and that it would also be unethical to do so.

It is imperative that if judgments about schools are going to be made they must address the entire assessment sphere and get to the level of understanding. That would be the definition of fairness. Any thing short of that is, by definition, unfair. Commissioner Morath, who uses a single test as if it can assess the whole of a student and a school, would be wrong to suggest that what he has done is fair or that his judgments are accurate. A simple understanding of assessment, as well as what a standardized test is, how they work, and the limits in their design, are the only things standing in the way of the Commissioner seeing this.