Wednesday, May 5, 2021

Learning Loss and What it Doesn’t Mean

Let's start with a few basic assumptions.

Two choices existed during the pandemic for teaching and learning:
    1. Bore the kids with the scripted content in unfamiliar learning environments.
    2. Take advantage of the unfamiliar learning environments to engage students in ways those students would find meaningful. 
Only one of those choices was presented as a viable option by the powers that be.

What should be obvious to all of us is that as educators we have students in our presence for a limited time and we should take advantage of every minute. Guidelines are of course useful and necessary in helping us do that, while a script to be followed under any circumstances would not be. Scripts don't understand context and used uncritically will create inefficiencies and certainly fail to take advantage of the limited time we have.

What about when times aren't normal? What are the consequences of sticking to the script during a pandemic when everything we thought we knew about the script and how to deliver it gets blown up? We could still try and stay on script which isn't likely to produce the same result as normal times (which wasn’t great to begin with), or we could throw the script away. We could seek pedagogical opportunities in our immediate surroundings that could not help but be at that core of our students’ thoughts and surroundings and use that to create learning opportunities that would otherwise be impossible. If we consider the limited amount of time we have with students and need to use each moment wisely, the wisest use seems to me one that creates the greatest amount of learning benefit for students in each of those moments. Staying on script during a pandemic seems a sure fire way not to do that.

Let me present a hypothesis, one I cannot prove but that I believe is more than just a little probable. When we review standardized test data in reading and mathematics that covers the last two years of schooling, we will see a decline in the amount of reading and mathematical content in the possession of students who endured it compared to what we might expect in more normal times. We can interpret that as a loss in reading and mathematics or as part of a body of evidence that shows our attention was for a moment focused elsewhere in order to maximize learning under difficult circumstances. But then we could only interpret it that way if we knew the focus was elsewhere.

What I suspect is that the political obsession with test scores probably prevented the shift of focus to more relevant subjects more easily learned during a crisis. If that is the case, and it most likely is, then what we are observing in the reading and math scores is evidence of a wasted opportunity. Students were never going to learn as much reading and math during a pandemic as they otherwise would. That doesn't mean we shouldn't have taught it, only that such an outcome was inevitable no matter how much time we dedicated to it. So rather then follow a script whose outcome was going to be less than ideal, why not use that time more wisely? Why not turn a difficult situation to our students’ advantage? Why spend any moment of our students’ precious educational career inefficiently when another choice is available?

The great learning loss that we should be talking about is not in the students or in their test scores but in the continued willful ignorance of those who continue to insist on educational approaches that cannot possibly result in the effect they want, who insist on standardizing and scripting the educational process for all children, due to either willful ignorance or a lack of imagination. If we stretch the educational career of a student a cross their thirteen years of schooling, we should hope to see as many of the moments in that career as well spent as possible. The odds are we aren't going to see that during the pandemic because the pedagogical choice of doing what was best for students wasn’t there as an option. That needs to be seen as a problem with a remarkably simple solution.

Monday, April 26, 2021

COVID and the revelation of a facade: schooling is built for something other than student needs

Organizations can shape themselves to best serve their stakeholders, the members of the group without whom the organization loses its reason to exist, or someone else. Perhaps those in the organization. Or perhaps those outside the organization who hold power and sway over it. Either way, the choice would result in very different organizations.

A hospital built entirely to ease the lives of medical staff will look and feel very different than one designed entirely to ease the pain of patients. What looks like a choice between patients and doctors isn’t really a choice—choose the doctors at the expense of the patients and the patients will go elsewhere. It’s easy to imagine parallels in grocery stores, restaurants, tech companies, online retailers, and most certainly schools.

Of course, organizations do try and strike a balance. Which is understandable because workers should be treated well, but also something we should be aware lest that balance stray too far from the stakeholder. If we need to err, it should be towards them.

One important outcome of the pandemic is to lay bare for all to see just how far schools have been forced to move away from what will benefit students towards what is convenient for a few grownups.

What are some of those conveniences? Grade levels. Standardized content at each grade. Standardized test scores that guarantee a mixed message on schools that justifies underfunding them and underpaying teachers. Grade retention (if based on test scores, a double whammy). Class rank. College entrance exams. Data-driven decisions absent professional interpretations. Continuously assigning marks of failure to those students with the greatest need so you can avoid filling those needs. And on and on it goes.

What I hope we will lose during this pandemic—although the early signs aren’t good—is the naïveté in thinking these things are done to benefit students. The truth is these things make it easier for a few grown-ups in power. It’s much easier and cheaper to crudely drop students into age based buckets and throw standardized content at them than figure out what they need and provide it. It is much easier to declare poor and minority children as failures for having greater needs than others because that blames them rather than a society that needs to change and lets those grownups off the hook. It is much easier to treat a standardized test score falsely as a mark of quality, than address the inequities in the patterns it is designed to reveal.

You get the picture.

What is revealed in the current hand-wringing is that the grownup oriented system broke down precisely because it was grownup oriented. Once all the attention went to the students, those grownups lost their way. How will you promote or hold back students? What do we do now that schools are off the standardized script and schedule (nether of which were that good to begin with)? What do we do if suddenly we can’t confirm the erroneous bias that students in wealthier neighborhoods are smarter than those in poorer neighborhoods? What do we do now that the bias in the old system has been laid bare?

A school designed to benefit students will not look like schools prior to the pandemic that were designed to satisfy some perceived adult need. The teaching and learning that happened during the pandemic, occurring as it did in a highly disruptive environment that called a great deal into question, offers a few insights we would be wise to consider in building a student-based educational system:
  1. Standardized testing is for grownups. If those grown-ups are thoughtful researchers, it can be useful, but mostly it’s used for reasons it was never designed to support.
  2. Standards are about controlling teachers, not meaningful literacy, or numeracy.
  3. Student need for all students in the old system, regardless of actual need, was defined via the standardized content.
  4. Equity in the old system is a myth. It was considered to be achieved when all students could be shown to be in the proximity of the same content. That way the equity box could be checked by adults.
  5. During a crisis you can get mad that you can’t find the students so you can check the equity box, or mad that you live in a system in which you can’t find the students. Only one of those has an actual equity component in it.
  6. The old system confirmed the biases adults had about schools in rich and poor communities, while the pandemic showed that every child is always both a learner and a struggler, and we create a false pedagogical platform when we think otherwise.
  7. Students never stop learning, even during a crisis. They all got a little smarter, just not according to the script.
  8. Had no script existed, the learning opportunities that presented themselves over the last year would have offered much in the way of priceless content to be explored. Social and racial justice, democratic processes and challenges, leadership, climate change, and social responsibility, each presented, in real time, rich fields for inquiry and exploration that were current and relevant. Instead, the test must go on, so the adults who don’t know how to educate children insist teachers hammer away on reading and math content at the expense of a great learning opportunity.
  9. Etc.
Building back better is a really nice mantra. Let’s extend that to schools and finally design them around our mission and not our convenience.

Sunday, April 11, 2021

Let’s finally be done with high stakes testing

Let’s finally be done with high stakes testing and the accountability charade that gets wrapped around it every year.

Accountability requires the truth to work. If you tell me I’m effective or ineffective and that’s the truth I can improve. If you’re wrong, however, you’ll do more harm than good. You’ll put me at risk of changing what may not need to be changed and not changing what should.

Accountability in organizations is no different. Only the truth about both the good and the bad allows for meaningful improvement.

The public school system is perhaps the most important of our public institutions when it comes to the future of this country and the preservation of our democracy. If it fails, we fail. Of all our institutions that must account for and be accountable for the truth, schools are paramount. Which means they must have the truth.

Which, since the era of test-based accountability began, has not existed. This may come as a surprise to most, but the tests states use in what they name but is not accountability tell us nothing on their own about a school’s effectiveness or quality. This is because they were never designed nor intended to do that. The assumption that they can magically become what they are not is both old and wrong.

It stems from a simple but harmful bias. Which is easy to see if you understand what the research instrument we know as standardized testing allows us to see. You can get a sense of that if you think about the amount of mathematical knowledge possessed by a large group of eighth graders on any given day. We know those eighth graders will possess very different levels of knowledge and studying the reasons and patterns why would create real opportunities to make improvements.

One of the largest contributors to the differences in mathematical knowledge will be the cumulative years of exposure lifetime to date. For the most part, eighth graders will have similar amounts of exposure in school, but the non-school exposure will vary wildly. Even the non-school exposure during their current school year will vary wildly.

The most significant contributor to this variation will be the socioeconomics affecting each child’s life. Those from wealthier conditions are likely to have far greater non-school exposure than those from poorer conditions, and as such on a single date during eighth grade that will be reflected in an analysis of the amount of knowledge possessed.

Let me clear, the differences observed would have nothing to say about who is smarter, whether the school is effective or ineffective, the quality of a teacher, etc. Some students are smarter than others, some schools are effective and some ineffective, and some teachers are better than others, but that can’t be seen in this type of analysis.

If someone made the tragic mistake of thinking the results of such an analysis were an indicator of quality, that would mean they would be guilty of assigning labels of success or failure without knowing that was the case. And if the schools were made to act on these labels in an accountability environment, schools would be acting on a set of untruths. It would make them worse, not better.

Worse still, it would be virtually guaranteed to assign labels of failure to schools in poor communities for being in poor communities, and labels of success to schools in wealthy communities for being in wealthy communities. The nonsense in that would be stunning.

This nonsense is exactly what test-based accountability does. It takes an analytical tool that allows us to see something worth seeing, worth understanding, and worth acting on, and uses it to make imaginary judgments that just so happen to align with one of societies longest held biases: that schools in poor communities are bad and schools in wealthy communities are good. The correlation of the negative judgments to where large numbers of our minority citizens live, and the disproportionately negative impact that creates, should not be lost upon us.

And because the system as it is designed—whether intentionally or not is not the point—makes schools less effective, especially those judged negatively, the effect is both to preserve the bias and give it a reason to exist.

I can assure you, as someone who works every day with public schools on this issue of building meaningful accountability systems, that every school is effective in some ways and ineffective in others, and that with the truth they can all improve. And I can assure you that there are effective schools the state labels as failures and ineffective schools the state applauds as successful merely because of their addresses.

That, I can also assure you, hurts and damages the public educational system in ways we cannot afford. It squanders talent, harms students and communities, and puts our future at risk. The effect is the most hurtful among our most vulnerable.

An industry leader I know once commented to me that any other profession or industry would have collapsed under the weight of such conditions. The fact that public education perseveres in its mission in spite of all this, he said, is a remarkable testament to the men and women who serve it.

Just imagine if the time and energy spent fighting a bad system in order to do what is right for our children could all be dedicated to doing what is right for our children. It is a good thing to think.

Accountability won’t go away when test-based accountability dies, but it will be different. It will need to be disciplined, student and community focused, and most critically based in the truth. Arguments like mine against high stakes testing are really just calls for a true educational accountability. For the truth about our schools.

High stakes testing should be done away with this year, next year, and for as far into the future as any of us can imagine. It doesn’t tell us the truth, it prolongs and intensifies biases that hurt us as a society, and most embarrassing of all, it can’t, by design, do what it purports to do.

Sunday, July 12, 2020

The need for a new kind of accountability, not a new test

Metrics-driven accountability systems absent context will always fail. Big statement, I know, but there’s lots of evidence for this fact, including the history of how school accountability has been done for the past 20-30 years. The fact is that metrics are technical tools that require expert interpreters and context, and that crude interpretations absent expertise is a recipe for invalid conclusions.

Let me show you why.

Take one of the simplest metrics I know: graduation rates. Which school is better at graduating kids: a high school that graduates 99% of its kids, or one that graduates 65%? The prevailing wisdom says that goes to the school that graduates 99% of its students. But is that actually true? The answer: you can’t know absent expertise and context. And if you think you can you are flat out wrong.

Take the 99% school. Imagine what that school probably looks like: middle or upper middle-class incomes, parents who are highly educated, minimal violence on the streets, most kids have healthcare, etc. I realize I’m generalizing here, but just go with it for a minute. Assuming that’s the school, what are the odds those students would graduate regardless of the school they attended? For that matter, what are the odds they’ll head off to college, again, regardless of the school?

The answer: pretty darn good. The simple fact is that in regard to whether or not the student will graduate, the school is adding very little value, maybe even none. Consider that if 100% of the students who graduated would graduate regardless of the school it would in fact be none. The school may well be adding tons of value in other ways, and if so, deserves lots of credit for doing so, but not in this one—the metric of graduation rates for them is virtually meaningless.

Now take the 65% school: lots of parents with incomes at or below the poverty line, regular violence on the streets, parents who want to support their kids but struggle to know how given their own educational experiences, minimal access to healthcare, or even books, etc. What would be the odds that those students would graduate without that school? In a different school? Or would they even be in school? It isn’t difficult to imagine that we could locate some considerable number of students in that school where their graduation was directly caused by those in the school.

In that case, the lower graduation rate contains evidence that the school is adding considerable value in this regard, and that absent the school the outcomes for some students would be decidedly different.

So, which is better at graduating kids? It may well be the school graduating fewer, not more of its students. The 99% school may not even know how to graduate students from the 65% school, meaning a student at risk of graduating would have a higher probability of graduating from the 65% school than the 99% school. If we ignore expert interpretations and context, we risk presuming just the opposite. If we then tell the 99% school that they’re good at something and they aren’t, that isn’t helpful. And if we tell the 65% school it’s a failure and to change everything, they risk getting rid of what’s working, which is in fact harmful.

Which is the more effective school: one with high predictive (most people call these standardized) test scores, or one with low predictive test scores? Again, common wisdom—and our formal policy—says the school with high scores, which again is an interpretation made absent expertise and context and thus is highly likely to be invalid or wrong. That expertise and context may well show that the lower scoring school is highly effective at teaching and learning in a challenging environment with a huge benefit for its students, while the high scoring school offers very little in the way of that sort of value.

Telling the high scoring they’re great at teaching when they may not be, or the low scoring school to change everything because they are failures improves nothing. In fact, just the opposite. What matters is the truth--maybe the high scoring school is the more effective school, but maybe it isn't. To jump to a convienient conclusion puts you further from the truth, not closer.

It doesn’t matter how you cut a test score: growth, value add, etc., or any metric for that matter, the opportunity for understanding where a school is or is not effective is only possible with expertise and context.

We need to finally let this fact, that blunt metrics-driven accountabilities don’t tell the truth about any organization or school, drive our actions going forward. Evidence is always a part of any good accountability, but so are expert interpretations of that evidence within the context in which the organization operates. Thinking that evidence tells the truth entirely on its own is just nonsense. If you happen to be from the business world and don’t believe this, look at the notes to a financial statement of any publicly traded company and all you will see are professional interpretations and contextualization, without which you lack the ability to make an accurate judgment of that company. Expert interpretations and context are what enable valid judgments and decisions to be made.

My primary concern at the moment is that while conversations are cropping up around the country about how to do a better educational accountability, they are frequently about a better metric, a better test, a better way to do value-add, etc. If you want to repeat the past regarding educational accountability, replacing what we have with one of these will guarantee that happens. If you finally want to understand where schools are or not effective—which is the only way to get every child a great education in schools that are always and constantly improving—we need to learn to do accountability in a very different way.

If you are interested go to bravedway.com, sign on, and you’ll start to learn how districts across the country are doing just that. What's there is free for the time being, so I hope you will. We have a chance to rethink educational accountability we may never have again, and we really need to take advantage of it.

Monday, June 29, 2020

Closed versus open system in the current world

Eddie Dean, a restaurateur and philosopher in Dallas, occasionally sends me notes that make me think. This morning it was about open verses closed systems. He’s sent me similar notes before since this is a topic we’ve frequently discussed.

And over the past few weeks I’ve spent considerable time with my friend, George Thompson, talking about the difference between a learning organization, which is an open system, and bureaucracies, which are closed systems. It's always interesting to me how different conversations with very different people will coalesce like that.

Of course, my own work is about accountability, which has a considerable opinion on these topics as well and is the lens I choose.

Accountability in a closed system is about the rules. Increasing accountability means finding new ways to hold those within the organization to account. We have two massive closed systems that society is asking some pretty serious questions about at this very moment: policing and attitudes about race and our racist history. If we continue to treat both as closed systems, which we have for a very long time, we’ll add some rules to policing and some training on racial sensitivities and call it good.

What is so interesting and compelling—and unnerving to those who are comfortable within the old closed systems—is the nature of the current calls for change. They are not to add rules or ask people to please play nice, but to trade out those old closed systems for open systems, to point out that no matter how many more rules or trainings are thrust into the old systems that won’t solve the problem.

I often point out that whatever result we get it is because we are in a system that is perfectly designed to deliver that result. Policing in its current incantation is currently designed with force as a primary tool of control, so we shouldn’t be surprised when force is used. Our race relations in this country are currently designed to marginalize non-white people economically, socially, historically, and educationally—with one result being a disproportionate use of force on non-whites. So, we shouldn’t be surprised when those systems work exactly as designed and create that marginalization and a disproportionate use of force.

Our solutions of holding the police more accountable through rules on chokeholds or holding society accountable through things like affirmative action or non-discrimination ordinances, are perfectly rationale responses from within those systems—and infinitely better than nothing. But they leave the problem intact by functioning within the old closed system.

What is being declared right now is that we need new systems but based on a very different design that is capable of learning and growing and changing as we learn and grow and change. That is the novel piece of what is happening now, and what gives me a real sense that a better way may finally be possible.

School and by extension how we do school accountability are closed systems. Conversations about how we do schooling and school accountability are about more or different rules within that system, more control when things don’t go policy makers’ way, and a strange obsession with making things like they once might have been years ago. The same is true for our traditional approaches to police reform or race relations. A return to how we imagine things were yesterday seems preferable for some than looking ahead to the uncertainties that come with possibility.

But that past is gone. It's dead, and good riddance in so many ways. We can do better, which will require a willingness to embrace open systems, which will require us to embrace the concept of a learning organization in our policing and race relations and our schools, and to realize that until we do the future is at risk of looking an awfully lot like the past.

I’m looking forward to it.

Sunday, May 31, 2020

What happens if we remember we're all actually related?

We are all of us cousins. Every human being on the planet. It may not be through a long-lost aunt or a great grandfather, but it’s probably not much more than a great great great grandparent. That’s remarkable. If you believe in science, we all come from a common ancestor from 200,000 years ago and our ancestors’ paths have probably crossed multiple times since. If you believe the world came into being six or seven thousand years ago—I don’t but I’ll grant it for the moment—then our common ancestor is even more recent, and we’re more cousins than ever.

We need to acknowledge this, that we’re all related, all one family. Millions of our cousins around the world are sick. Millions more are impoverished and living in slums. Millions are governed by cousin tyrants who don’t seem to care about their extended family, or forgot they are part of one. And many millions more continue to wonder why bombs and militaries are more important to some than food, healthcare, education, and children. A few thousand of our wealthiest cousins control most of the world’s resources and could change the course of history if they wanted, but they’ve shown few signs that’s what they intend to do.

From within this big collection of cousins we have some who are just flat out terrible, and they deserve a spanking and then some. But we also have a ton of our cousins willing to do something, even though it’s hard. These are the cousins who see a wreck in the middle of the night and without hesitation risk their life to pull another cousin to safety. Or who put on last week’s soiled surgical mask to help a cousin overcome COVID. Or who teach the future generation of cousins and give a hungry child their lunch, because that’s just what you do when you have a little more than someone who has nothing. Or that protest against racism, sexism, despotism, bias of all kinds, and generally mean people. All these cousins and more need to pull together now more than ever. The future depends on it. Our youngest cousins may not have a future if we don’t.

I live in America, by the simple fact of birth. I’m white, male, and while I was raised by loving parents who struggled financially it was never a question whether or not I would make it. And I did. I’m not rich, but squarely middle class, and I have what I need. And here’s the truth. I worked really hard, but I didn’t work any harder than a million others who didn’t make it. I legitimately tried, but I can’t say I tried harder than all the rest. And yet I’m here and so many aren’t.

I always had an invisible advantage, an unseen leg up on the part of America that didn’t look like me. I have never walked down a street worried about being singled out and silenced or harmed. I have never walked by a police officer hoping they weren’t one of the few bad ones and today is about to be my unlucky day. I walk out my door every day expecting I’ll get a fair shake. I never worried I was being under paid. No one has ever crossed the street to avoid me or been fearful just by being in my presence. My bet is that if I ever commit a crime, I’ll be given the chance to turn myself in, and probably even negotiate on the terms of my surrender. If I do go to prison, my invisible advantage is likely to get me the benefit of the doubt when it comes to my sentencing.

And let me be clear about something—I get a lot of attention when I walk down any street. I’m a one-armed man. I lost an arm nearly to my shoulder in an accident forty-nine years ago when I was six, so people—cops included—have been staring at me my whole life. I got stared at when I was six and it happened a few days ago on my fifty-fifth birthday, and most days in between. And while a few who don’t know me may feel sorry for me (shame on them for judging—they should get to know me first), not one time did I come under suspicion for my difference. Not one time was I ever mistreated by an authority for being who I was. Not one time was I ever presumed guilty for being born. I used to think that made me lucky, but that’s wrong. To claim I’m lucky to be born white presumes it’s best to be white, when it should be best to be who you are. It has never been the case in this country where everyone is given the chance to do that.

Way too many of us have forgotten that 200,000 or six thousand years ago, take your pick, we would have called the same people grandma and grandpa. They wouldn’t look exactly like us or communicate like us, but that’s not the point. The point is we’re all connected.

Imagine explaining to this grandma and grandpa slavery, and describing how the tiniest of genetic differences, the pigmentation in one’s skin, led to the notion that a certain color made some cousins worth not very much as humans, but a great deal as property. Imagine explaining the massive scope of slavery in America’s history, and the fact that much our country was built on their uncompensated backs. I imagine these original grandparents would express outrage and fury at that sort of treatment of their family members, and then relief that it was outlawed a hundred and fifty years ago. I can also imagine them expressing even more outrage when they learned that it took a century for the country to finally admit a bit of wrongdoing and extend some basic civil rights to those descendants of former slaves who had been denied even that. And even more outraged when they discovered the number of cousins who had their fingers crossed when the admission was made. And even more if they could see the number of people who act as if they’re sorry it was even said.

Imagine telling them that a lot of people in the wealthiest country that their great great great great great grandchildren had ever created now regularly apply bias to practically everyone with non-white skin, and accept as good and right treatment of those with dissimilar pigmentations that they condemn and punish harshly when done to the similarly pigmentated. And now it’s not just pigmentation, but language as well. Who knows what will be next? Our grandparents would likely wonder why so many of the cousins always seem to need someone to pick on, or even hate, and how that could possibly make a person feel better.

Last week some of my cousins who I don’t know thought it was a good idea to gather their semi-automatic weapons and march into the Michigan senate. Their pigmentation happened to be white and so they were kindly escorted out and given a scolding. This week an unarmed black cousin who I also didn't know but had a lot fewer opportunities than me and may or may not have tried to buy something with a counterfeit twenty-dollar bill died when a cop decided handcuffs and compliance with his demands weren’t enough and kneeled on his neck until he was dead. I’m just glad that the cousins with their semi-automatics weren’t black and didn’t meet that cop, because I don’t think they would have been kindly escorted anywhere, except to prison, via tear gas, handcuffs, and some knees on some necks.

I’ve had it. I haven’t always understood the advantage I have of being a white guy because from my angle it looked invisible. Just part of the status quo. But it isn’t invisible to lots of my cousins. It’s not innocent or innocuous. And the more it gets ignored the more likely it is to become malignant, to justify violence against those who do see it by those who refuse to. I’m learning to see it. I’ve been learning for a long time, and I’ll be learning it for the rest of my life if that’s what it takes. I hope someday we’re all able to see it, maybe even at the same time, because at that exact moment suddenly there won’t be anything to see. We’ll all just be cousins again.

Let’s get there. It’s time.

Monday, May 18, 2020

Being accountable to a test result...

Being accountable to any test result is being accountable to the wrong thing. Right now, the most important test in the world is for the Coronavirus. The information it provides is immensely useful, and yet to treat that information as more than information about the presence or absence of the virus is a mistake.

Neither outcome tells us anything about a person’s overall health. Neither outcome signals anything about what has happened or what will happen. And both outcomes come with a caveat—there is a small possibility of the result being wrong, of suggesting you have it when you don’t, or that you don’t when you do. To treat either outcome as more than it is absent contexts, details, and a whole lot of additional information renders any next step invalid, likely to be unhelpful, or even harmful.

All tests suffer from this limitation. It is a consequence of trying to squeeze as much precision as possible out of a single result, and the necessary price we pay for needing and trying to do so. More accurate results provide confidence that studies of the contexts, details, and any applicable information can be more expertly applied. But really, all any result does is move us a step or two away from chaos. It does not, as is so commonly and wrongly presumed, put us a step or two away from surety. And while that is still so much better than having no information at all, it is no more than one piece of a much larger puzzle.

What would be terrible for all of us is a lockstep approach that failed to consider context, that applied a generic solution to a result, or that refused to consider the unique conditions of an individual. Medicine would be reduced to a simple decision tree and we would be infinitely worse off than we are. It would be like thinking we’re through with a puzzle after the first two pieces come together.

Educational testing based on a specific methodology—the variety used in state testing programs, or the norm-referenced tests sold commercially, such as the Iowa Test of Basic Skills, or NWEA’s MAP—is now guilty of encouraging that exact sort of behavior. These too are tests that produce a narrow result that move us a step or two from chaos but no further. The results are nothing more than points on a continuum (some of which will be wrong) based on a moment in time that lacks context, cause, or professional interpretation. Yet to sell more product or to support bad educational policy, the declaration gets made that the results are more than they are, that they can directly inform teaching and learning, indicate quality or effectiveness, and replace professionalism.

This is as false and misleading and harmful as thinking that a diagnosis equates with a solution. All test results require interpretation through the broader technical lens of a professional equipped with the full context of the individual’s situation and current best practices. And they require the ability to question that lens, to recognize it as always incomplete and able to be improved upon. Only then is the professional capable of determining an optimal path forward for that student or patient while at the same time being responsible for making that path better for the next time.

I used to be kinder to the test publishing world—especially when I was in it and it was paying my bills and I still believed we were capable of staying within the limitations of what a test is—but the field has strayed way too far from its usefulness of putting tools in the hands of a researcher and instead has become something else altogether.

We would never tolerate straying so far from what a thing is in the tools that will help us through the pandemic because the consequences would be unthinkable. We shouldn’t tolerate it in the education of our nation’s children for the exact same reason.