Monday, June 29, 2020

Closed versus open system in the current world

Eddie Dean, a restaurateur and philosopher in Dallas, occasionally sends me notes that make me think. This morning it was about open verses closed systems. He’s sent me similar notes before since this is a topic we’ve frequently discussed.

And over the past few weeks I’ve spent considerable time with my friend, George Thompson, talking about the difference between a learning organization, which is an open system, and bureaucracies, which are closed systems. It's always interesting to me how different conversations with very different people will coalesce like that.

Of course, my own work is about accountability, which has a considerable opinion on these topics as well and is the lens I choose.

Accountability in a closed system is about the rules. Increasing accountability means finding new ways to hold those within the organization to account. We have two massive closed systems that society is asking some pretty serious questions about at this very moment: policing and attitudes about race and our racist history. If we continue to treat both as closed systems, which we have for a very long time, we’ll add some rules to policing and some training on racial sensitivities and call it good.

What is so interesting and compelling—and unnerving to those who are comfortable within the old closed systems—is the nature of the current calls for change. They are not to add rules or ask people to please play nice, but to trade out those old closed systems for open systems, to point out that no matter how many more rules or trainings are thrust into the old systems that won’t solve the problem.

I often point out that whatever result we get it is because we are in a system that is perfectly designed to deliver that result. Policing in its current incantation is currently designed with force as a primary tool of control, so we shouldn’t be surprised when force is used. Our race relations in this country are currently designed to marginalize non-white people economically, socially, historically, and educationally—with one result being a disproportionate use of force on non-whites. So, we shouldn’t be surprised when those systems work exactly as designed and create that marginalization and a disproportionate use of force.

Our solutions of holding the police more accountable through rules on chokeholds or holding society accountable through things like affirmative action or non-discrimination ordinances, are perfectly rationale responses from within those systems—and infinitely better than nothing. But they leave the problem intact by functioning within the old closed system.

What is being declared right now is that we need new systems but based on a very different design that is capable of learning and growing and changing as we learn and grow and change. That is the novel piece of what is happening now, and what gives me a real sense that a better way may finally be possible.

School and by extension how we do school accountability are closed systems. Conversations about how we do schooling and school accountability are about more or different rules within that system, more control when things don’t go policy makers’ way, and a strange obsession with making things like they once might have been years ago. The same is true for our traditional approaches to police reform or race relations. A return to how we imagine things were yesterday seems preferable for some than looking ahead to the uncertainties that come with possibility.

But that past is gone. It's dead, and good riddance in so many ways. We can do better, which will require a willingness to embrace open systems, which will require us to embrace the concept of a learning organization in our policing and race relations and our schools, and to realize that until we do the future is at risk of looking an awfully lot like the past.

I’m looking forward to it.

Sunday, May 31, 2020

What happens if we remember we're all actually related?

We are all of us cousins. Every human being on the planet. It may not be through a long-lost aunt or a great grandfather, but it’s probably not much more than a great great great grandparent. That’s remarkable. If you believe in science, we all come from a common ancestor from 200,000 years ago and our ancestors’ paths have probably crossed multiple times since. If you believe the world came into being six or seven thousand years ago—I don’t but I’ll grant it for the moment—then our common ancestor is even more recent, and we’re more cousins than ever.

We need to acknowledge this, that we’re all related, all one family. Millions of our cousins around the world are sick. Millions more are impoverished and living in slums. Millions are governed by cousin tyrants who don’t seem to care about their extended family, or forgot they are part of one. And many millions more continue to wonder why bombs and militaries are more important to some than food, healthcare, education, and children. A few thousand of our wealthiest cousins control most of the world’s resources and could change the course of history if they wanted, but they’ve shown few signs that’s what they intend to do.

From within this big collection of cousins we have some who are just flat out terrible, and they deserve a spanking and then some. But we also have a ton of our cousins willing to do something, even though it’s hard. These are the cousins who see a wreck in the middle of the night and without hesitation risk their life to pull another cousin to safety. Or who put on last week’s soiled surgical mask to help a cousin overcome COVID. Or who teach the future generation of cousins and give a hungry child their lunch, because that’s just what you do when you have a little more than someone who has nothing. Or that protest against racism, sexism, despotism, bias of all kinds, and generally mean people. All these cousins and more need to pull together now more than ever. The future depends on it. Our youngest cousins may not have a future if we don’t.

I live in America, by the simple fact of birth. I’m white, male, and while I was raised by loving parents who struggled financially it was never a question whether or not I would make it. And I did. I’m not rich, but squarely middle class, and I have what I need. And here’s the truth. I worked really hard, but I didn’t work any harder than a million others who didn’t make it. I legitimately tried, but I can’t say I tried harder than all the rest. And yet I’m here and so many aren’t.

I always had an invisible advantage, an unseen leg up on the part of America that didn’t look like me. I have never walked down a street worried about being singled out and silenced or harmed. I have never walked by a police officer hoping they weren’t one of the few bad ones and today is about to be my unlucky day. I walk out my door every day expecting I’ll get a fair shake. I never worried I was being under paid. No one has ever crossed the street to avoid me or been fearful just by being in my presence. My bet is that if I ever commit a crime, I’ll be given the chance to turn myself in, and probably even negotiate on the terms of my surrender. If I do go to prison, my invisible advantage is likely to get me the benefit of the doubt when it comes to my sentencing.

And let me be clear about something—I get a lot of attention when I walk down any street. I’m a one-armed man. I lost an arm nearly to my shoulder in an accident forty-nine years ago when I was six, so people—cops included—have been staring at me my whole life. I got stared at when I was six and it happened a few days ago on my fifty-fifth birthday, and most days in between. And while a few who don’t know me may feel sorry for me (shame on them for judging—they should get to know me first), not one time did I come under suspicion for my difference. Not one time was I ever mistreated by an authority for being who I was. Not one time was I ever presumed guilty for being born. I used to think that made me lucky, but that’s wrong. To claim I’m lucky to be born white presumes it’s best to be white, when it should be best to be who you are. It has never been the case in this country where everyone is given the chance to do that.

Way too many of us have forgotten that 200,000 or six thousand years ago, take your pick, we would have called the same people grandma and grandpa. They wouldn’t look exactly like us or communicate like us, but that’s not the point. The point is we’re all connected.

Imagine explaining to this grandma and grandpa slavery, and describing how the tiniest of genetic differences, the pigmentation in one’s skin, led to the notion that a certain color made some cousins worth not very much as humans, but a great deal as property. Imagine explaining the massive scope of slavery in America’s history, and the fact that much our country was built on their uncompensated backs. I imagine these original grandparents would express outrage and fury at that sort of treatment of their family members, and then relief that it was outlawed a hundred and fifty years ago. I can also imagine them expressing even more outrage when they learned that it took a century for the country to finally admit a bit of wrongdoing and extend some basic civil rights to those descendants of former slaves who had been denied even that. And even more outraged when they discovered the number of cousins who had their fingers crossed when the admission was made. And even more if they could see the number of people who act as if they’re sorry it was even said.

Imagine telling them that a lot of people in the wealthiest country that their great great great great great grandchildren had ever created now regularly apply bias to practically everyone with non-white skin, and accept as good and right treatment of those with dissimilar pigmentations that they condemn and punish harshly when done to the similarly pigmentated. And now it’s not just pigmentation, but language as well. Who knows what will be next? Our grandparents would likely wonder why so many of the cousins always seem to need someone to pick on, or even hate, and how that could possibly make a person feel better.

Last week some of my cousins who I don’t know thought it was a good idea to gather their semi-automatic weapons and march into the Michigan senate. Their pigmentation happened to be white and so they were kindly escorted out and given a scolding. This week an unarmed black cousin who I also didn't know but had a lot fewer opportunities than me and may or may not have tried to buy something with a counterfeit twenty-dollar bill died when a cop decided handcuffs and compliance with his demands weren’t enough and kneeled on his neck until he was dead. I’m just glad that the cousins with their semi-automatics weren’t black and didn’t meet that cop, because I don’t think they would have been kindly escorted anywhere, except to prison, via tear gas, handcuffs, and some knees on some necks.

I’ve had it. I haven’t always understood the advantage I have of being a white guy because from my angle it looked invisible. Just part of the status quo. But it isn’t invisible to lots of my cousins. It’s not innocent or innocuous. And the more it gets ignored the more likely it is to become malignant, to justify violence against those who do see it by those who refuse to. I’m learning to see it. I’ve been learning for a long time, and I’ll be learning it for the rest of my life if that’s what it takes. I hope someday we’re all able to see it, maybe even at the same time, because at that exact moment suddenly there won’t be anything to see. We’ll all just be cousins again.

Let’s get there. It’s time.

Monday, May 18, 2020

Being accountable to a test result...

Being accountable to any test result is being accountable to the wrong thing. Right now, the most important test in the world is for the Coronavirus. The information it provides is immensely useful, and yet to treat that information as more than information about the presence or absence of the virus is a mistake.

Neither outcome tells us anything about a person’s overall health. Neither outcome signals anything about what has happened or what will happen. And both outcomes come with a caveat—there is a small possibility of the result being wrong, of suggesting you have it when you don’t, or that you don’t when you do. To treat either outcome as more than it is absent contexts, details, and a whole lot of additional information renders any next step invalid, likely to be unhelpful, or even harmful.

All tests suffer from this limitation. It is a consequence of trying to squeeze as much precision as possible out of a single result, and the necessary price we pay for needing and trying to do so. More accurate results provide confidence that studies of the contexts, details, and any applicable information can be more expertly applied. But really, all any result does is move us a step or two away from chaos. It does not, as is so commonly and wrongly presumed, put us a step or two away from surety. And while that is still so much better than having no information at all, it is no more than one piece of a much larger puzzle.

What would be terrible for all of us is a lockstep approach that failed to consider context, that applied a generic solution to a result, or that refused to consider the unique conditions of an individual. Medicine would be reduced to a simple decision tree and we would be infinitely worse off than we are. It would be like thinking we’re through with a puzzle after the first two pieces come together.

Educational testing based on a specific methodology—the variety used in state testing programs, or the norm-referenced tests sold commercially, such as the Iowa Test of Basic Skills, or NWEA’s MAP—is now guilty of encouraging that exact sort of behavior. These too are tests that produce a narrow result that move us a step or two from chaos but no further. The results are nothing more than points on a continuum (some of which will be wrong) based on a moment in time that lacks context, cause, or professional interpretation. Yet to sell more product or to support bad educational policy, the declaration gets made that the results are more than they are, that they can directly inform teaching and learning, indicate quality or effectiveness, and replace professionalism.

This is as false and misleading and harmful as thinking that a diagnosis equates with a solution. All test results require interpretation through the broader technical lens of a professional equipped with the full context of the individual’s situation and current best practices. And they require the ability to question that lens, to recognize it as always incomplete and able to be improved upon. Only then is the professional capable of determining an optimal path forward for that student or patient while at the same time being responsible for making that path better for the next time.

I used to be kinder to the test publishing world—especially when I was in it and it was paying my bills and I still believed we were capable of staying within the limitations of what a test is—but the field has strayed way too far from its usefulness of putting tools in the hands of a researcher and instead has become something else altogether.

We would never tolerate straying so far from what a thing is in the tools that will help us through the pandemic because the consequences would be unthinkable. We shouldn’t tolerate it in the education of our nation’s children for the exact same reason.

Thursday, April 23, 2020

On Teacher Evaluation During a Crisis

The headline in my email this morning from Ed Week asked whether it was appropriate to do teacher evaluations in light of the Coronavirus. I wish they would ask the more honest question: is it appropriate to beat up on teachers during the Coronavirus, or should we give it a rest for a year?

If the evaluation systems were based on a true accountability, this question wouldn’t exist. The fact that it does, that accountability and teacher evaluation in schools are in fact being put on hold—means that we don’t have anything even close to an effective accountability or evaluation environment. I continue to argue that if it can be put it on hold you have to stop calling it accountability because it isn’t. I would argue the same for evaluations.

Education's myopic autopsy-based approach to everything inserts a punishment and punishment avoidance mentality into the process, not a how can we be great mindset that is at the heart of great organizations. As I study accountability in those organizations, they do things along the lines of the how can we be great mindset.

Here’s how they do that.
  1. They imagine themselves standing in front of a stakeholder at some point in the future. They ask, “what will I need to say at that moment to prove my effectiveness? To show that I’ve done something great and that my work matters?”
  2. They ask, “what would count as evidence of effectiveness or movement towards greatness?”
  3. They look at the current state of things and figure out what needs to be done so that at that future moment they can state that they have indeed been effective and done something great, with the evidence to show it.
  4. They get to work with a shared understanding of what effectiveness and greatness look like.
  5. When they hit snags—and they always will—they think about that accountability moment in the future and how best to get back on track.
  6. They seek help and support early, when it can make a difference.
  7. The goal in the organization is for every person to have highest evaluation marks possible, because that would mean the organization is highly effective and ready for whatever comes next.
It really is that simple.

In a crisis, nothing changes. In fact, a crisis is when this system is most effective. It is when we most need to develop a clear understanding of what greatness at some point in the not too distant future needs to look like, of what would pass for evidence of that greatness, and what needs to be done between now and then to make it happen.

Millions of educators have already answered that first question in this crazy new environment no teacher could have prepared for: what will greatness look like? They don’t have to ask if their work matters—it does.

They are at this very moment in the process of getting to that new definition of greatness. And as they hit snags—and they have hit a ton of them and aren’t even close the end of it—they don’t punt the moment of greatness down the road, but adjust, and figure out a way around them. And part of that figuring is seeking answers from others and asking for help when they need it.

They get—although they may not use these words—that evaluation should never be a gotcha at some point down the road, based on a day’s worth of test scores from last year, but rather, a summary of what I’m doing right now. Which means my days aren’t spent trying to avoid trouble in the future but figuring out ways to do great things.

If I were in the classroom, I would beg for someone to evaluate me in exactly this way. I would deserve it, because it would finally show the truth. It would show where I was effective during a terrible time, where I was challenged and needed to adjust, and whether or not I accomplished what I set out to do. All of that would be shared with my principal and teacher leaders who are now rooting for me, not staring over my shoulder trying to catch me at something, and it wouldn’t come as a surprise at the end of the process as to whether I had been effective or need to rethink things going forward.

And if that system can work well in a crisis, imagine what it could do when some sense of normalcy returns.

So, should we put the current teacher evaluation programs on hold during the Corona Virus? No. We should simply end them all together. In their place we should have an evaluation system based on the how can we be great mindset, which is how it works in effective organizations, rather than the punishment-avoidance nonsense we’ve had for years—that has never worked to make any organization better than it was.

We do that and maybe something good can come from this mess.

Thursday, April 16, 2020

A chance to rethink accountabilty

In this age of the Coronavirus and its overwhelming impact on literally everything, a bright spot in an otherwise ominous cloud is the way we are thinking differently about old problems, rethinking our relationships with each other, and reflecting on what is actually important.

We should do the same with educational accountability. And we have a window in which to do it.

Of all the problems to rethink, educational accountability should be at the top. For the past two decades (longer in some places) educational accountability has followed the "better autopsy" method for improvement, which will always fail. At the end of a school year the state performs an autopsy (and a partial one at that) and then forces schools to ask, "what could we have done last year to have had a better autopsy last year?" and then whatever the response do that this year.

The better autopsy accountability is nonsensical for lots of reasons, but none more so than it will force schools not to change with the times. It presumes that whatever conditions existed last year and the year before will continue (forget that the world is changing faster than we can ever imagine). It makes our our job in education to get kids ready for a world that does not yet exist by getting them ready in a world that hasn't existed for years. In other words, the closer we can align ourselves to a definition of things that was developed years ago but doesn't exist any more (if it ever did), the more likely we are to be declared successful in what is arguably a dumb system. And the more successful we are in that world, the less prepared our students will be for the one that is surely coming.

But it is also nonsensical because it isn't actually accountability. Accountability in effective organization is about the future. It is about ensuring that those in the organization are the right people to take it forward, or that the organization is prepared to do the work we need it to do. Accountability is about what we do in answer to the question, "will my child be safe in school today, and tomorrow, and the next day?" A business that substituted the better autopsy approach instead of actual accountability would, like schools have for years, find it difficult to change, impossible to adjust to new circumstances without tremendous amounts of energy better spent elsewhere, and in the meantime risk stagnating itself into oblivion.

The better autopsy mindset existed in education long before what passes for educational accountability put it on steroids--which helps explain why education looks surprisingly similar to what it looked like when I was in school in the 1970s. And now it's time to knock it off, and we have an opportunity to do just that.

We have some things planned over the next few months, so stay tuned. And if you're interested drop me a note at (new email--new organization will be announced shortly) and we'll get you on the list for announcements.

Wednesday, January 15, 2020

The gross misunderstanding in educational accountability

For a word used with ease in educational policy circles, accountability is a term that is surprisingly misunderstood and misused.

Seeing this is relatively simple. Ask an audience to brainstorm a list of terms they associate with accountability and a pattern will quickly emerge. Many of the words will be positive such as:
  • Transparency
  • Effectiveness
  • Responsibility
  • Outcomes
And many of the words and phrases will be negative, such as:
  • Feet to the fire
  • Testing due to lack of trust
  • Blame
  • Shame
If you list these words in two columns on a sheet of paper what you will be observing are the two sides to accountability.

The negative terms represent what happens when an organization refuses to be accountable and/or is perceived as failing. In that case, accountability is something imposed on that organization by outside stakeholders for the purpose of bringing the organization in line. Such an accountability focuses the organization on failure prevention at the expense of everything else.

The positive terms represent what happens in effective organizations. These are organizations that internalize the principles behind these terms and attempt to exemplify them in their efforts.

This type of accountability focuses the organization on how best to sustain itself long-term, and how best to communicate its effort to its stakeholders.

Both types of accountability are perfectly valid depending on the circumstance.

What should be clear is that the objective for any organization should be an accountability focused on long-term sustainable excellence. This properly aligns the organization with its long-term goals and the idea of continuous improvement.

What should also be clear is that imposing an accountability of failure prevention by stakeholders must be performed thoughtfully. Its intent is not long-term sustainable excellence, but just the opposite: an immediate, short-term failure correction. The intent of an imposed accountability is to focus the organization and its resources on correcting the failure at the earliest possible moment or the organization’s existence may well be at risk.

An imposed accountability’s purpose is thus temporary: to force an immediate correction after which the organization can turn its focus towards long-term sustainable excellence. When an organization is having its feet held to the fire its job is not long-term sustainable excellence but something else. The sooner it can correct its errors and turn its attention towards long-term sustainable excellence, the sooner it can return to a state of effectiveness.

It would be deeply illogical and harmful to any organization required to operate in the perpetual shadow of an imposed accountability when the goal is long term effectiveness. The reason for this is simple: it would make the formal focus of the organization failure prevention, and thus attempts at long-term effectiveness would be perceived as secondary.

Even if the organization’s leaders recognized they were in an illogical system and attempted to focus stakeholders on their long-term approach, the fact that the imposed accountability was at the behest of stakeholders while the long-term approach was not, means the imposed accountability is likely to triumph. At best this would cause any positive message to be diluted, and at worst ignored or not believed.

Getting the balance right is always a challenge as organizations consist of lots of moving parts and it will regularly be the case that some of those parts are deserving of an imposed accountability. So long as such accountabilities are temporary that part of the organization can correct itself and return to a focus on the long term the accountability system. In that case the overall accountability system will be seen as contributing to the overall well-being of the organization.

The objective must be for any organization to spend the majority of its existence in an accountability focused on long-term sustainable excellence, and as little time as possible under the pressure of an imposed accountability. Only then will it be in a position to deliver effectively for its stakeholders

Sunday, January 5, 2020

How standardized tests do what they do (which isn’t what most people think)

Standardized test is the name most people assign to the tests used in state accountability systems, commercially available norm-referenced tests, and college admittance tests such as the ACT and SAT. I have long encouraged folks to drop the term “standardized,” since that merely refers to the conditions under which tests can be administered, rather than what this narrow family of tests are and do.

Instead, I prefer to call them predictive tests. This describes what they are intended to do.

I have also strongly encouraged a more critical use of vocabulary regarding predictive testing. This is because of the massive confusion that results from the plethora of terms now applied to testing that don’t mean what most people think, such as standards-based, or criterion-referenced.

What sets a predictive test apart from all other forms of testing is its ability to produce predictive scores. Simply (and crudely) put, if I am slightly above average this year you can predict that I will probably be slightly above average next year. If I am not, if I am well-above or below average, you can note it and begin the search for causes. Perhaps there are lessons to be learned or perhaps not, but as a signal for where to look such test scores have some use.

Confusion is created when people presume that their names for testing, such as standards-based or criterion-referenced, are parallel forms of testing to a predictive test. This is inaccurate. If the tests produce consistent results across administrations, they are first and foremost predictive tests. You may have drawn the content from a state’s written standards and labeled it a standards-based test, or drawn a line in the sand and assigned it a label, in which case you created a criterion (as you have assigned a score meaning that is external to the test). Or you may have conducted a comparative study after the fact that allowed you to apply norms. Regardless, the style of tests in which you are operating is predictive.

And, by the way, creating this narrow sort of instrument requires real specialization and training, as the sorting function will only occur in a consistent fashion with test items that perform within a narrow set of statistical criteria, and that combine to create a specific effect. This is a far cry from a teacher building a test to understand the effectiveness of their teaching or whether students learned a lesson—that isn’t even in the same ballpark. The last thing a teacher should care about regarding learning is whether their items sort kids into a curve, while that concern is first and foremost in order for a predictive test to work.

The greatest mistake people make with a predictive test is to presume that the consistency in the results has more meaning than it does, when the fact is that the meaning is surprisingly limited.

The consistency is created by first finding average and then calculating how far from average each test taker is. Since averages are reasonably consistent over time, as is a student’s relationship to average, the results will be as well.

The usefulness in this is that a student’s position is predictive as described above, and movement can be explored for potential lessons. The resulting orderings are also useful in that they show broad patterns behind them, often regarding socioeconomics, gender, race, etc. As researchers identify these and policies and procedures are put in place, future parallel instruments can be used to understand the effectiveness of those policies and procedures by noting whether or not negative patterns dissipate.

A perfect ordering on an entire domain is simply not possible—that would result in a test that was thousands of items long. Instead, test makers locate a few items that will order students about the same as if the ordering were done on the entire domain. This makes the test a proxy for the domain, and still useful in spite of the fact that it is not a statistically representative sample of it. So long as the ordering on the limited selection of content will be roughly the same as on the entire body of content it is still useful in the hands of a thoughtful researcher who understands how the tested content was derived.

The fact that such tests are proxies for the larger domain adds another limitation to the scores: they are estimates only, with some amount of imprecision in each. That just means that while a majority of the time students taking similar tests on consecutive days will score similarly, some will not, and some will have scores that differ a great deal. Again, in the hands of a researcher who understands these limitations and that the scores are simply a broad signal for where to look for patterns and causes, these limitations don’t render the results useless. While they are limited, they can be useful so long as that use can tolerate the fact that scores are estimates based on a proxy and nothing more.

The primary confusion comes because the predictive test methodology produces reasonably consistent scores over time even though the test is based on a proxy for the entire domain. The resulting estimates (scores) are still sufficiently consistent over time to allow for researchers to find some value in them. But that doesn’t magically transform them into something they are not, opening up a world of uses beyond their design. Any use that assumed so would be silly.

Which is why the use of state test scores can rightfully be called silly. They are derived from the predictive test methodology yet are treated not as proxies, but as representative of an entire domain, worthy of teaching to and guiding learning when that cannot be the case. They are treated not as estimates useful for research, but absolutes to make judgements. And worst, they are treated as signals of quality when that was never in their design.

This last point has been particularly disastrous for schools that serve students from historically marginalized communities. It is a fact that if you order students as of a day on a domain such as literacy—whether via proxy or a more complete measure—and some aspect of society contributes heavily to students’ ability to acquire knowledge within that domain, the ordering will reflect that. But as of that moment no judgment is available to be made. Some set of students may be behind because of real failure in their efforts or those of the school, in which case remedies for failure should be available and applied. But they may just as well be behind due to a lack of opportunity. In that case a failure judgment and remedy would be wrong, even unethical, as it would be the wrong remedy.

Rather, a different remedy should be applied that addresses the issue of being behind as being behind, but not failure. Mislabeling the problem would be a huge mistake as it would create perceptions that may not be real, force actions that run counter to need, and justify historical biases. Even worse, labeling being behind as failure risks converting being behind to failure, in which case the current system of test-based accountability could be said to have been a contributing cause to the further suffering of those who can least afford it, to the detriment of our nation as a whole.

In short, every role educational policy asks predictive tests to play is outside and beyond their design, with a profound number of ill effects that come from their bad assumptions. Predictive tests cannot be used to judge quality or effectiveness, guide or drive instruction, or indicate the effectiveness of policy.

So, there you have it: predictive tests work by being predictive, but in order to be predictive they can’t be much else, and they certainly cannot be used as the primary tool in school accountability. The sooner we all realize that fact the better.