Wednesday, August 8, 2018

The most bogus claim I've heard in months: that school grades are fair

Education Commissioner Morath in the great state of Texas is about to release grades for schools. His quote:

"The idea that the design of the system was meant to highlight both high levels of student achievement and high levels of educator impact makes this essentially the fairest system in the history of the state of Texas." (Article by Julie Chang in the Austin American Statesman, August 7, 2018--see it here--italics are mine.)

The claim in italics is bogus. And it is easy for anyone to see why.

Think of what it means to assess a student. You can do that by imagining the full range of assessment done to create understanding regarding a student or a school as a large sphere, with many layers to it. Trying to understand all the complexity to properly assess student or school needs and assign appropriate judgments is a constant, on-going thing. It requires trained teachers, lots of effort and energy, and proximity to the students being assessed.
Tests are, by design and definition, a focused, limited form of assessment. You can think of each test as focused on a small portion of the surface of the sphere. But that's it. No test exists that can provide a complete assessment, standardized tests aren't even designed go below the surface, and even all the tests in the world can't do the full job of assessment.

When doctors use tests to guide their assessment of a patient, they generally do lots of tests--why? Because tests frequently produce contradictory or inconclusive results. It is then the doctor's job as the chief assessor to interpret the results from the various tests to the benefit of their patient. It is the job of educators to do the same. Should either not do that their conclusions risk being dead wrong for either the patient or the student, with serious consequences either way.

Doctors understand the complete lack of validity in making a complex prognosis from a single test--it would be unethical to do so. Educators understand the complete lack of validity in extending the results from a single, narrow test to a broader judgment that ignores the more critical assessment sphere--and that it would also be unethical to do so.

It is imperative that if judgments about schools are going to be made they must address the entire assessment sphere and get to the level of understanding. That would be the definition of fairness. Any thing short of that is, by definition, unfair. Commissioner Morath, who uses a single test as if it can assess the whole of a student and a school, would be wrong to suggest that what he has done is fair or that his judgments are accurate. A simple understanding of assessment, as well as what a standardized test is, how they work, and the limits in their design, are the only things standing in the way of the Commissioner seeing this.


  1. Thank you for continuing to provide us the data and impetus necessary to combat this faulty thinking!

  2. So...we're going to start giving credit for emotional security as part of routine assessment of how well (or not) the system works. Just add it to the grades or create another component of the evaluation process. Sorry, but I want my educators to teach the subjects. I prefer that students learn the subjects. Their failure to learn is not necessarily based on the system of metrics. Either you know the material (or can easily recall it for the test) or they do not. How is grading the results of a test not indicative of how much they have learned, memorized, or can recall?

    1. A very common misunderstanding. A few things to consider:

      1. Tests come in lots of shapes and sizes and have designs specific to a purpose. For example, a test a teacher gives after teaching a unit test for what was to have been learned, exactly as you describe.
      2. Researchers use a very different kind of test. It is designed to rank order kids from the furthest below to the furthest above average. They do this to search for patterns they can then investigate,
      3. All kids can pass the teacher-developed test if they learned the material.
      4. Not all kids can ever be above any point on the test researchers use. That would be like everyone being above average, which is impossible. Thus no matter where you draw a line on such a test, not all kids can ever be above it.
      5. Teacher developed tests do not produce consistent results over time, because learning is always messy and hardly linear. However, the test design used by researchers does. That doesn’t make one test better or worse than the other—just different.
      6. For thirty years policy makers have used the words describing teacher-developed tests of learning, but put the tests designed for research into their systems. This is where the confusion you describe occurs—the words are right, but the tool is not.

      So, to answer your question, the student responses on a teacher-developed test designed to assess for learning provide lots of information about what was learned, but the results aren’t very consistent over time. The responses on a test designed for research sort kids in a consistent fashion, but then in turn can’t tell us much about what was learned since their design is limited to sorting.

      Hundreds of excellent articles and books on the topic if you’re interested. My book is reasonably well known but I love Dan Koretz’s recent offering, Measuring Up: What Educational Testing Really Tells Us.

      Thanks for the reply—

  3. These test do not test basic recall. The test ask the students to do multiple things at one time. The questions are not 36 x 75 = ___.

    Here is an example from the 4th Grade, 2017 math test. Remember the kids taking this test are 10.

    The coaches at Xavier Elementary School bought cases of sports drinks for a field day. They bought 76 cases of drinks. Each case contained 24 drinks. All the drinks were given out to students. Each student received 3 sports drinks. How many students received sports drinks?

    Adults have a developed prefrontal cortex, the part of the brain responsible for decision making, where as kids are still developing theirs. So while this question might seem easy to an adult, kids often have a tough time deciding how to even start working on this problem.

    This is just one question on one from one out of 17 STAAR Test given each year. Every one of them has question which require higher level thinking skills of evaluating, analyzing, and applying their knowledge to new situations on a day they are stressed about a test.

  4. Dead on here John Tanner... we need to stop killing our kids and our schools with testing (although there is a time and place for them) and start making the learning enjoyable. The only kids that enjoy waking up in the mornings to go to school (arguably rarely love) are the ones that test well and do “school” well. This needs to stop and not the purpose or the teaching and learning profession.

    I was a high school dropout and told I would amount to nothing my entire K12 career. If you were told you SUCKED at your job for 12 yrs how long would most people stay? Yet we expect 50% of our kids to enjoy they’re 12 year jobs when we continue to tell them mulitiple times a year they’re not good enough. Its BROKEN!

    If the commissioner or most policy makers were assessed in their job one way by professionals (educators that went you their schools) they surely would NOT pass the test. We can change this... but we have to work together and listen to our professional and most of all....our kids!

    PS: Excuse my spelling and grammar. I never grew up wanting to be a writer or English teacher. So I’m not! I do what I love and do “okay” ;)