California’s troublingly flat result on the 2017 CAASPP English/language arts assessment is similar to the results that are being reported in twelve of the thirteen other Smarter Balanced Consortium states. This discovery, by retired psychometrician Doug McRae, casts our California problem in an entirely new light.
The question is no longer, “Why didn’t our students make progress in English/language Arts?” The question is now, “Why didn’t the CAASPP assessment capture the progress that everyone expected to see?” This expected progress of between 1 to 3 percent (according to Mr. McRae) was nowhere in sight. Instead of progress, what every state has reported in English/language Arts is either decline or the absence of improvement. California’s year-to-year change was effectively flat: 1/100th of a percent more students in grades 3 to 8 met the standard in 2017 than in 2016.
The story has percolated, at last, to national visibility. On January 4, the education news site, Real Clear Education, ran an opinion piece, co-authored by Mr. McRae and William Evers, under the headline, “Is the Smarter Balanced National Test Broken?” But Mr. McRae has been sharing news of his discovery since late September, when he sent a letter to EdSource, which he later shared with the State Board of Education. His news was not exactly welcome. Instead of being thanked for his public service, he was chastised by Keric Ashley, CDE co-deputy superintendent, for daring to question the reliability and validity of the assessment.
Supt. Steve Betando, Morgan Hill USD’s leader, shared McRae’s findings at a CSBA Conference presentation on December 1. That panel presentation (which also included Deputy Supt. Joshua Schultz of Napa CoE) is one I put together on the topic of “Mismeasurement of Districts’ Vital Signs.” The representation you see below of California students’ results on the English/language arts CAASPP test, was the centerpiece of Supt. Betando’s talk. Over sixty board members and superintendents attended.
Grade-level comparison of California students’ English/language arts results, 2016 to 2017
ACSA’s leaders have already registered their dismay. Wes Smith led a contingent to the November 2017 meeting of the State Board. More comments from ACSA are likely to be heard at the State Board meeting of January 18-19, when the ETS contract renewal comes up for a vote.
The stakes here are enormous. The accountability dashboard rests on a foundation of data that must be seen as rock-solid reliable. If there are problems in the CAASPP results and all other SBAC states’ assessments, the accountability system itself will face a credibility problem. Mr. McRae’s recommendation is that the SBAC consortium agree to an evaluation, that they open what have been closed doors, and share what they know of their problems fully. If the problems are, as Mr. McRae suspects, whether there are enough items, whether those items are properly scaled, and whether the items that replaced retired items are of equivalent quality and rigor, then an audit should turn up evidence that supports or negates that hunch.
In any case, it’s time for candor and clarity. Let’s take a breath, refrain from slaying the messenger, and appreciate the news that California’s test results appear to be as strange as those in other SBAC states, and start probing as Doug McRae recommends.
[Note that Doug McRae is a pro-bono advisor to School Wise Press and its K12 Measures Project.]
