Are We Jumping to Conclusions About Progress too Quickly?

This owl has noticed the annual buzz about schools' Academic Performance Index (API) and Adequate Yearly Progress (AYP) news. The newspapers are filled with numbers and the headlines are filled with drama. "Failure" is the message in one newspaper story. "Success" is the theme in another. This owl wonders how newspaper reporters and educators can be so sure of their conclusions. How exactly do these numbers reveal how much learning is occurring?

THE API VIEW SHOWS GOOD NEWS AND AYP IS BETTER THAN EXPECTED

The API results were startlingly good. Even gains for all categories of students should be cause for clinking of glasses. Gains of between 11 to 15 API points were evident. But look at the pace of schools crossing the magic line of 800. Compared with 2005-2006, elementary, middle, and high schools all show substantial gains.

School Type	2005-2006 API > 800	2008-2009 API > 800	Three Year Improvement
Elementary	35%	48%	+35%
Middle	24%	36%	+50%
High	14%	21%	+50%

The news on the federal accountability front was mixed. More schools made Adequate Yearly Progress (AYP) than expected. Even though the bar to clear for AYP increased by about one-third in English/language arts and math, the percentage of schools making AYP barely dropped: from 52 percent to 51 percent. (Interestingly, one percent more Title I schools made AYP this year than last: 43 percent made AYP in 2007-2008 and 44 percent made it in 2008-2009.)

Viewing the AYP news for elementary, middle, and high schools, the challenges for middle and high schools becomes more visible.

School Type	2008-2009 MET AYP	2009-2010 MET AYP
Elementary	58%	61%
Middle	33%	29%
High	49%	37%

Top of Page

THE PI NEWS IS PAINFUL

The Program Improvement news paints a more painful picture. The total number of schools in PI went from 2,254 in 2008-2009 to 2,796 in 2009-2010 (an increase of 24 percent). The number of schools in Year 5 of PI is now a frightening 1,077. Just over 38 percent of all schools in PI are indeed in Year 5, and some of those schools have been in that state for ten years. If you include the 334 schools in Year 4, about half of all schools in PI are at the end-game stage.

We now have 298 districts in PI, about one out of every three districts in the state. About 60 percent (174) are in Year 3, the end-game for districts. What will the state board be doing about this at its November meeting?

QUESTIONING WHO WE ARE MEASURING

This owl wonders whether measuring a school is itself a good idea. My question has two dimensions. First, isn't a school a changing collection of students and teachers? How much "noise" is in the data about schools, after all? Second, if what we're pursuing is an estimate of progress, wouldn't it be wiser to measure the effect of each teacher on each student over time? After all, it is teachers who districts hire.

First, consider the "noise" from student mobility alone. What if the student attrition rate or mobility rate is 15 percent? In addition to losing 33 percent of your students when your eighth graders graduate and losing 15 percent of your continuing two grade levels, you get a new batch of sixth graders. This owl did the arithmetic, and it reveals that only 57 percent of your students are continuously enrolled from one year to the next.

If this owl were to set out to measure progress, I'd want to be able to act on what I learned. I'd want to know the gain score from year to year, student by student. You may ask, "What is a gain score? Some owlish invention?" Nope. This owl's not that clever. Gain score is what results when you measure what your students know at the start of the year and what they know at the end of the year. You normalize those results for all students in the same grade level across your district. If you were in Tennessee, where this is the law of the land, you would normalize those results for the state. This owl should add that it would help if you had a vertically equated test that lets you measure gains from one grade level to the next fairly.

I'd also want to know the results teacher by teacher. If I knew which teachers were best in each subject, and which teachers were best with low, middle, or high performing kids, in my hypothetical school, I'd match teachers with the students and subject matters they'd excel at. That might get me in trouble with the union, but it would make me a star with the board, with leadership, with parents, and students.

Top of Page

WHEN TEACHERS OF THE YEAR GET LAY-OFF NOTICES, SOMETHING'S WRONG

It's bad for everyone when schools get both the credit and the blame they don't deserve. It's bad for teachers when they get neither credit nor blame for anything. Isn't it time, now that we're ten years into the accountability era, to wake up and fly right? Isn't it time to pay closer attention to who we are measuring, how we are measuring them, and for what purpose?

Let's define progress in a way that allows us to see how each student is advancing relative to his own track record. Let's get California's testing program into the modern era and equate the scoring system across grade levels.

Let's give credit to teachers who move their students more than a year's distance in a year's time. Let's learn at last which teachers are best with lagging students and which are best with leading students, and then match those teachers up with the students who they help the most. Let's find the fifth grade teachers who excel at math and give them a chance to shine all day long teaching only math.

Let's give principals and districts the freedom to decide which teachers stay and which get laid off when positions get cut. Superintendent Erich Kwek at South Whittier was lucky enough to have the California Elementary Teacher of the Year for 2008, Michael Allen Long, in his district teaching fourth and fifth grades. Still, Superintendent Kwek laid him off when he learned that his star teacher lacked enough years in service to avoid landing on the March 15 notice list. Only a happy summer surprise enabled Superintendent Kwek to bring his star teacher back.

We have the means to do all this. We have value-added assessment practitioners who can tease out measures of teacher effectiveness from the data as it is. We have trustees who are ready to push the right questions forward. We have districts like Poway USD that have taken assessment to a higher level and have startling gains to show for it. (Listen to our June 2009 teleconference with Poway USD's leadership on the School Wise Press Web site.)

To continue receiving this email, and to learn more about our other email offerings, please click here.

Curious to know more about this issue? Read more below.

Top of Page

REFERENCES

PowerPoint presentation by Deputy Superintendent John Collins and Ray Wilson of Poway USD for the School Wise Press teleconference of June 2009.

Video of Poway students participating in their own assessments. (Deputy Superintendent John Collins and Ray Wilson of Poway USD for the School Wise Press teleconference of June 2009.)

Jack O'Connell's press release of Sept. 15 about the accountability results: API, AYP, and PI.

Table One: Percentage of Schools At or Above Target of 800 on Growth API Scores, 2002-2009.

Table Seven: Percentage of All Schools and of Title I Schools Making AYP, 2008 and 2009.

"State, U.S. disagree on progress at some L.A. schools," - LA Times, Sept. 16, 2009.

Browse The Owl Newsletter Archive | Subscribe to "THE OWL" Newsletter

© Copyright 2009, Publishing 20/20. All rights reserved.