question about value-added measurement

My copy of the Harvard Education Letter arrived with an excerpt from a new book on value-added measurement. I have a question about this passage:

Misconception 2: Value-added scores are inaccurate because they are based on poorly designed tests. Most standardized tests are indeed flawed, but this is not a problem created or worsened by value-added.
Seven Misconceptions About Value-Added Measures
by Douglas N. Harris
Harvard Education Letter - March|April 2011
p. 8

Is that correct?

As I understand it, New York's state tests can't be used for value-added purposes. The tests are shorter in some years, longer in others, and somehow don't correspond to a one-year measurement of learning. Or so we were told. Certainly they don't provide any sense of where a child might be within a year's worth of content. New York tests are scored 1 to 4, so if your child scores a middling 3, what does that mean on a scale of 10 months? Nobody knows.

I had been assuming that in order to use a standardized test as a value-added measurement, the tests had to be normed month-by-month as the Iowa Test of Basic Skills is normed:

Grade Equivalent (GE)
The grade equivalent is a number that describes a student's location on an achievement continuum. The continuum is a number line that describes the lowest level of knowledge or skill on one end (lowest numbers) and the highest level of development on the other end (highest numbers). The GE is a decimal number that describes performance in terms of grade level and months. For example, if a sixth-grade student obtains a GE of 8.4 on the Vocabulary test, his score is like the one a typical student finishing the fourth month of eighth grade would likely get on the Vocabulary test. The GE of a given raw score on any test indicates the grade level at which the typical student makes this raw score. The digits to the left of the decimal point represent the grade and those to the right represent the month within that grade.

When your child takes the ITBS from one year to the next, it's simple to see whether he's made a year's progress in a year's time. If, at the end of grade 3, he scored a 3.10 on computation (grade 3, month 10), he should score a 4.10 on computation at the end of 4th grade.

But how would you make that determination using the New York tests?

Or is there some other comparison you make from year to year?

Robert Pondiscio on curriculum vs value-added

When I think of the curriculum and teaching methods I was required to use in my classroom, the idea that my effectiveness might be dependent upon them makes me want to lie down with a damp wash cloth on my forehead. Manipulatives and discovery instead of basic arithmetic? Endlessly revising ”small moments” and teaching the writing process to 10-year olds instead of basic grammar? No time for even basic science, social studies because of district demands for ever larger math and literacy blocks? If it fails, it’s on me? Seriously?

Erin Johnson left a comment:

Robert, Why do you think that the LA Teachers Union (or the national unions) have not highlighted the issue of curricula?

I have recently been in contact with a LA teacher who was rated “more effective” in math by the LA Times. She states that her good rating was probably due to the fact that she “subversively” uses Saxon math instead of her district adopted program. Do ed reformers expect that teachers will subvert the curricula adoption process?

And here is Robert again:

I’m not sure curriculum reform is on anyone’s radar screen in a big way, including the unions. I used to regularly subvert…er…adapt my math curriculum to assure automaticity on basic functions. 5th graders counting on their fingers or multiplying with arrays is an offense to my sensibilities. I had less flexibility on ELA since there was lots of joint planning and execution involved. I’d go as far as saying my school’s ELA program (“It’s not a curriculum, Mr. Pondiscio, it’s a philosophy,” I can still hear the staff developer reminding me) is what turned me into a curriculum advocate.

Curriculum effects and value-added

how the unions lost the media, part 2

Union leader calls on L.A. teachers to boycott Times

part 1

LA Times Teacher Rankings

Marginal Revolution has a post about the recent LA Times teacher rankings here.

I'm interested to see what the fallout of this will be. Will other towns crunch the same data?

From Alex Tabarraok's commentary from the link:

I don't blame the unions for being up in arms and I feel for the teachers, for some of them this is going to be a shock and an embarrassment. We cannot simultaneously claim, however, that teachers are vitally important for the future of our children and also that their effectiveness should not be measured. As systems like this become more common students will benefit enormously and so will teachers.

They are already finding a few "Beloved Teacher Mrs. X"s who are ending up in the bottom 10%. But, we're told, that's okay, because they teach valuable "critical thinking" skills that are not measured on the tests.

LA Times: Excellent and terrible teaching found in the data

The LA Times is beginning another series of articles about LAUSD, this series based on access they've had to LAUSD's longitudinal test data. Prior articles focused on money, with the Times creating an easily readable database listing all LAUSD employees' salaries. This time, they are focused on teaching.

The first article's intro says "A Times analysis, using data largely ignored by LAUSD, looks at which educators help students learn, and which hold them back."

To accomplish this,

The Times obtained seven years of math and English test scores from the Los Angeles Unified School District and used the information to estimate the effectiveness of L.A. teachers — something the district could do but has not.

The Times used a statistical approach known as value-added analysis, which rates teachers based on their students' progress on standardized tests from year to year. Each student's performance is compared with his or her own in past years, which largely controls for outside influences often blamed for academic failure: poverty, prior learning and other factors.

The article profiles a couple of strong and weak teachers, and apparently, more articles are forthcoming that will do more profiling. It seems that after analyzing the data, the authors went to the classrooms of those in the top decile and bottom decile for student improvement to view the teachers in action.

Miguel Aguilar at Broadous Elementary School is one of the strongest. "On average, his students started the year in the 34th percentile in math compared with all other district fifth-graders. They finished in the 61st."

That's an impressive improvement. I wish I understood enough details of the underlying scoring to know how this relates in standard deviations. Is improving a student one standard deviation when they one below the mean as difficult as improving a student one standard deviation when they are at the mean? Certainly it's not the same effort to move a students from 1 standard dev away from the mean to 2. What assumptions can be made about equal difficulty in movement of scores measured in percentile?

The article repeats what we all know as well: you must raise the bar.

"On visits to the classrooms of more than 50 elementary school teachers in Los Angeles, Times reporters found that the most effective instructors differed widely in style and personality. Perhaps not surprisingly, they shared a tendency to be strict, maintain high standards and encourage critical thinking.

But the surest sign of a teacher's effectiveness was the engagement of his or her students — something that often was obvious from the expressions on their faces."

The article goes on to argue that their analysis shows that excellence in teaching and weakness in teaching matter a great deal.

"Among the findings:

• Highly effective teachers routinely propel students from below grade level to advanced in a single year. There is a substantial gap at year's end between students whose teachers were in the top 10% in effectiveness and the bottom 10%. The fortunate students ranked 17 percentile points higher in English and 25 points higher in math."

The LAT is creating a database for release "in the coming months." I can't wait. I wonder if it will shed light on the value of mediocre teaching?

It's great to see this article. With luck, it will propel other journalists to perform similar studies. Perhaps some enterprising ed bloggers can FOI this information for their district, and perform the same analysis.

student evaluations & college calculus at the USAFA

In primary and secondary education, measures of teacher quality are often based on contemporaneous student performance on standardized achievement tests. In the postsecondary environment, scores on student evaluations of professors are typically used to measure teaching quality. We possess unique data that allow us to measure relative student performance in mandatory follow‐on classes. We compare metrics that capture these three different notions of instructional quality and present evidence that professors who excel at promoting contemporaneous student achievement teach in ways that improve their student evaluations but harm the follow‐on achievement of their students in more advanced classes.

[snip]

[O]ur study uses a unique panel data set from the United States Air Force Academy (USAFA) in which students are randomly assigned to professors over a wide variety of standardized core courses. The random assignment of students to professors, along with a vast amount of data on both professors and students, allows us to examine how professor quality affects student achievement free from the usual problems of self-selection. Furthermore, performance in USAFA core courses is a consistent measure of student achievement because faculty members teaching the same course use an identical syllabus and give the same exams during a common testing period.5 Finally, USAFA students are required to take and are randomly assigned to numerous follow-on courses in mathematics, humanities, basic sciences, and engineering. Performance in these mandatory follow-on courses is arguably a more persistent measurement of student learning. Thus, a distinct advantage of our data is that even if a student has a particularly poor introductory course professor, he or she still is required to take the follow-on related curriculum.6

[snip]

Our findings show that introductory calculus professors significantly affect student achievement in both the contemporaneous course being taught and the follow-on related curriculum. However, these methodologies yield very different conclusions regarding which professors are measured as high quality, depending on the outcome of interest used. We find that less experienced and less qualified professors produce students who perform significantly better in the contemporaneous course being taught, whereas more experienced and highly qualified professors produce students who perform better in the follow-on related curriculum.

[snip]

Results show that there are statistically significant and sizable differences in student achievement across introductory course professors in both contemporaneous and follow-on course achievement. However, our results indicate that professors who excel at promoting contemporaneous student achievement, on average, harm the subsequent performance of their students in more advanced classes. Academic rank, teaching experience, and terminal degree status of professors are negatively correlated with contemporaneous value-added but positively correlated with follow-on course value-added. Hence, students of less experienced instructors who do not possess a doctorate perform significantly better in the contemporaneous course but perform worse in the follow-on related curriculum.

Student evaluations are positively correlated with contemporaneous professor value-added and negatively correlated with follow-on student achievement. That is, students appear to reward higher grades in the introductory course but punish professors who increase deep learning (introductory course professor value-added in follow-on courses). Since many U.S. colleges and universities use student evaluations as a measurement of teaching quality for academic promotion and tenure decisions, this latter finding draws into question the value and accuracy
of this practice.

These findings have broad implications for how students should be assessed and teacher quality measured. Similar to elementary and secondary school teachers, who often have advance knowledge of assessment content in high-stakes testing systems, all professors teaching a given course at USAFA have an advance copy of the exam before it is given. Hence, educators in both settings must choose how much time to allocate to tasks that have great value for raising current scores but may have little value for lasting knowledge. Using our various measures of quality to rank-order professors leads to profoundly different results. As an illustration, the introductory calculus professor in our sample who ranks dead last in deep learning ranks sixth and seventh best in student evaluations and contemporaneous value-added, respectively. These findings support recent research by Barlevy and Neal (2009), who propose an incentive pay scheme that links teacher compensation to the ranks of their students within appropriately defined comparison sets and requires that new assessments consisting of entirely new questions be given at each testing date. The use of new questions eliminates incentives for teachers to coach students concerning the answers to specific questions on previous assessments.

[snip]

Students at USAFA are high achievers, with average math and verbal Scholastic Aptitude Test (SAT) scores at the 88th and 85th percentiles of the nationwide SAT distribution.8 Students are drawn from each congressional district in the United States by a highly competitive process, ensuring geographic diversity. According to the National Center for Education Statistics, 14 percent of applicants were admitted to USAFA in 2007. Approximately 17 percent of the sample is female, 5 percent is black, 7 percent is Hispanic, and 6 percent is Asian. Twenty-six percent are recruited athletes, and 20 percent attended a military preparatory school. Seven percent of students at USAFA have a parent who graduated from a service academy and 17 percent have a parent who previously served in the military.

[snip]

These findings have broad implications for how students should be assessed and teacher quality measured. Similar to elementary and secondary school teachers, who often have advance knowledge of assessment content in high-stakes testing systems, all professors teaching a given course at USAFA have an advance copy of the exam before it is given. Hence, educators in both settings must choose how much time to allocate to tasks that have great value for raising current scores but may have little value for lasting knowledge.

Does Professor Quality Matter? Evidence from Random Assignment of Students to Professors
Scott E. Carrell
University of California, Davis and National Bureau of Economic Research
James E. West
U.S. Air Force Academy

=======

6 For example, students of particularly bad Calculus I instructors must still take Calculus II and six engineering courses, even if they decide to be a humanities major.

3 to 1

GROVER J. “RUSS” WHITEHURST: One of the points, Jane, that's made is that if we had linked data systems, we would be able to have a fair accounting of what works and what doesn't. My bias is it's a lot harder than just having the data systems. But you’re an expert on this.

JANE HANNAWAY: [W]e were focusing on in particular was value-added measures of teachers. Teachers, I think, are arguably, and there is evidence to show the most important in-school factor affecting student achievement, number one.

Number two, there’s huge variability across teachers and how productive they are, a three to one ratio in terms of teachers who are at the low end of the distribution and teachers at the high-end of the distribution.

[snip]

Determining which of those teachers are indeed the most productive ones is a difficult technical task that we're working on. I think the bottom line, where we are right now, is that we can estimate these differences fairly well, not perfectly.

FACING THE FUTURE: FINANCING PUBLIC SCHOOLS
Monday, December 1, 2008

Jaye Green on "no stats all star"

[Shane] Battier is what business guys call a “white space” employee. The term refers to the space between boxes on an organizational chart. A white space employee is someone who does whatever it takes to achieve organizational goals and makes the organization work much better as a whole.

As we move into the era of value-added analysis for teacher merit pay, this article provides much food for thought. School leaders must consider carefully what they will reward, and give some consideration to how white space behavior is rewarded. Rewards should not just be based on individual learning gains- reaching school wide goals should also be strongly rewarded. Otherwise my incentive as a math teacher will be to assign six hours of math homework a night- and to hell with everyone else (see Iverson, Allen).

the no stats all star

For my money, "to hell with everyone else" is a better explanation of Paul Attewell's findings on math-science tracking in winner-take-all high schools (pdf file) than Attewell's assumption that wealthy schools deliberately privilege 10% of their student population at the expense of the other 90%.

high SES = high scores

Several years ago, I received a phone call from a total stranger who was about to move into my school district and wanted me to help her identify good schools. She assumed that because of what I do for a living, I ought to know this...I explained some of the things that I had looked for when I had checked out schools and classrooms for my own children -- for example, a high level of student engagement, clear explanations from teachers before students undertook tasks, a level of enthusiastic activity when it was appropriate, and spirited discussion among the students. 

[snip]

She was not pleased. She clearly wanted an answer that was uncomplicated and that would entail less work.... [ed.: me, too. why ferreting out the one decent school in a very large district should be a full-time job is beyond me. This kind of thing is the problem, not the solution.]

[snip]

A few weeks later, I mentioned this conversation to a friend who at the time ran a large testing program. He replied that he received calls of that sort all the time and that few callers wanted his answers either. They wanted something simpler: the names of the schools with the highest test scores, which the callers considered enough to identify the best schools. He told me that in one conversation, he had finally lost his patience when the caller resisted a more reasonable explanation and had told her, "If all you want is high average test scores, tell your realtor that you want to buy into the highest-income neighborhood you can manage. That will buy you the highest average score you can afford."

Measuring Up: What Educational Testing Really Tells Us

by Daniel Koretz

p. 5-6

Why would anyone choose this image for the cover of a book? 

Just asking.

nominally high performing, part 3

I still haven't gotten around to posting excerpts from Richard Elmore's article on nominally high performing schools. I will, soon. 

A nominally high performing school is a school that looks good on paper thanks to high-SES students. But when you take value added measurements, you find that while the students are high performing, the school is not. Instead, it falls into a category William Sanders, inventor of value added measurement for schools, calls slide and glide:  

I've caught the most political heat from some of the schools in affluent areas, where we've exposed what I call "slide and glide." One of the top-dollar districts in the state had always bragged about its test scores, but our measurements showed that their average second-grader was in the 72nd percentile. By the time those children were sixth-graders, they were in the 44th percentile. Under our value-added scheme, the district was profiled in the bottom 10 percent of districts in state. 

Helping Teachers Raise Achievement: an interview with William Sanders 

Here is Ted Hershberg on nominally high performing schools:

The problems with AYP ["adequate yearly progress," the NCLB measurement of school quality] are clearly evident in ... schools whose students are meeting their AYP goals, but where little growth is occurring. Most often found in affluent communities where high-test scores go hand-in-hand with family income, these schools can be referred to as "slide and glide" because they appear to be resting on the laurels of their students. It is important to understand that NCLB does nothing to hold these schools accountable for providing their students with the annual growth to which they are entitled. In a global economy characterized by fierce competition for demanding jobs that pay high salaries and benefits, this is a highly significant shortcoming. 

"Value-added Assessment and Systemic Reform: A Response to America's Human Capital Development Challenge"

Ted Hershberg

Professor, Public Policy and History

Executive Director, Operations Public Education

University of Pennsylvania

In the past week the Times has carried two stories on P.S. 8, a "hot" school in Brooklyn that has just received a grade of "F" on its NYC report card. Interestingly, while the first story, starting with the title (In Brooklyn, Low Grades for a School of Successes), falls squarely within the tests don't measure quality genre, the second is the single hardest-hitting news story about an affluent and popular school I have ever seen, certainly in the New York Times.

From the first article: 

A respected Brooklyn Heights elementary school so popular in its gentrifying neighborhood that it has doubled enrollment since 2002 is set to get an F in the second year of the Bloomberg administration's heavily contested system of grading individual schools, renewing questions about the methodology behind the grades.

Mind you, that is the lede. The point of this story is not that P.S. 8's grade of "F" renews or perhaps raises questions about the quality of the school. 

No. 

P.S. 8's "F" can only renew and/or raise questions about the quality of the grading. Which was already questionable. 

The school, Public School 8, was once avoided by the well-off residents of neighboring brownstones but has been the paragon of a turnaround tale in recent years, leading Mayor Michael R. Bloomberg to declare in 2006 that if the rest of New York’s schools made similar strides, “the future of this city would be assured.”

This year, at a July 29 news conference announcing plans for an annex to accommodate the flood of students wanting to attend P.S. 8, a parade of public officials praised the school and its principal.

Schools Chancellor Joel I. Klein said, “You have built a very successful school here,” and Deputy Mayor Dennis M. Walcott added, “You have done a monumental job in both recruiting as well as maintaining a school where parents want to send their children.”

then comes the piling on:

When the first round of report cards was unveiled last fall, there were some counterintuitive results and many complaints, but now, P.S. 8 could be the most highly regarded and popular school to receive an F.

Last school year, 67 percent of its students passed standardized tests in English and 83 percent in math, and many who know the school said such a grade would be misleading and preposterous.

“It’s a real indictment of the grading system if it takes a school that is improving rapidly and is already doing pretty well and brands it with an F,” said City Councilman David Yassky, whose district includes P.S. 8.

from there the story moves into a terrific section on the tests and how they work: 

In New York City’s formula, the largest factor in determining a school’s grade, 60 percent (up from 55 percent last year), is based on how, say, this year’s fourth graders did compared to their third-grade scores, an analysis known as a “growth model.” Student performance — median scores and how many passed the test — counts for 25 percent, but is calculated not in absolute terms but by judging each school against schools citywide as well as against 40 other schools with similar demographics. In addition, 5 percent is based on attendance, and 10 percent on the surveys.

the principal speaks:

Last year, P.S. 8 received a C — which Mr. Phillips, the principal, described as disappointing in a letter to parents that took issue with the grading system and schools that emphasize test-taking over enrichment activities.

“It is my professional and personal opinion that this grade does not reflect our school and the work we do,” wrote Mr. Phillips, 45. “This grade will give us food for thought, but it will not change the way we move forward into the future or alter our core values.”

note: the principal is a graduate of "a prestigious fellowship program ... for outstanding principals at Columbia University’s Teachers College." 

the co-president of the PTSA weighs in:

Several P.S. 8 parents suggested that the F said more about the grading system than the school. They cited events like the annual read-a-thon fund-raiser, an art program that culminated in student work’s being showcased at the Guggenheim, and the school’s recent selection as Brooklyn’s Rising Star Public Elementary School for 2008 by Manhattan Media, a publisher of weekly newspapers.

“To me this kind of grading system seems inadequate to express the education that’s occurring within an entire school, and in fact becomes quite detrimental to the education process,” said Joanne Singleton, co-president of the Parent Teacher Association.

“When you walk into P.S. 8, the children are smiling, they are happy to be there,” she added. “They are comfortable in the school environment, they are comfortable interacting with their faculty, and it is truly a community of learning.”

If you were wondering why parents don't have a union, which you probably were not, this is it. 

So that was Thursday. 

Today's article appears to have been ghostwritten by Andrew Wolf:

How could a red-hot school in Brooklyn Heights — with surging enrollment from middle-class and wealthy families, with test scores that are above average, and with extras paid for by parents’ fund-raisers — be declared a failure?

For some people, news that Public School 8 on Hicks Street will be getting an F on its upcoming report card cinches the case that education officials have lost their test-taking marbles.

And yet there is a strong argument that the F grade is just the sort of blunt truth-telling needed for schools that are highly regarded in the vaporous, unchallenged esteem of conventional wisdom.

More than 80 percent of the kids at P.S. 8 passed a standardized math test. Two-thirds passed the language arts test. In 2006, the mayor said the school should be imitated. In July, the schools chancellor announced that an annex would be built to accommodate the demand in what he said was a “very successful school.”

In reality, children who start the year at P.S. 8 with decent or good scores in math and English actually have gone backward, said James S. Liebman, the chief accountability officer for the city’s Department of Education.

“You drop them off at the beginning of the year, and on average, by the end of the year, your child lost ground in proficiency,” Mr. Liebman said.

Children on the lower end of the scale — the ones who had the most room for improvement — made only the slightest gains compared with those at similar schools, Mr. Liebman said, while at most schools across the city, there were big improvements.

“Where was the child last year, and where is the child this year?” he asked. “You’re comparing them to themselves.”

For many people, the F grade for P.S. 8 ratifies their skepticism about standardized tests. If all the children, like those in Lake Wobegon, are above average, how could the school be failing?

decline at the top

“If you use high-stakes tests and nothing else, you’re measuring ZIP code, race, socioeconomic status,” Mr. Liebman said. “Most importantly, you need to measure how much kids improve after a year at their school.”

[snip]

On average, Mr. Liebman said, the higher-performing students at P.S. 8 lost a little more than one-tenth of 1 percent in proficiency in English and math; the lower performing students gained about a tenth of a percent. Why should anyone care about such numbers?

They make a major difference by the time the students are 18, Mr. Liebman said. Students get scores between 1 and 4. Of those who finish eighth grade with a 3.0 proficiency in math and English, just 55 percent graduate from high school four years later. For those with 3.5 scores, the graduation rate is 75 percent.

“I know it’s troubling to people in the neighborhood, and it should be troubling,” he said of the F grade. “The point is, compared to any other school in the city, this school is off the charts on the low end.

“We’re trying to move away from a school that gets by on its reputation.”

At P.S. 8, Image Didn't Match Performance
Jim Dwyer

This is amazing. I have no idea -- no guess, even -- how such a turnaround could have occurred. 

For the time being I'm going to assume that I need to be reading anything and everything Jim Dwyer writes. 

bonus points:

The kids at the top are showing the least progress. Perhaps enrichment isn't the best education to offer kids who are already being enriched at home? Which is not to say that I would wish the JASON Project on disadvantaged kids, either. 

Interesting that the principal was ordered not to speak until the scores were "officially released."

The 2nd story is not accompanied by a photograph of students sitting on the floor with nary a book, pencil, or desk in view.

I believe this is the first time I've seen a newspaper or magazine allow a testing expert to make the point that education is cumulative. Over time a small loss here, a missed opportunity there, the kind of thing routinely shrugged off by administrators in wealthy schools, add up. It's the miracle of compound interest in reverse. 

The author of the first story appears not to grasp the concept of value added: 

Also contributing to the F grade was P.S. 8’s rapid change in population. A quarter of the students now qualify for free lunch, compared with 98 percent in 2002, and more than half the students are white or Asian-American, up from 11 percent in 2002. Most of these changes are happening among the youngest children, before tests begin in the third grade.

Eighty-nine percent of last year’s prekindergarteners at P.S. 8 were white, for example, as were 60 percent of kindergarteners, 49 percent of first graders and 54 percent of second graders. The test-taking grades — 3, 4 and 5 — were 27 percent, 31 percent and 19 percent white, respectively. Throughout the country, there is an achievement gap between black and Hispanic students and their white and Asian counterparts.

As the demographics changed, test scores shot up, but most of the change came between 2002, when 32 percent of the students met state standards in English and 26 percent in math, and 2005. For the last three years, scores have improved more slowly, with about two-thirds passing in English, and 71 percent, 81 percent and 83 percent in math.

She seems here to be assuming that the rapid increase in test scores caused by an influx of high-SES students is the basis on which the school was graded in the past, or would have been. 

It is not.

What is value-added assessment?

value added assessment at Education Consumers Foundation

stagnation at the top, part 2

Here’s a simple principle: at the least, schools should be expected to help all students make a year’s worth of progress over the course of a year—even students that start school in September two or three grade levels above. And we should reserve our greatest praise (and perhaps rewards) for schools that accelerate the progress of all of their students and help each one reach his or her full potential. And that principle should apply to all of our children, regardless of the color of their skin, the size of their parents’ pocketbook, or their zip code. Anything else strikes me as unfair, unkind—and politically unsustainable.

Are public schools for all kids, or just some kids?

and, from the Heartland Institute:

Clowes: Is there any reason why students in schools with high concentrations of poverty should learn any less than students in an affluent district?

Sanders: Interestingly, I've caught the most political heat from some of the schools in affluent areas, where we've exposed what I call "slide and glide." One of the top-dollar districts in the state had always bragged about its test scores, but our measurements showed that their average second-grader was in the 72nd percentile. By the time those children were sixth-graders, they were in the 44th percentile. Under our value-added scheme, the district was profiled in the bottom 10 percent of districts in state. They were not happy. You'd think I had nuked the place.

With our value-added approach, we can demonstrate that our measure of school effectiveness is totally unrelated to traditional socioeconomic indicators. We have more than 1,300 elementary schools in this state; their effectiveness is totally unrelated to the racial composition of the school or the percentage of children in the federal free and reduced-price lunch program. That's looking at measures of progress, not at raw test scores.

You shouldn't hold teachers and principals of school districts accountable for things over which they have no control. You should hold them accountable for those things they do have control over. Schools and teachers don't have control over the achievement level when children walk in the door, but they do have control over how much that level is raised during the year.

If that is sustained over time, it becomes like compound interest, and what you see is populations of children constantly rising to higher and higher levels of achievement in later grades, regardless of where they started.

interview with William Sanders

Miss A and Moneyball

from palisadesk, an elementary school teacher in Washington D.C.:

Harvard Study on teacher effects

A brilliant piece of research and analysis appeared in the 1978 Harvard Educational Review. Authored by McGill University sociologist and administrator, Prof. Eigil Pedersen and colleagues, "A new perspective on the effects of first-grade teachers on children's subsequent adult status" is masterfully written and caps more than a decade of extremely sophisticated data collection and analysis. It attracted little mainstream media attention at the time.

Pedersen and his team were sociologists, looking for factors that made children from impoverished backgrounds successful in later life. Following in the wake of Coleman and Jencks, who both detailed the powerfully negative predictive value of low SES on academic achievement, Pedersen's insights and meticulous research were probably not given the attention they deserved, and their most powerful implications were missed.

Pedersen selected for his study a low-income, chronically low-performing school in an urban industrial area situated in the article "in a large city in the Northeast." It was, in fact, Royal Arthur School in the former Protestant School Board of Greater Montreal -- a mammoth brick edifice with windows barred like a jail, situated across the street from a brothel, and sandwiched between toxic industrial factories.

This school had by far the lowest levels of academic achievement of the 80 elementary schools in the school district at the time, and a high rate of discipline problems. Fewer than ten percent of pupils went on to graduate from high school; more than half dropped out before completing Grade 10.

Pedersen had access to pupil records covering a 25-year period, from about 1935 to 1960. Along with detailed achievement, attendance, and discipline records, the school also had scores of standardized IQ measures on every pupil, given in Grades 3 and 6. While tabulating other data (from which he gathered many useful and interesting conclusions, adverted to in the article), he noted a strange, and unexpected, phenomenon. There were a number of cases in which individual students' IQ scores changed dramatically, either up or down, in some cases by more than 2 standard deviations -- the difference between being "low average" and "gifted." This is a huge differential, and something that is not supposed to occur except under extraordinary circumstances. Pedersen's curiosity was tweaked. In addition to analyzing his other data about student performance, he decided to take a large set of pupils in 3 categories: IQ's that had risen substantially, IQ's that fell substantially, and IQ's that remained more or less constant, and examine these three cohorts in more detail.

Noting that the first measure of IQ was taken early in Grade 3, he wondered if students' achievement in Grade 1 was a predictor of their Grade 3 IQ. So that led him to examine and tabulate student data by Grade 1 class. There were three Grade 1 teachers who were on staff throughout the period when the pupil records were sampled. Pedersen called them Miss A, Miss B, and Miss C; others who came and went after varying short terms, he presented in aggregate form as "Others." Studying his matrix of data, Pedersen was astonished to observe consistent and striking differences between the three long-term teachers, compared to random variation apparent in the aggregate "Others."

Miss A had taught most of the students who showed a large IQ increase between Grades 3 and 6, and it didn't matter whether they were male or female. Miss C taught most of the students who showed a large decrease in IQ, and again, it didn't matter whether they were boys or girls. Miss B taught many female students who showed an increase in IQ, and males who showed a decrease. When they investigated further, the researchers found that Miss B had always evidenced a preference for girls, seating them in the front rows, holding them to a higher standard, and engaging them more frequently, while boys were banished to the back of the room.

This was all interesting, but not what Pedersen's study was about. He and his fellow-researchers went on to a more ambitious project, locating a large number of adults in their late 30's who were graduates of the school, and interviewing them and getting data about their current lives. He developed a complex dependent variable he entitled "adult status" which involved level of education, income, type and cost of home, employment status, etc. He got independent analysts to develop a classification system from 1 to 9 to represent "adult status." An example of 1 would be "on welfare, has never worked" and 9 would be "professional" (dentist, university professor, etc.). The bottom 3 were grouped as "low status," the middle 3 as "medium" and the top 3 as "high status." This team collaboratively assigned each participant an adult status ranking from 1 to 9.

So, Pedersen et al were looking at a number of variables, and studying data from a number of sources. However, when they had tabulated their findings, one things stood out so stunningly they had to stop what they were doing and change direction entirely. It was what they saw when they looked at "adult status" vs. "first grade teacher."

The mean adult status for the aggregate "Others" was 4.6 -- smack in the middle between 1 and 9, and exactly what , with random variation (and a large sample) would be predicted.For Miss C, it was 4.3. For Miss B., 4.8. All within the range expected. For Miss A it was a stunning 7.0 ! Even more remarkable was the percentage of pupils in the various groups. The other teachers averaged 29% "high status" adults and 40% "low status." For Miss A, an amazing 64% were in the "high status" category......and the percent in the "low status" category was --- ZERO.

The researchers went back to their data collection to rule out alternate hypotheses. They found that classes were not stacked in Miss A's favour, there were no significant differences between her assigned pupils and other class groups in terms of parents' occupations, families on welfare, behavior problems or other identifiable issues. All classes were heterogeneous. Miss A. had retired by the time Pedersen had reached this point in his study but he was able to interview former colleagues and pupils to identify factors in her success. She loved her work, she expected every student to learn and succeed, she gave extra help to kids who needed it, she was a low-key but firm disciplinarian, she showed affection to the children and had an amazing memory for their names and biographies -- remembering them in department-store chance encounters 25 years later.

But a major reason for her results is probably summed up by a former colleague who went on to be an administrator with the PSBGM: "It did not matter what background or abilities the beginning pupil had; there was no way that the pupil was not going to read by the end of Grade One."

SHE TOOK RESPONSIBILITY FOR STUDENT LEARNING.

If all, or even just 99.9% of, students learned to read in their year with Miss A, they were a long step ahead along the road to success.....and their IQ increase between Grs. 3 and 6 is likely due to the increased language aptitude engendered by their reading skill (kids who can't read consistently show an IQ decrease over the elementary years).

In her 34 years at this one high-poverty school, "Miss A" taught about 1200 students. While the researchers were unable to follow up all of these students, the cohorts they studied showed such consistent patterns that we can be sure she had an immeasurable effect on countless lives which could easily have gone nowhere, due to such unfavourable beginnings. The difference between the achievements of her students and the "average" of the other teachers' students is HUGE.

When I read and studied all of this, all I could think was, WOW.

Of course, it is an inspiring story. But also a cautionary, and disturbing, one. Why?

Because it was only because of Pedersen's work that anybody NOTICED what an extraordinarily effective teacher this person was, and how her effectiveness changed lives for decades to come (there's more about that in the article). In the "real world" of urban school systems, effectiveness is not only not recognized, it is rarely even noticed. But for Professor Pedersen's serendipitous research and meticulous data analysis, Miss A's. astounding effect on generations of low-income students would have gone unremarked, except by those students and their families. The "Miss As" among us today -- and they are out there, perhaps not in every school, but certainly not isolated phenomena -- fly similarly under the radar screen while initiatives to "improve schools" come and go with predictable regularity and minimal overall effect.

Not all education professionals would like Miss A. Not all share the view that it is the teacher's responsibility to TEACH every student, never mind what the student brings to the table. Successful learning does not depend upon intrinsic factors in the students (motivation, self-esteem, nourishing home environment, et al) OR in the teacher. It can be MADE TO HAPPEN. Michael Pressley's work on motivational behaviors of effective primary teachers clearly demonstrate this.

Miss A proved it, and her successors among us today continue to do likewise. Alas, I doubt the message her accomplishment sends -- that teachers can be far more effective than anyone dreamed -- will be taken to heart any more now than it was 28 years ago. If we collectively had a commitment to ALL students learning, what we do in schools would look a great deal different than it does.

A belated thank you, Miss A -- you put the lie to the persistently proffered argument that students' background, SES, IQ or previous experience is an insurmountable barrier to success in school and adult life. Perhaps some day we adults will be "developmentally ready" to learn from you. I hope so.

For the whole article, check interlibrary loan and look for Pedersen, E., Faucher, T. A., & Eaton, W. W. (1978). A new perspective on the effects of first-grade teachers on children's subsequent adult status. Harvard Educational Review, 48(1), 1-31.

* "Miss A" was in fact named Iole Appugliese (YO-lay AP-poo-lee-YAY-zee), and she was born in Montreal to Greek immigrant parents. Her pupils, who could not pronounce her name, called her "Miss Apple Daisy.") She retired only when she discovered she had terminal cancer. An interesting finding of Pedersen's was that more ex-pupils remembered being in her class than actually were. While many could not remember who their other early grade teachers were, every one remembered "Miss Apple Daisy" 's name and had memories and stories to share. Judith Harris hypothesized that part of the reason for her consistent student results was due to her creating a community culture of learning and achievement that carried her pupils through elementary school. By Grade 7, the largest predictor of student achievement (outweighing IQ, SES, family status, etc.) was which first grade teacher a pupil had had. Amazing stuff.

One alumnus Pedersen interviewed remarked that he had a five-year-old about to enter first grade. Sadly he sighed, "I only wish there was a Miss Apple Daisy for him."

Indeed.

I mention Moneyball because of a rondelay amongst Kevin Carey, Matthew Tabor, and Leo Casey over the question of whether teachers can or cannot be an undervalued "commodity" in the sense that hitters with high on-base percentage were before Bill James and the Oakland As demonstrated their true worth.

What value-added research is showing us – and no one seems to have absorbed this yet – is that a good teacher is vastly more valuable than anyone recognizes:

"The biggest factor affecting student achievement is teacher effectiveness," said Sanders, who emphasizes that class size effects and differences in ethnicity, family income, and urban-suburban location fade into insignificance when compared to teacher effects.
Excuses, Expectations, and Learning Gaps

All of us have known or heard of teachers like Miss A: teachers who change young people's lives.

But has any of us suspected that the Miss A's of this world were changing IQs? That the Miss A's in our schools were altering forever the future education, income, and life prospects of the children they taught?

Or that they were doing so for hundreds upon hundreds of children year in and year out?

The answer is no. None of us imagined or knew.

nominally high performing, part 2 ...

Mindless Math Mutterings posts the passage from Richard Elmore that has given me a formal term for the kind of affluent school district William Sanders calls slide and glide:

nominally high performing

One of the huge fallacies of performance-based accountability systems is the misconception that nominally low-performing schools don't know what they are doing and that nominally high-performing schools have something to teach them.

[snip]

Most high-performing schools in our highly segregated society have gotten there not by knowing a great deal about instructional practice or improvement but by getting and holding on to students in high socioeconomic groups. The practice in most nominally high-performing schools is emphatically not about improvement but about maintenance of a certain level of confidence with the surrounding community. When I speak about improvement with people in these schools, they often look at me as if it had nothing to do with them. Most of the knowledge about improvement is in the schools where improvement is occurring, and most of those schools are, by definition, schools with a history of low performance.

A Plea for Strong Practice
Richard Elmore

The funny thing is that inside a high-performing district everyone "knows" the schools are high-performing because of the families. The families know it; the school knows it. It's an open secret.

But while people know this, they don't ask what value the school is adding to what the students brought with them to school. Instead the schools is assumed to be high-performing because the students are high-performing (and student performance is never seen in comparison to private or international schools).

Can't remember whether I've mentioned our middle school principal's Back-to-School-Night skit performed in the fall of 2006 a few short weeks after he'd taken his position here.

It was a skit inspired by No Child Left Behind.

"I don't like No Child Left Behind," he said. "But it does have one good idea, which is evaluating the quality of your school. It's a good idea to evaluate the quality of your school. So I'm going to tell you the 4 characteristics that determine the quality of a school."

Off to the side were standing 4 middle school girls holding poster boards flat against their legs.

"The first characteristic in evaluating the quality of your school," the principal said, "is the school."

Girl number 1 turned her poster over revealing characteristic number 1: School.

"But if you're putting it all on the school," the principal continued, "that's too much."

In retrospect, that was the moment I should have started sending away for applications to Catholic schools. That I didn't see this at the time can probably be attributed to the fact that I was curious to discover what other characteristic one could use to evaluate the quality of one's school besides the quality of the school.

"The second characteristic you can use to evaluate the quality of your school," the principal said, "is the quality of the students."

The second girl from the left turned her poster board over.

Students

"The third characteristic is quality of the parents."

Poster board: Parents

"The fourth characteristic is quality of the community."

Community

"You have a quality community," the principal said. "I've been driving around your town admiring all the nice lawns with the beautiful grass. In my old school, one day I was walking a student home and there were some boys standing in the street and one of them said, 'Principal, do you want to buy some drugs?'"

And there we left it. Proof positive that we have a high-quality middle school because we have high-quality students, high-quality parents, and a high-quality community.

All of which it is possible to determine by the quality of our lawns.

nominally high-performing

In the TIMES today, NYC schools get their first value-added report cards:

The grades released yesterday contained many surprises, with some schools with top-notch reputations receiving B’s, C’s, D’s — and even F’s, to the astonishment of some parents.

That is because unlike traditional methods of judging schools, this one involves a complex calculation that assigns the most weight to how individual students improve in a year’s time on standardized state tests. It also compares schools with similar populations, as judged by demographics and incoming students’ test scores, and assigns final grades based on a curve. More than 60 percent of the schools earned A’s or B’s.

[snip]

Mr. Bloomberg said that the reports were devised to give parents crucial insight into their schools, and that if the grades upended longstanding school reputations, well, that was precisely the point. “We should be asking ourselves why some of the schools we thought were doing well aren’t serving students as effectively as other similar schools,” he said. Still, some parents lashed out at the enterprise, saying it overemphasized standardized tests.

“The way you treat our educators is part and parcel of the way you treat our students — constantly barraging them with narrow, deadening tests and demoralizing them with meaningless scores,” Jan Carr, whose son attends the Salk School of Science, a coveted Manhattan middle school that received a C, wrote in a letter to the chancellor.

[snip]

Several esteemed elementary schools that middle-class parents often factor in to their real estate decisions — including Public School 6 on the Upper East Side, P.S. 87 on the Upper West Side, P.S. 234 in TriBeCa and P.S. 321 in Park Slope, Brooklyn, — received B’s. Other popular schools fared worse. P.S. 154 in Windsor Terrace, Brooklyn, received a D, as did Central Park East I in Harlem.

Yes indeed.

P.S. 234 in TriBeCa.

Ground Zero in the NYC math wars.

A whole lotta tutors there at P.S. 234, and they still swing only a B.

The fact is in District 2 our kids are learning math like champs. Our scores are extraordinary,” said Daria Rigny, District 2 superintendent.

But the whispers in the schoolyards are that the high scores owe much to the widespread use of tutors.

“If you just studied TERC and took the test for Stuyvesant you wouldn’t be prepared. The only way is to be tutored,” said Jonathan Levine, the father of one current and two former P.S. 234 students. He says this makes two tracks, those who tutor and those who don’t.

One math tutor who works with many Downtown kids said most of her students are 4th and 5th graders whose parents worry that they will be unprepared for state tests and middle school. “They don’t even tell me what it is they want them to learn. They just say ‘could you teach them the math that I learned?”’ The tutor, who did not want to be identified, said some kids do struggle with TERC. “They try to show me how they solve problems in their classes and half the time they don’t understand it.”

I still don't see how value-added assessment adequately controls for tutors.



50 New York Schools Fail Under Rating System

Dear Sir

Jeff H said...

Dear Boeing,

We have been given the contract to design and build the engines for your new aircraft. I must tell you at the outset that we will require you to do all the design, testing, building and certification on your own. We are here to be the guide on the side, not the sage on the stage. We will allow your engineers to come to our facilities and see what an engine looks like but don't expect us to explain how it works, you must discover that for yourself. Imagine how much better your company will feel about itself if you discover how to build an engine instead of us building it for you. I can see that our partnership will greatly increase the self esteem of your company.

Thank you
General Electric

Dear General Electric,

Several of us here at Boeing were beginning to wonder just what value your multi-billion dollar contract would be adding to our project. Thank you for clearing that up for us. We look forward to a long and profitable relationship*.

Sincerely,
Boeing

* With Pratt & Whitney.

gut feelings

My copy of Gerd Gigerenzer's Gut Feelings arrived yesterday, and today I've found an interview with Gigerenzer in the TIMES science section.

Interesting passage on school quality:

In the 1990s, I was living in Chicago, where there are high dropout rates from the high schools. People often asked, “Is there a way to know which school has the lowest dropout rate?” There existed data measuring different cues of school performance: the pay of teachers, the number of English-speaking students in a class, things like that.

I wondered: could one feed these into a computer, analyze them and obtain a prediction on which high school produced the fewest dropouts? We did that. And we were astonished to find that computer-based versions of Franklin’s bookkeeping method — a program that weighed 18 different cues — proved less accurate than going with the rule of thumb of “get one good reason and ignore the rest of the information.”

Q: What was the “one good reason” that got you the right answer?

A: Knowing which school had high daily attendance rates. If two schools had the same attendance levels, you needed one more cue — good writing scores — and then you could ignore the rest.

Through Analysis, Gut Reaction Gains Credibility
by Claudia Dreifus

Interesting.

Writing instruction is probably the single most labor-intensive form of teaching, so..... high writing scores as a measure of school quality may in fact be a measure of teacher (and student?) effort.

Our high school apparently has fantastically high attendance rates, which the principal once offered as evidence that they were doing something right. I remember thinking that made sense.

Low writing scores, though. At least, the scores on the first year of the SAT writing section looked low to me. Ineffective writing instruction has been a perennial parent complaint.

I suspect that with the loss of the middle school math chair and the adoption of Trailblazers we'll see math scores decline, too, unless parents and students can find a way to make up lost ground in high school. I don't know much about the math teachers there, but the word on them ranges from "pretty good" to "very good." Still, I don't see how a high school teacher, no matter how good she is, is going to teach fractions, decimals, percents, and algebra 1, algebra 2, & geometry.

Hess & Finn recommend changes to NCLB changes

No Child Left Behind needs some Work

Have yet to read.... looks like they're recommending value-added measurements as one revision to the law.

glutton for punishment, part 2

It may be time for me to stop reading the Sun's op-ed page -- either that, or stop reading the Sun's op-ed page until Andrew Wolf gets back from wherever it is he's gone off to.

The Parents Job II
by Diane Ravitch

Several weeks ago, I wrote an article in the Sun titled "Don't Blame the Teachers." My basic argument was that proposals for merit pay, performance pay, and other incentive plans assume that teachers are slackers who need more pay to do their best. [ed.: or, alternatively, proposals for merit pay assume that teachers are normal human beings who respond to incentives like everyone else on the planet] I contended that in many classrooms, teachers confront students who have negative attitudes about academics, are not pushed to succeed by their parents, and are immersed in an anti-intellectual popular culture.

The responses I received to my article were fascinating. One prominent businessman wrote to complain that I was letting incompetent teachers off the hook. A friend who is a member of a New England state board of education said that she had met many lazy teachers. A parent advocate complained that I was blaming parents. [ed.: parent advocate? as in advocate for parents?] A prominent journalist who writes about race and poverty said that I was blaming poor kids for their poverty, even though my article was about all sorts of students, not just poor kids.

Not surprisingly, the letters I received from teachers thanked me for recognizing that what happens to their students outside the classroom and the attitudes they bring to class are even more important than what happens in their classes.

ding! ding! ding!

wrong:

You shouldn't hold teachers and principals of school districts accountable for things over which they have no control. You should hold them accountable for those things they do have control over. Schools and teachers don't have control over the achievement level when children walk in the door, but they do have control over how much that level is raised during the year.

and:

"The biggest factor affecting student achievement is teacher effectiveness," said Sanders, who emphasizes that class size effects and differences in ethnicity, family income, and urban-suburban location fade into insignificance when compared to teacher effects.

back to Ravitch:

A response from a high school science teacher in Brooklyn, David R., was unusually thoughtful and detailed. He pointed out that he has students in his classes from 70 different nations, speaking almost as many languages, most of whom are lower-middle class or in poverty. Many of his students regularly miss their first or second period classes because they have a two-hour trip to school or they are expected by their parents to baby-sit for a younger sibling in the morning or they just don't care about showing up.

This same teacher said that even when attendance is good, large numbers of his students have poor language skills.... Many were indifferent even when the teachers used videos or PowerPoint or other visuals to try to stir their interest. [ed.: they were indifferent to POWERPOINT?????]

But even among his native-born students, he said, apathy is rampant. Recently he warned a student that if she didn't improve her achievement, she was unlikely to pass her Regents' science exam. "She shrugged her shoulders. That shrug wasn't the shrug of one student: that was a collective shrug of many, many students." Despite field trips and projects and every trick known to the teachers' trade, many of the students "just don't care." [ed.: field trips, projects, AND POWERPOINT - and they still don't care????]

When the teacher reports a high failing rate, the principal naturally wants to know what he did wrong. What could he do differently? [ed.: clueless administrators couldn't be part of the problem, could they?] He knows what should be done for students to have higher levels of achievement. He knows the answers, but they are out of his hands.

Get the students to study instead of watching television or playing on their computers or hanging out with their friends. Get them to sleep at a reasonable hour. Get them to comprehend the connection between what they accomplish in school and their chance to have a decent income and life after school. Get them to see the value of visiting museums and libraries. Get them to spend free time improving themselves instead of sleeping late, partying, or going to the movies.

Almost everything that students need to do differently takes place at home. None of it costs an additional dime. It is all within the control and the choice of students and their parents.

I have an idea.

Maybe I should write an unusually thoughtful and detailed letter to Diane Ravitch.

I could tell her how I have a kid who studies, makes the connection between what he accomplishes at school and his chance to have a decent income and life after school, sees the value of visiting museums and libraries, and spends at least a little of his free time improving himself (he's taken to running with his dad & last week he read two books after being shamed into it by his friend M., who has read 5 this summer).

He does all of these "free" things that don't "cost an additional dime," and yet he still can't take 10% off a price and come up with the dollar amount he'll have to pay!

Maybe he's not getting enough PowerPoint.


what is to be done?

[E]ven the best teachers cannot be held responsible if students don't show up for school, don't understand English, don't see the point of learning, and care more about what's on television or what their friends are doing than what they do with the rest of their lives.

Even the best teachers can't compete with a powerful pop culture that makes heroes of dumb entertainers while failing to put before our young any heroes worthy of emulation.

....powerful pop culture....

....heroes of dumb entertainers...

....heroes worthy of emulation....

....our young....

Until we as a society begin to recognize that students and parents must take responsibility for the part of their lives under their direct control, we will continue to be dissatisfied with our schools. Scapegoating the teachers won't solve our problems.

hokey-dokey, then

Good advice!

I am going to take responsibility for the part of my life that is under my direct control.

.............................

So.....I'm wondering whether the part of my life that is under my direct control includes my turning-13 year old son.

value added comes to Westchester

The Cohoes city school district, outside Albany, is considering a gifted program for elementary students and adding college-level courses after discovering that its top students improved less on standardized tests in the past two years than everyone else in the district.

In Ardsley, N.Y., a Westchester County suburb, administrators intend to place more special education students in regular classes after seeing their standardized test scores rise in the last year.

And as the New York City Department of Education begins grading each public school A to F for the first time this fall, more than half the evaluation will be based on how individual students progress on standardized tests.

All three changes resulted from an increasingly popular way of analyzing test scores, called a “growth model” because it tracks the progress of students as they move from grade to grade rather than comparing, say, this year’s fourth graders with last year’s, the traditional approach.

Concerned that the traditional way amounted to an apples-to-oranges comparison, schools in more than two dozen states have turned to growth models. Now a movement is mounting to amend the federal No Child Left Behind Act, which is up for reauthorization this year, to allow such alternative assessments of student progress.

money 'graph:

[M]any school officials in affluent suburbs favor growth models because they evaluate students at all levels rather than focusing on lifting those at the bottom, thereby helping to justify instruction costs to parents and school boards at a time of shrinking budgets.

Maybe, maybe not.

Some of these schools are going to be slide and glide. I think the Scarsdale school system is a candidate - though I'm not sure how the tutoring there factors in. (may have to hit refresh a couple of times)

In New York, education officials are developing a statewide growth model that will be in place by the 2008-9 school year, to be used as an additional way to measure student learning. Fifteen New York school districts, mainly in the Albany and Catskill regions, have experimented with growth models on their own through a voluntary program started by two regional support educational agencies in 2005. The districts typically pay these agencies from $1,000 to $6,000 to train administrators and staff, and an additional $2.50 a year for each student for the data analysis, which is partly reimbursed through state aid.

“There is absolutely a need for this kind of data,” said Timothy G. Kremer, executive director of the New York State School Boards Association, which represents about 700 districts. “It keeps the focus on student achievement, and not on whether you’re going to pave the parking lot or who’s going to get hired as next year’s coach.”

[snip]

Even some supporters of growth models have expressed concerns that they could shift attention and resources away from the neediest students. Kati Haycock, director of the Education Trust, an advocacy group for disadvantaged children, said she was worried about growth models’ focusing too much attention on students at the top. “It risks so broadening the federal government’s involvement that its historical role will be dissipated,” she said.

Thanks, Kati.


money 'graph, part 2:

"When you look at achievement, every single wealthy suburb has high test scores,” noted Theodore Hershberg, a professor of public policy and history at the University of Pennsylvania. “That’s a terrible way to measure the performance of a school or an individual teacher because what you’re really looking at is family background or family income.”

In the high-performing Ardsley schools, where more than 87 percent of the students passed state reading tests this spring, district officials have long mined scores on their own, compiling a thick data book for review and coining the saying: “In God we trust, everybody else bring data.” [Didn't Margaret Spellings coin that phrase? I certainly remember her using it.]

But this year, they employed a more sophisticated growth model, which showed, for instance, that seventh-grade special education students had benefited from learning in regular classes. So this fall the district will expand the mainstreaming to the elementary and high schools. “This gives us the ability to measure whether a program has any teeth or is all fluff,” said Richard Maurer, the superintendent. ^*

wow

Ardsley is next door to Irvington.

Has a major autism program.

So.... naturally we moved to Irvington, not Ardsley.

On the other hand, Ardsley also has TERC.

So it's probably just as well we're here.

Which raises a question.

Value-added compares each child to himself. Does he achieve at least one year's growth each year in school, one year's growth as one year's growth is defined for him?

That's fine, but if a school has always had TERC isn't your child's typical "one year growth" going to be pretty poor?

A kid has TERC for 5 years, then every year thereafter the school will be basing "data-driven" decisions in the revelation that your kid has once again achieved one year's growth....

So I'm thinking value-added isn't going to end the math wars.



This is amazing:

Cohoes school officials have spent more than $1 million on programs for their most struggling students in the past five years, and wanted to find out how much they had progressed. They learned that the lowest-level students were doing fine, while their high achievers were starting to fall behind.

Charles S. Dedrick, superintendent of the 2,200-student district, said that parents had complained that their children were scoring too low on the Advanced Placement exams to receive college credit, but he thought there was just a problem with the A.P. coursework. Now, after examining over time the state test scores of students in advanced classes, he sees a more systemic problem. So the district has made top-level students a priority, too, and is considering starting a gifted program, expanding A.P. and college-level courses, and adding an International Baccalaureate program to keep them challenged. [I am hoping that in addition to "challenging" these kids the school will actually teach them. My experience of "challenge" to date has meant top-down instruction at school, direct instruction at home.]

“The fact is we serve all students, and not just the lower-end students,” said Mr. Dedrick, who travels across the state to speak about growth models to school superintendents. “If you’re just concentrating on one group of kids, it’s not fair because both sets of parents pay taxes.”

source:
Schools Move Toward Following Students' Yearly Progress on Tests

I see many signs that our high-end kids are underachieving. First of all, I know for a fact that C. has not read a single book even close to his reading level this year. [scroll down]

Beyond this, though, the fact that the district is dedicated to keeping kids out of accelerated and Honors courses, as opposed to recruiting more kids into these courses tells me kids aren't reaching their potential.

I also suspect - haven't taken the time to look at this closely - that we're in the same boat as Cohoes when it comes to AP scores. I could be wrong, but I don't think our kids are getting the scores they could and should be getting.

I don't think our average kids are doing well, either (though I don't know about high-end SPED kids. From what a couple of parents have told me, they're in better shape than the regular, non-SPED, average-achieving kids). Still, the rhetoric around here is Kati Haycock. Last winter the middle school principal [scroll down to end] told a large group of parents at a school board meeting that he was going to implement the middle school model because it would help the low-end kids. paraphrasing: "I only care about the lower kids, because they're the ones who are struggling."

A parent was told the same thing by another administrator: "We don't care about the high-achieving kids. They're fine." Words to that effect.

These are direct statements made to the parents of high-achieving kids. "I don't care about your kids" - this is the sentiment expressed openly and unapologetically as taxes climb each and every year.

Our taxes when we moved here 9 years ago were around $12,000/year; we're up to $22,000 this year.

And the middle school principal, in his first year on the job, feels perfectly comfortable telling a group of parents that he "doesn't care about the high kids."

In general there is a constant undercurrent - sometimes not so "under" - of hostility towards pushy parents who think their kids are smarter than they are. "Don't push your kids" -- this is the advice given openly to parents of high school kids in this $21,000/per pupil district.

"Everyone has a place."

When I told the math chair, back in 6th grade, that "Christopher needs to be able to take math in college" she said, "he needs to take math to graduate from high school." Subject closed. She had zero interest in my concern that he be prepared for college math.

And then there was the middle school principal's Back to School night skit on school quality.

He'd just come to Irvington from Albany, where he was principal at a school attended by disadvantaged kids.

"I don't like NCLB," he said, "but it has one good idea, which is evaluating your school."

How does one evaluate one's school?

"There are four criteria," he said.

"Number one is the school. You can evaluate the quality of your school by evaluating your school. But if you're putting everything on the school, that's too much."

The next 3 criteria were:

• quality of the students
• quality of the parents
• quality of the community

Irvington, he said, had high-quality students, high-quality parents, and a high-quality community (which he knew because he'd been driving around town admiring the nice lawns).

ergo: the school is high quality

This was the first time any of us had ever laid eyes on the man, and he performed a skit - he had 4 middle school girls holding up big posters with the 4 school-quality criteria written on them - about How to evaluate your school that said nothing about the school, the teachers, the curriculum, or the students' achievement.

slide and glide


update from concerned parent:

I was talking to a friend whose daughter will be going to the middle school. They are only allowing something like 3% of the 7th graders to take algebra. Everyone else will be taught using Connected Math Program.

If that's not institutionalizing underachievement, I don't really know what is.

I think it's time for me to read Jay Mathews' book Class Struggle. I have the book, and have read the first chapter, but it's time to sit down and concentrate.

Mathews wrote the book, as I understand it, because he had the compare and contrast experience of having written a book on Jaime Escalante and then moving to Scarsdale where there was a notable absence of Jaime Escalantes.

Cheri Pierson Yecke writes about a similar phenomenon in The War against Excellence, a terrific book every parent with a middle school-aged child should read.



The War Against Excellence
value added assessment PDK
value added assessment FAQ page
interview with William Sanders
A new way of judging how well schools are doing

Jay Mathews on the class struggle
Jay Mathews column on wealthy schools, AP courses, SAT scores
are wealthy schools worse?
value added comes to Westchester


^* Can we apply value added to character ed?

game on

Yesterday we were sent an email bearing the Village budget report by a member of the new "citizen's oversight committee."

Ed and I both suspect unauthorized use of an email list or two.

Here's the text:

please take particular note of the Police Budget on pages 12-15 as it is getting the most amount of press

<> I am sending this out as a citizen of Irvington--not as my role as a member of the CBC
my intent is not to advocate a viewpoint--but to disseminate this report --I know how busy people are and many will not access the link to retrieve

As far as the content--I believe the report speaks volumes and it is up to you THE TAXPAYERS OF IRVINGTON TO SEND A MESSAGE TO YOUR MAYOR AND BOARD OF TRUSTEES --IF YOU HAVE AN INTEREST IN YOUR TAXES AND HOW THEY CAN AFFECT YOUR QUALITY OF LIFE IN OUR VILLAGE

YOU NOW HAVE THE INFORMATION AND I HOPE THE INCLINATION TO COMMUNICATE YOUR VIEWS TO THE BOARD

HERE ARE THE EMAIL ADDRESSES TO VOICE YOUR OPINION to the mayor and board of trustees-I know many people cannot make the budget meetings in the evenings--

I haven't looked at the report (train wreck! train wreck!) but Ed did. Irvington, a village of roughly 6500 has, I believe, 77 full-time employees. Something like that. We have 27 policemen & women. Our crossing guards are policemen paid $80,000/yr; same deal with the people handing out parking tickets. The town is losing money on parking tickets, as a matter of fact, because we pay a policeman $80,000 to distribute them.

Village employees shovel snow and rake leaves, collecting overtime as needed. Which is particularly annoying to Ed and me, seeing as how we live on a "private road," meaning a road so private no one knows who owns it. Our road isn't maintained by the Village; it is called "the bumpy road" as a direct result. We pay for our own snowploughing & we rake our own leaves, etc. Sort of the same way parents in the school system pay for their own teachers.

There are a lot of private roads in Irvington, as it turns out -- and all of the folks who live on them are paying for town services privately.

People have been talking about the possibility of a tax revolt in Irvington.

After reading the budget report last night, Ed came upstairs and said, "It's already begun."

First the fields emails; now this one.



letter to the editor:

The local paper published this letter from me a day or two before the school board election & budget vote:

5-7-2007

To the editor:

The Irvington school district has the right mission statement – to create a challenging and supportive learning environment in which each student attains his or her highest potential for academic achievement – but too little sense of mission.

In math, the international standard for all students is algebra in the 8th grade. Yet here in Irvington, only 30% of students take algebra in 8th grade, and many of these students are being tutored or taught at home by parents.

Or compare IUFSD to the KIPP Academy, a charter school in the Bronx serving disadvantaged black and Hispanic children. Eighty percent of KIPP’s 8th graders take algebra and pass Regents Math A.

Per pupil spending at KIPP: roughly $10,000 per student

Per pupil spending at IUFSD: $21,000

One fourth of our eighth graders are invited to take Earth Science in the 8th grade; Pelham teaches Earth Science to all of its 8th grade students, including students in special ed.

The 7th grade ELA curriculum assigns books with measured reading levels of 5th and 6th grades; the social studies curriculum calls for students to make PowerPoint presentations; the Spanish class requires students to spend their weekends cutting out pictures of clothing from magazines and pasting them onto a piece of paper with labels in Spanish.

In K-5, Math TRAILBLAZERS takes more time to teach less content, and the district has deliberately slowed the progress of mathematically gifted children over the strenuous objections of their parents. Gifted children are to be “enriched,” not accelerated.

Enrichment means asking children to solve simple algebra problems without teaching them algebra.

Last but not least, I am still waiting for an explanation of last year’s bell curve exercise at the middle school. (Some) sixth grade teachers drew bell curves on their whiteboards, explained to students that they were average, and told them they should consider the Cs they would be receiving a good grade.

After the vote on May 15, Irvington’s school board will need to direct its immediate energies to getting a compromise fields project underway.

But I believe that, as work on “Plan B” proceeds, our elected representatives can also begin the process of holding administrators accountable for making our school mission statement a reality.

Catherine Johnson

From what I can see, the whole town - school & village - is a slide and glide operation.