kitchen table math, the sequel: Challenge Index gives percent passing!

Thursday, June 11, 2009

Challenge Index gives percent passing!

For years, people have been hounding Jay Mathews about his Challenge Index, which measures percent of graduating seniors taking AP tests. Taking, not passing. The Challenge Index measures only percent of graduating seniors taking AP tests. Mathews set the Challenge Index up this way in order to persuade schools to allow disadvantaged students, who had a high likelihood of failing the test, into AP courses.

That tactic succeeded brilliantly, particularly in wealthy suburban schools like mine, which presumably adopted open enrollment sometime after the Challenge Index came into being. In the years that I've been reading the list, Irvington High School has typically ranked amongst the top 200 high schools in the country. Of course, since nobody reads the fine print, people assumed the Challenge Index was actually an Achievement Index: a ranking of how well students do on the test.

Last year Mathews finally bowed to reality:
The minute I saw that Coolidge High School in the District had given a startling 750 Advanced Placement tests last May, and that only 2 percent of those exams had received passing scores, I knew I was in trouble.


Some readers have argued that the Challenge Index is an invitation to unscrupulous principals to stuff students into AP and create a false aura of academic success, but I don't think that is possible. Teachers, parents and students have to accept the AP courses as valid, as many of them have at Coolidge. A principal who tried such a scheme without community support would soon be looking for another job.

Why I Changed the Challenge Index
Jay Mathews
Thursday, December 11, 2008; 9:45 AM

That second paragraph doesn't hold for affluent school districts, where you have winner-take-all placement mechanisms (scroll down) that produce an enormous pent-up demand for AP courses in college-bound students who have been tracked out of Honors. In that case, when a principal decides that AP courses will take all comers, the community assumes that pushy parents have pressured the school into letting unprepared students take AP courses they don't belong in.

When students fail the AP test, that is confirmation that pushy parents have had their way with the principal.

This year, the Challenge Index includes a new number for "Equity & Excellence": the percentage of all graduating seniors, including those who never got near an AP course, who had at least one score of 3 or above on at least one AP test sometime in high school. Irvington's Equity & Excellence figure is 58. Our ranking this year is #241. Meanwhile Hastings, two towns over, has 73% of graduating seniors passing at least one AP test in their high school career & a ranking of 353.

More than 90% of Irvington graduating seniors enroll in college; 58% pass one AP test in their high school years.

bonus factoid: Scarsdale has 60% of graduating seniors passing at least one AP test without offering any AP courses for the past two years. I hate to even imagine how much more tutoring those parents are paying for (scroll down).


SteveH said...

Maybe I'm not looking carefully enough, but what is the exact formula that includes the number of kids who pass the AP tests?

I think we should come up with our own formula at KTM. We could do it for K-8 schools too, but we don't have any sort of national test to use as a basis. State tests are all different and deal only with low end cutoff skills.

ChemProf said...

Just some info for the parents out there -- these numbers are worse than they look. While a 3 on the AP is nominally passing, many colleges won't give credit for anything but a 4 or 5 (and for a few, only a 5 gives credit). Even my moderately selective college won't accept 3's.

Some schools will take 3's, but generally, they aren't highly ranked.

SteveH said...

"While a 3 on the AP is nominally passing, ..."

Do you know what a '3' translates into as a raw percent correct score?

We also have the issue of help at home or tutoring. We need to reduce the Challenge Index by a factor based on the number of tutors in the town.

Barry Garelick said...

How about follow up to see how the students who passed the AP exams fare in the subsequent college course(s) their freshman year. I asked Jay Mathews about this and he said "Oh, they do fine". Really? Has he checked? How come he hasn't written a column about that?

ChemProf said...

"Do you know what a '3' translates into as a raw percent correct score?"

As far as I know, that isn't published. Theoretically, a 3 is equivalent to a C, but in our experience, there's a quantum leap between students who get a 3 versus a 4. Places like Harvard or Harvey Mudd College see the same kind of jump between a 4 and a 5, so only accept 5's.

I also have been told by the pre-med advisor that med schools won't accept AP scores at all for basic coursework like General Chemistry, Biology, and Physics (although they will take it for math, sometimes).

As for how students do in the subsequent course, I don't know that anyone has done any real research on that. I will say that students who decide to retake first semester Gen Chem even though they have a 4 or 5 on the AP do really well. Those who choose to go on to second semester typically struggle a little at the beginning, often because it has been a while since they reviewed their chemistry, but generally get at least B's in my course (and remember a 4 on the AP is only supposed to be equivalent to a B). However, that's just my anecdotal evidence. I do usually advise students to take Gen Chem or Gen Physics again, but to skip out of Gen Bio, English 1, or Calculus if they have the AP credit, but that has more to do with the way the courses build at my institution than a general recommendation.

SteveH said...

I know that I will encourage my son to take AP classes, but the goal won't be to get advanced credit in college. I would encourage him to take the courses again in college unless he had some first hand knowledge (as in ChemProf's comments) about the courses.

I'm more interested in how the AP classes or weighted GPA is used in college admissions. Has any college published (either officially or not) the formula they use for cutoff in college applications? I assume that this would be a formula that determines if an application gets into the "look closer" pile, or if it gets rejected without a further look. I want to look at the formula and see what kind of weightings they give the SAT, ACT, weighted GPA, unweighted GPA and class rank. Are there any other major variables they use? Some claim that class rank is more important than GPA, but do they then include a factor for the number of students in the school? Do they have any factor for the the quality of the high school? Perhaps they weight SAT or ACT scores so high that class rank is not all that important.

My son is headed to high school in another year and he might decide to play the class rank game. It gets posted every year and it's something that many kids focus on. A high class ranking might be good for the ego, but exactly how much does it help versus trying to get better SAT or ACT scores?

concernedCTparent said...

Steve, A is for Admissions by Michele Hernandez discusses the Academic Index.

Although the Ivy League schools spent many years denying they used any kind of formula, they in fact have been using a ranking formula since the 1950’s called the Academic Index, AI for short. Dr. Michele Hernandez, in her book A is for Admission, was the first to reveal this formula to the public.

You can play with various inputs on the Academic Indez Calculator:

I also suggest subscribing to the newsletter and checking out the website for some interesting articles:

concernedCTparent said...
This comment has been removed by the author.
SteveH said...

Thank you for the links!

Do they go into a detailed explanation of the formula in the book? I came across the formula on another site, but it was given as:

AI = SAT(verval+math)/10 + CRS

where the CRS is the "Converted Rank Score" based on the size of the school and your class rank.

However, the AI calculator on the Hernandez site asked for the top three SAT II scores. Is there a variation of the formula that takes these scores into account?

Many sites talk about the number of people who apply to some top colleges and what percent get accepted. What I want to know is how many get past the AI cutoff versus how many get accepted? Then, after you get past round one, how many get accepted based on their academic numbers only, assuming that they aren't weirdos? Of the remaining applicants, what is the percent who get accepted and how do they make that choice?

I think it was Harvard that said that recommendations by high school teachers are very important. How true is that? That seems to me to be an enormous (!) control issue in the relationship between students and their teachers. Does anyone have any comments about that? I think that many places want 3 recommendations. Do they expect them from teachers? Do they discount other recommendations?

I'm a newbie at this, but I want to get a head start. I want to be able to tell my son what is important and what isn't. Instead of going crazy trying to get his class rank up one or two positions, I want to show him numbers that quantify the process and perhaps get him to put his efforts elsewhere.

Catherine Johnson said...


Nobody takes 3s.

Although Matthew Tabor told me that some colleges will let you place out of lower level courses with a 3.

Catherine Johnson said...

Steve - the person to read is Michele Hernandez.

Also, get on her email list. I haven't been reading my emails from her (aack!) but the ones I did look at were great.

Catherine Johnson said...

oops - just noticed Concerned Parent already left the Michele Hernandez link.

SteveH said...

I've seen a version of the AI formula where you can split the SAT I part in half and use the contribution of the SAT II (average of the best three) for the other half.

The interesting tables are the ones associated with the CRS (Converted Rank Score). It's very nonlinear. The top score is 80 if you have a class rank of 1, but drops to 70 for a class rank of 10th (senior class of 400). These 10 points are equivalent to 100 SAT points. If your class rank is up near the top, you can get extra points for each position you pass.

The curve is shifted up or down depending on the number of students in the school. After the top 2.5% of the class, the CRS slope tapers off quite a bit. If you are below that point, you would have to work very, very hard to get more AI points from an improvement in class rank.

The maximum score you can get from the SAT part is 160 (1600 / 10) and the most you can get from the CRS part is 80 for a total of 240. From what I read somewhere, the average for Ivy league students is 220. At our high school, that would mean something like 750's for both SATs and a class rank int the top 10.

I haven't figured out what colleges do with the writing portion.

Jay Mathews said...

Good questions. It is hard to say how students who pass AP do when they retake the same course in college because so few of them do that. In a few cases, if it is their major and they did well enough to get credit for the AP, they go take the next level course. (Most others never take a course in that subject in college, like my liberal arts son who got a 5 on Calculus BC but never took math in college.) There is very little research on how they do when they take the second tier college course after getting credit for AP. I called 10 colleges asking about this. Harvard had data from two courses saying they did not do as well as students who had taken the college's intro course. Claremont McKenna had data from two courses and said they did better. The other 8 schools had no data. We do know conclusively however that students who passed AP tests in high school do significantly better in college, as measured by GPA and completion rates, than students who did not take AP tests.
The college board gives AP tests to college freshmen who have just completed the college intro course in that subject and say those who get 3s on the AP test averaged about a C-plus on their own college course's final exam. The majority of college students attend schools that do give credit for 3s---the community colleges and many of the big state schools. The selective schools that give no credit for 3s only educate about 10 percent of college students. I know I write a lot about selective college students, but I think it is important to keep in mind that they are not the least bit representative, and in this case a 3 can be a big deal for the majority of students.