Steve on high school rankings & non-linear optimization

Steve has left several comments on the U.S. News & Newsweek rankings of high schools. In this one he responds to the question I asked about whether a parent could base a decisions about where to live in the rankings:

"In other words, say you're a parent looking to move to a district with good schools. Could you base a decision in these rankings?"

"Base a decision"? No. Is it of no value? No, but it's all relative. Schools think it's important because it's good PR. Our high school has a reference to it on its home page, and it's only a silver medal!

Over the long run, schools can take advantage of the formula. Anytime you have an extremely important formula that condenses a lot of information down into a single number, it's open to gamesmanship. But, the more that schools play the game, the less useful is the formula. I've seen cases where it's a constant arms race between those who want a formula to reflect reality and those who want to beat the formula.

If a school pushes all students to one AP class or another whether or not they are properly prepared, then that might improve the score, but the education might not be better. Or, it could be a false or local optimum and they completely miss a much larger global optimum that uses a different approach.

Important formulas generally force a trend towards one particular solution. Uncertainties in the formula can hide other solutions that might offer much better results. Instead of reflecting reality, they drive reality.

Think of a formula that tries to represent a topographic map. You want to find the highest point on the map but all you can do is plug in your latitude and longitude into a black box that will give you a height. You keep doing this and try to search for the highest point. If your second point gives a lower height, you turn around and go in the other direction and check the height.

This is called non-linear optimization and I have many books that discuss solutions to this problem. If you know derivative information, you can search faster. If not, you can find slopes numerically.

The problem is that you might find a mountain peak, but it's the shortest peak of the mountain range. Another problem is that the black box might not represent reality very well. There might not be a mountain peak there at all.

Another, more subtle issue is that (due to uncertainty) the very highest peak location is no better than a location that is 10 percent lower. If you think of a long mountain range, it might be much easier to climb to the slightly lower end of the mountain range than it is to climb the other end where the absolute highest point is located. In other words, an easier optimum to a problem (within 10 percent) might be located far, far away.

Rather than push all high school kids onto at least one AP track in high school, the easier approach might be to fix the problems in K-6 before they get large. They won't find this solution if they are focused on a formula that uses only high school data.