help desk - How to Measure Anything

Does anyone know anything about this book?

How to Measure Anything: Finding the Value of Intangibles in Business by Douglas W. Hubbard

data-driven loops & noise

more from anonymous on data-driven instruction:

There are two kinds of data loops in play. The first, I'll call loop 1. The second I'll call loop 2.

Loop 1 is the "We've measured your school's performance and found it lacking" loop. This is aggregated data that has been massaged to produce your AYP (measure of Adequate Yearly Progress, Mass. MCAS). From this measure, schools are to produce a School Improvement Plan (SIP) which is basically goal setting sans concomitant resources to actually effect a change. Since the SIP is a dead end, that particular loop looks more like a croquet wicket. It's not a loop at all.

Loop 2 is the "Your last year's students failed MCAS" loop which is given to teachers about 5 months after the students in question have left your embrace. Usually this is aggregated also, at least in my school it was handed to us on a printout. I had granularity down to the question but not down to the student.

Somewhere in this measurement system is precise, standard by standard knowledge of each student's current (actually 5 month old) ability. We don't get that.

The real problem is that loop 1 should be used for "oh my, I think we need a remediation here" and it is used for "oh my, time to reset the goal posts." If loop 1 is not used to drive some kind of structural change, then anything you learn (and you can't learn much) in loop 2 becomes a 'nice to know' kind of thing but it doesn't fix Johnny's inability to add.

Data not acted upon is noise and old data is rancid.

Call me crazy, but I don't see the problem here.

This school should just tell all its struggling students to Seek Extra Help.

Then, when the struggling students don't Seek Extra Help, or do Seek Extra Help but Extra Help doesn't Help, they should tell the parents, "If your child doesn't come in for Extra Help, there's nothing I can do."^*

That's what my school does.

It works, too.


^* direct quote

more fun with numbers
data-driven instruction redux
data-driven loops & noise

"statistics hides problems"

Keeper Comment from Ken's post on ambiguity in teaching:

I've said in the past that many teachers think that the problem of education is defined by what walks into their classroom. As kids get older, it's very easy to blame the kids and external causes.

"As Engelmann suggests, let's save the excuse making until we clean up our instructional act."

Unfortunately, this is done by trying to bring more kids up to very low cut-off levels. I'll call this the guess-and-check approach to educational improvement.

This reminds me of students trying to fix a computer program that has many different internal errors. The errors interact and produce all sorts of odd results. There are no clear cause and effect relationships. Invariably, students try to fix the program by changing something and looking at the results. The program might be fixed in one area, but the fix might cause a problem in another location. This happens because they don't take the time to really understand what is going on in the code line by line. Things change, but they don't really know why.

A lot of research abhors individual anecdotes. It's almost a dirty word. However, by performing a detailed analysis of individual cases, one really understands what's going on. You see a direct connection between cause and effect. Fixing this problem won't fix all of your problems, but it is a necessary step in the process.

Statistics hides problems. It's a process of reducing large amounts of data (many errors) into a more manageable amount. In doing so, information (problems) can be lost or confused. You might think you know what's going on, but you don't. If you want to understand why some kids are successful and some are not, you have to analyze a lot of individual cases. You aren't looking for one error and one solution. You're looking for many.

Why do so many educators try to find the "one thing", like better teacher preparation, that will solve the problem?

Guess and check.

..............................

Statistics hides problems.

Ed calls this death by data.