kitchen table math, the sequel: Do You Only Get What You Measure

Monday, November 30, 2009

Do You Only Get What You Measure

I attended an interesting faculty meeting today. These are once a month affairs where we are tasked to work on a continuing project. The current project was for each grade level team to analyze our state test data and report out on any significant findings along with ideas on what we might do to mitigate any adverse findings.

Math was the first topic. Turns out (no surprise here, we've been doing this since the seas parted) number sense is a big deal. It's the highest percentage test item and also where we do the poorest.

When the elementary teachers made their presentations they described their efforts in this very area of arithmetical calculation, number sense stuff. Since I'm pretty frustrated with my kids' lack of ability in this domain, my ears were perked. They work really hard on this. I know this from personal observation. But, there was one omission.

Someone asked how many kids leave third grade knowing how to add. Crickets! It's not measured.

I posed a question about what is meant by 'knowing how to add'. Crickets! There is no criteria.

Then I was asked what I meant by asking that. My response had to do with objective measurement vs. subjective measurement. Crickets! Nobody has objective measures.

Here's what the consensus answer was (I'm paraphrasing). "We have a pretty good idea what most of our kids can do when they leave us." There you have the big omission. We have a completely fuzzy string of descriptors all wrapped up in one sentence; pretty good, most of, and can do. Not only is there no objective measurement taking place, there isn't even an awareness of what one should look like.

Suddenly the scales dropped from the eyes of this grasshopper. We're getting what we measure! Subjective measures lead to subjective results. We aren't asking for kids to know facts. We're asking kids to get answers and it's perfectly OK if they do this with fingers, toes, and mystical incantations to math gods. We even meet the, by God, state standards with this fuzz as they ask for no more.

I'm of the school that says if you can't measure it then it doesn't exist. Have we reached a point that it is culturally unacceptable to do anything that isn't fuzzy? Am I working in a measurement resistant culture? To me, with my background, uncovering fuzziness is just an indicator that I need to do more work to expunge the fuzz. To my colleagues it seems like fuzziness is not a clue, it's an objective.

Please, just disabuse me of this if you think I've just had too much coffee, but I think it would be really interesting to find out if there has ever been a correlation study in education to see if it's true that "You get what you measure." From this anecdote it sure seems to have some truth to it.


Redkudu said...

If you think it's bad in Math, try opening this discussion amongst a bunch of English teachers.

TerriW said...

I can't speak for the teaching profession, but talk to someone who has tried to lose weight before -- what's the difference in how you eat when you write down absolutely everything you eat vs. when you don't? What's the difference in your exercise routine when you log your workouts vs. when you don't? Etc, etc.

"We have a pretty good idea what most of our kids can do when they leave us."

Yeah, and I have a "pretty good idea" about what I ate today, except for all the stuff that "didn't count" -- like cleaning off my kids' plates, or the stuff I just plum forgot about.

RMD said...

If you want to measure learning, the folks using Precision Teaching are the furthest ahead
look at for supporting info and the precision teaching wiki

Allison said...

HOW DID YOU CHANGE THE FONT???? Is that a new blogger feature?

okay, moving on..

I phrase it as
"if you did't measure it, it didn't happen." which translates to "if you can't measure it, it can't happen"...

I just attened a math pd seminar even on "practical" Differentiated Instruction for grades K-8. The "practical" part was done by making EVERY SINGLE question offered as homework or test a question with "multiple right answers".

The questions were math-free. They were things like "here is a set of 4 objects; pick 3, and explain why those three go together and the 4th is left out." Which three didn't matter; the criteria for set-ness didn't matter (and didn't have to be mathematical AT ALL);

so 2, 5 ,6, 10: take out the 5 because 5 is spelled five, which has four letters, while the others are spelled with three letter words. 10 can be removed for having two digits; 2 can be removed because the others are in alphabetical order then; etc.

This allowed everyone to contribute a "right" answer, no matter what level of skill or ability they were at. So everyone could feel good about their ability to "do math".

Of course, practical DI such as this doesn't work if you have objective measures. But that would defeat the purpose of having a achieved diversity in this classroom. And that was the speaker's stated goal: to have everyone feel good about how their differences could "teach" us all to see something new in the math class.

farmwifetwo said...

I told the psychometrist last week that little boy (8) used a calculator for math. He doesn't understand he can memorize the facts like he can words.

"No problem. We're more interested in problem solving than actual calculation".... says it all right there, doesn't it???

My SIL (a teacher) is upset her son (Gr 3) is a phonetic speller even if she writes out the sentence properly and has him redo it. She asked his teacher about it. They only have to spell the words on the wall correctly... otherwise phonetic spelling is "ok".

She's fine with this.... HUH??? IMO their is no rhyme or reason to the spelling lists sooooooooo....???

Anonymous said...


There is no rhyme or reason to spelling lists anymore. I was told when my fifth grader came home with 5 words, one of them being "red", that they were now doing more "brain based" spelling.

Here at KTM, home of the Save Your Own club, I suggest your friend immediately start a homeschooling spelling curriculum. I wish I had done that, also, along with all the other things that I ended up doing.

The whole "problem solving" and "critical thinking" mantras now in the grade schools just give them cover for not teaching critical skills theyll need later. Why, the kids can just go look stuff up or grab a calculator. I hear this stuff from even good grade school teachers. It's like the classroom is for them and not the students.


SteveH said...

They are trying to teach math with anti-math.

"This allowed everyone to contribute a 'right' answer, no matter what level of skill or ability they were at."

I've seen these descriptions of differentiated instruction at our schools. The goal is to make the full-inclusion process appear to work. The goal is not some quantifiable amount of learning. When they talk about problem solving, they don't see it as something you can quantify. Since they unlink mastery of basic skills with problem solving, they don't believe in measuring them. How convenient. It all fits so neatly.

There is a definite process orientation rather than a goal orientation. That's why they love Everyday Math. They can go through the motions and the spiral will take care of the rest. When they finally hit the big filter in algebra, the kids are long gone and you can easily put the blame on them. Just look at the kids who succeed.

SteveH said...

"...uncovering fuzziness is just an indicator that I need to do more work to expunge the fuzz."

When I was an engineering student, I wondered why one needed to be so accurate with a calculation if we just apply some huge safety factor to the result. Apparently, many educators don't even try. It's even better to assume that it can't be measured, ... but they will know it when they see it. This is a good reason to work hard at class participation.

palisadesk said...

I'm glad someone mentioned Precision Teaching -- really, better termed "precision measurement," as it does not prescribe or refer to a particular curriculum or methodology, but rather a results-oriented measurement scheme to provide ongoing feedback to the teacher and learner on what is working (or not).

RMD cited, but there is a plethora of information there. However, three papers that would interest some KTMers and add to the understanding of measurement and mastery are these:

1. Johnson and Layng's seminal paper on generative instruction:
Breaking the Structuralist Barrier: Literacy and Numeracy with Fluency

2. Binder, Haughton and Bateman's paper :
Fluency:Achieving True Mastery in the Learning Process
3. Carl Binder's short piece on measurement here

I highly recommend all of the above.

Measurement is key. As SteveH said in another thread, measuring performance is a simple matter -- provided the specs are clearly spelled out. When what constitutes expected performance is woolly or undefined, it's another story. Schools alas often have such vague "expectations" that precise measurement is impossible.

One can, however, be precise in one's own teaching at least in some areas. Literacy and numeracy both lend themselves to a clear, hierarchical ordering of skills and competencies which *can* be measured, and which one can demonstrate that a child is mastering.

RMD said...

I'm glad you commented on precision measurement (precision teaching) .. . I always respect what you have to say.

I LOVE the Binder, Haughton, Bateman paper. (I don't know the first). It is a tremendous overview.

Once you get on the fluency bus, particularly for fact knowledge like mathematics, you never go back.

SteveH said...

My question is whether the disike of content, skills (and the concomitant specific measurement tests) is the chicken or the egg. Paul mentioned once, and I had the same feeling, that this is all some sort of philosophical and pedagogical convergence of ideas into one neat package.

In our town, however, it seems like full inclusion is the basic assumption that drives everything else. It allows social promotion - actually, it's changed into developmentally appropriate. They want to treat all kids as equals, but they can't deny that some kids are more able than others. They have to do whatever is necessary to show that this is not a fatal flaw in their basic assumption. Our schools are struggling to show that differentiated instruction can work. As Allison mentioned on another thread, the attempts can be quite pathetic and can only work if you redefine what education is all about.

Then our schools change to rubrics with the claim that it gives more information to the parent. What it does is add more subjectivity to the process. It changes hard skills and knowledge into fuzzy concepts of understanding.

RMD said...

"My question is whether the disike of content, skills (and the concomitant specific measurement tests) is the chicken or the egg."

Dislike is caused by demand to perform without the requisite skills enabling performance (i.e., we are asked to do what we can't do).

So you need to teach skills to kids in small bites, correct them along the way, reinforce new behaviors, and practice them until they are automatic (i.e., fluent).

farmwifetwo said...

"So you need to teach skills to kids in small bites, correct them along the way, reinforce new behaviors, and practice them until they are automatic (i.e., fluent)."

But they don't. If I went by what was on my eldest's report card 2yrs ago in Gr 3, I would have thought him Math fluent. Even the EQAO (standardized testing Ontario) said it was "OK".

We'd spent from Mar to June doing all the rote pgs in the Saxon Math Gr 3 book and when summer came worked on the problem pages.

"You have 3 pencils and get 2 more" (abrev). Number sentence = 5+0. Answer = 5.

That is what he wrote. There was a nasty meltdown for 20min when I told him that was wrong and tried to show him how to fix it.

Why?? B/c it had NEVER been corrected. NEVER, by the teacher. NEVER had he gotten feedback.

Now 2yrs later such meltdowns wouldn't occur... but he's still very frustrated when I have to correct what is improperly taught at school.

How many parents realize what their kids know or don't know... not many. A rubric nor a teacher is going to show this to a parent or they will gloss over it and say "it'll be reviewed again next year"... Plus if the parent has never learned math... how do they know it's wrong in the first place??

RMD said...

farmwifetwo said:"But they don't."

of course they don't. they have little reason to do so.

A parent's relationship with a school that won't do its job is like any bad relationship . . . you can either find another relationship (go to another school), take care of your end of the deal no matter what the other party does (afterschool), or go it alone (homeschool).

I doubt the school is going to change. There is just no reason for them to do so. (see the blog "" for the most complete coverage of this topic)

farmwifetwo said...

Only in the US can you find a variety of school's and the ability to switch boards with relative ease. Here you are SOL, you get your home school unless you can prove that the program you require cannot be provided at that school. Then and ONLY then, can you switch unless you are headed to the Catholic board.

In the City you actually have the option of private schooling but only for those that are NT. In the sticks such options don't exist.

Part of me wants to homeschool, part of me knows I have neither the inclination nor the time to do so full time. I still have to work even if it's here at home.. I still require the respite... The guilt from coming to that conclusion is huge... but it's the truth. So I afterschool instead.

I have contacted the Special Ed dept today... I really didn't want to and if I had the teacher I was suppose to have I wouldn't have had to... They are in all day meetings today... I'm hunting for another class and upset that it won't be an inclusion one, but I can't have him dumped with the EA and not included anyways... BUT, I'm not removing my son from the one he's in... Until I'm good and ready.

RMD said...

"Only in the US can you find a variety of school's and the ability to switch boards with relative ease."

Not true. The best charter schools have waiting lists. The others aren't necessarily any better.

"So I afterschool instead."

I'm sorry that you don't have many good options.

I have been at afterschooling for years now, and I am able to stay at home so I've had a ton of time to research this.

Here are some suggestions if you're interested:

ALEKS - very nice because it guides your child along; I still need to read the explanations with my oldest, but they break it up into very small chunks; LOVE the math tables part

Funnix - only for beginning reading

Michael Maloney's products - great for continuing reading (through grade 8)

I also like the classical curriculum materials (I can't remember the name, but if you ask, I'll look it up)

I really only have 1/2 hour a day of their attention, so I have to pack it in.

Also, check out precision teaching on the PT wiki. It's easy to do. Remember, it's more precision measurement, but it gives you goals.

I've found that, with patience on my part, afterschooling has been very rewarding for us. But I don't have the stress of working a job too, so it's hard for me to gauge.

Best of luck.

palisadesk said...

RMD has good suggestions. I thought I would provide the link to Michael Maloney's site:
Teach Your Children Well

I have the reading materials, which are very nicely organized and Michael provides support by email or phone to parents. I haven't used them personally (the presentation books are not suitable for teaching a group) but know many people who have been happy with them. He has a math CD that is also highly recommended.

For spelling, the best I have ever seen for parents or tutors to use is one from the UK entitled Apples and Pears which has many characteristics of the DI program Spelling Mastery (it is mastery-based, teaches morphemes explicitly, teaches the rules for combining word parts -- doubling, changing y to i, and so forth), but is much less expensive, has shorter lessons and more repetition, and more emphasis on proper writing and sentence dictation. It goes up to a third or fourth grade level. You can see samples and a placement test on the site.

Here's the PT Wiki

farmwifetwo said...

Printed it off. I've been adding in some of the homeschooling material (I've made a shelf for it... not 100% complete.. other is over on the Autism shelf) I've read over the years. I too have done a lot of research on schooling methods and for us Classical is the best fit.

I too have found you can do a lot in 30min - their attention limit. Right now in Gr 3 and 5 after school during the week is too much, but Sat mornings/school holidays/summer we keep at it. That little bit has paid off more than I ever thought it would.

Paul B said...

re: font

I write all of my posts in Microsoft OneNote. There you can easily select any font you want along with some formatting (which I tend to avoid because it sometimes comes over badly). Then I do a cut and paste into the post. By some sort of magic unknown to me, this results in HTML ready text to paste even though there is absolutely no clue in OneNote that there is an HTML component at work. Go figure!

Sometimes I have to spruce up the header because it puts in stuff that the blog burps up as incompatible. I just delete the unmentionables and it works fine.

Hope that helps.