kitchen table math, the sequel: Front page articles on the edtech bandwagon

Wednesday, October 12, 2011

Front page articles on the edtech bandwagon

Fast on the heels of a front page New York Times exposé on how education technology has failed to raise test scores comes a front page Education Week article on the virtues of replacing teacher-centered lessons at school with technology-centered lessons at home.  The technology in question is that of the Khan Academy, whose library of lectures and problem sets is impressive in its vastness but not in its instructional feedback. If you input a wrong answer to a math problem you are told that your answer is wrong, but not why, nor how to fix it. Despite this, the Khan Academy has empowered teachers like 10th grade biology teacher Susan Kramer to skip over direct, structured instruction, and instead to watch her students "weave through rows of desks, pretending to be proteins and picking up plastic-bead 'carbohydrates' and goofy 'phosphate' hats as they navigate their 'cell.'"

To be fair, the Khan Academy (1) hasn't been around that long and (2) is the creation of a former hedge fund manager with degrees in math, computer science, and engineering but not in, say, cognitive science and child development. As such, the Khan Academy hasn't profited from the "over 20 years of research into how students think and learn" that underpins more established educational software programs like Carnegie Mellon's Cognitive Tutor.

So it was a bit disconcerting to find, fast on the heels of the Edweek's Khan Academy article, a front page article in Sunday's New York Times on Cognitive Tutor, and how it, too, has turned out to have no statistically significant impact on test scores. While I'd never had a chance to try it out (unlike the Khan Academy, Cognitive Tutor gates access to demos and charges big bucks instead of nothing at all), I'd heard only good things about it, and J enjoyed soaring through its algebra lessons during middle school. But as soon as I read the Times' description of its pedagogy, its limitations became crystal clear:
When the screen says: “You are saving to buy a bicycle. You have $10, and each day you are able to save $2,” the student must convert the word problem into an algebraic expression. If he is stumped, he can click on the “Hint” button.

“Define a variable for the time from now,” the software advises. Still stumped? Click “Next Hint.”

“Use x to represent the time from now.” Aha. The student types “2x+10.”
A math buff would soar right through this; for anyone else, the hints seem way too much of a crutch. There's no mechanism here for ensuring that you're working things out to the best of your ability before resorting to "hint"---i.e., nothing to stop you from clicking "hint" the moment you're not sure what to do. And what if your answer is almost right: say you forgot to include the initial $10, or let x stand for hours rather than days? As far as I can tell (I've now tried it out a bit), you're either right or wrong, and that's it. The program simply isn't sophisticated enough to highlight exactly what needs adjustment. And there's a very simple reason for this. As I discovered in creating a software  program that highlights grammatical errors in English phrases and sentences, this kind of perspicuous feedback takes a huge amount of coding (of the sort that you don't find in any other language teaching software program, thank you very much). Programming in the analogous feedback for mathematical expressions and equations strikes me as even more prohibitive.

On closer inspection, therefore, Cognitive Tutor seems inevitably to foster--in all but the brightest, most motivated students (the ones most able to basically teach themselves)--far too passive of a learning environment for lasting learning. Indeed, the only truly active learning environment that I've ever seen in any software program for any academic subject is that which a computer programming language platform provides for--what else?--computer programming. Only here does the feedback--the error messages or the unexpected outputs--precisely reflect what you've done wrong.

Will these recent exposés about the limitations of educational technology for subjects other than computer science have any effect whatsoever on the edtech bandwagon?

We might as well ask whether recent cognitive science findings have had any effect on how schools teach "higher level thinking." Or whether mainstreaming kids on the autistic spectrum has had any effect on mandatory group work and personal reflections. Or whether parental concerns have had any effect on schools choosing Reform Math. Or whether, for that matter, the Pope is Jewish.

(Cross-posted at Out In Left Field).


FedUpMom said...

My objection is that the hint is very poorly worded. "Define a variable for the time from now"? I thought the question was easy until I read the hint.

For me, the variable represents the number of days of saving. That's a lot clearer than "time from now".

Grace said...

So much for technology being an important factor in improving differentiated instruction in all those inclusion classrooms.

Bonnie said...

I thought the mantra these days is that Khan Academy will single handedly permit us to educate vast hordes of students at rock bottom prices, allowing us to fire all those evil teachers. It isn't true? Oh no!

SteveH said...

Katharine talks about what needs to go into a computer program that could begin to be an effective tutor. But what is effective; just raising really bad test scores? Is the assumption that there is no possibility for having a homogeneous classroom led by a prepared and engaging teacher? Is the assumption that these tutor programs will be used during class time? Why wouldn't you just use these turor programs at home, but leave class time for teacher/student engagement and talking about issues not easily covered by a computer program?

My son's math textbook does a good job explaining the material and building, step-by-step from simple problems to more difficult ones. New material is covered in class by the teacher and some examples are given. Students ask questions. A homework set is assigned that moves from simple problems to more difficult ones. For each homework problem, the textbook points back to the pages that relate to the problem.

So what can a computer program do that's better? In theory, it can do a lot to enforce the development of basic skills. There's the rub. Educational pedagogy does not value that very highly in K-8. They want to use technology for other things.

It shouldn't be difficult to write programs that develop and ensure specific timed mastery of basic K-6 math skills that guarantee readiness for pre-algebra in 7th grade. But that's not what many educators want from technology. They devalue the link between skills and understanding.

Strategically, mastery programs might be popular if they allow teachers to avoid being directly involved with that onerous task. But what happens if they devalue any kind of homework or assume that kids don't do homework? Perhaps they can set aside part of the class time for individual use of the mastery programs.

In K-6, the bigger issues are low versus high expecations and specific learning (content and skills) versus vague "authentic" learning. It seem to me that the limiting factor is pedagogy, not technology.

Anonymous said...

"As I discovered in creating a software program that highlights grammatical errors in English phrases and sentences, this kind of perspicuous feedback takes a huge amount of coding (of the sort that you don't find in any other language teaching software program, thank you very much). Programming in the analogous feedback for mathematical expressions and equations strikes me as even more prohibitive."

I have been thinking about this for a few years now, and I think for word problems this problem is quite large, but ...

There are a finite number of *common* mistakes kids make when solving various 'pure' math problems (think, multi-digit multiplication, or solving for 'x' in an algebra equation, or adding mixed numbers). It should be feasible to nail down each of these mistakes for each problem type and then provide feedback about the mistake (you forgot to carry ...).

This isn't a full solution, but I'd be delighted with a partial solution. Let the teacher manage the stuff that can't be automated.

-Mark Roulo

Anonymous said...

There were programs to diagnose children's arithmetic errors and provide corrective instruction. It was a hot topic in AI in the late 1970s and early 1980s.

A lot of that knowledge and development was ignored by later software developers. Everyone wants to reinvent the wheel.

Grace said...

Bonnie, I haven't read about anyone who wants to "fire all those evil teachers". Here's Bill Gates:

Teachers will always be important. What Sal’s engaged in right now is working with teachers to see where they can use his material to relieve them of a lot of things. And his interactive parts can track and the teachers can see is this kid behind, is this kid ahead. It can help the teacher organize the class. So the power of a combination of a strong teacher together with these resources, I see that as making a huge difference.

Debbie Stier said...

Great post, and describes perfectly my personal experience with trying to learn through technology (which surprised me).

I started out thinking "of course education will be more efficient with technology" -- but after a few months in the trenches I started thinking to myself -- wait, is it me? Am I too old? Or is there some universal message that "its" not working.

Now that all of these articles are coming out saying the results aren't living up to the promise, I'm not surprised (and reminded again to trust my gut).

But this post really sums it up perfectly. Thanks Katherine.

Adam said...

I've been thinking about implementing hints while developing my math practice site for basic arithmetic, and it seems to me that young children doing basic problems would have a harder time understanding the meaning of the hints themselves than the actual problems/concepts.

So now I've been thinking about trying out subtle hints such that the child isn't even aware of the hint. For example, 5 + 3 = _. If the child inputs 2, it would re-display the question with the operator "+" font increased 5%. If they miss it again it would it would also change color slightly.

I'm hoping this would allow them to eventually figure it out on their own without making it too obvious.

Jen said...

Have you seen this yet?

It pulls in all the pieces you mention -- including the motivation piece.

At a school where I worked, they had "Apangea" -- and if you thought just like the people who were setting up the problems, you were fine. But, I had the few kids who were good at math able to do the problems given, but unable to input the pieces in the way the program demanded. Other kids without a clue would just randomly try putting in the numbers from the problem into the boxes.

Guess who often ended up "more successful" -- but without ANY learning happening. :-p

Cranberry said...

Grace, you haven't been reading closely enough. Of course they are looking to spend less on the teaching staff as a whole.

The program supplements traditional classroom lessons with online instruction. Its model allows it to reallocate about $500,000 annually per school, based on savings in staffing and class structure, and put that money toward higher teacher salaries, tutoring, mentoring teachers, and other areas.

In Rocketship’s schools, students receive instruction throughout the school day from certified teachers. But the school also carves out about 40 minutes of time each day for a “learning lab,” supplementary lessons focused on math or literacy skills, which are led by noncertified teachers. (...)

Rocketship Education saves money from not having to pay a certified teacher for the lab portion of the day, says Judith McGarry, the vice president for marketing and development for the organization, headquartered in Palo Alto, Calif. It also keeps costs low because the learning labs allow for relatively large class sizes, serving up to 40 children, she says.

"Savings on staffing," "relatively large class sizes" = fewer teachers. There's no way around that.

It's analogous to hospitals replacing RNs with low-skilled labor.

Anonymous said...

I now work for Carnegie Learning (not in the tech area, but still). Don't buy it. It sucks. The end.

The education industry is completely misguided in its ideology. Since 2003, the curve of stupid has been exponential. Fix that first. Then "technology" MIGHT provide an advantage.

--J.D. Fisher

Katharine Beals said...

Thanks for all these comments, guys! Much to ponder.

SteveH is absolutely right that computers could serve in other, non-tutoring roles: timed math drills is the perfect example.

And it's interesting to think about how much of a partial tutoring role computers could offer if you could predict error type from specific wrong answers, as Mark Roulo and Adam note. This would seem most practical in cases where there's a close-to one-to-one correspondence between particular wrong answers and error types (multi-step problems might pose a problem here) and where common mistakes can be predicted automatically from particular problems (so that a human doesn't have to determine what they are and code them in by hand for each problem). I particularly like Adam's idea of giving feedback by incrementally highlighting specific components of the problem: that is precisely the strategy used by certain remedial (ABA-based) software programs for kids with autism!

GWP, do the programs you allude to that "diagnose children's arithmetic errors and provide corrective instruction" still exist? Are demos available to the public?

Jen, thanks for the link to that very compelling article, which I hadn't seen. Interesting observation about how the feedback works only for those on the same wavelength as the developers.

J.D. Fisher, what happened in 2003???

Grace said...

Of course they are looking to spend less on the teaching staff as a whole.

But that's not the same as wanting to "fire all those evil teachers", unless you equate "ineffective" with "evil".

The objective is to increase productivity, with the objective of higher salaries for the best teachers.

Anonymous said...


I should start by saying that I shouldn't be recommending that people not buy my company's product. I'm on the sausage-making side of stuff. It might be delicious sausage for some. I've never seen it implemented in a real classroom.

As far as 2003, you know, I can't quite put my finger on exactly what happened then. A group of like-minded editors and I were discussing this some days ago, and, for the most part, we all agreed on this date, but although there were theories, no one was really sure why. To be clear about what we editors were talking about, though, it was that explanation simply evaporated from educators' thinking about pedagogy.

The "no telling" ideology has always been with us in education. But from my perspective, prior to 2001-2003, it had been contained somewhat by a kind of silent agreement that explanation should be at the heart of instruction no matter what. That is simply not the case anymore.


Adam said...

Thanks for the additional info Katharine. Good to know this technique is being used elsewhere. Incidentally, I use a similar technique for time telling where I increase the font size of the minute and/or hour hand number, and then change the shade of the hand itself, and observed my daughter doing the practice drills. It took her about a week to realize what was going on with the hints.

Catherine Johnson said...

So what can a computer program do that's better? In theory, it can do a lot to enforce the development of basic skills.

Haven't read the comments yet -- but this is something I ponder. I mentioned in the post that I just put uput up that I took all of the ALEKS geometry course and quite a lot of the Algebra 1 course --- and I can't say that it was particularly valuable. I've also self-taught from Saxon Math textbooks; that was better.

At this point I simply don't see a software program serving as the primary 'text.'

The ALEKS experience was way too 'chopped up' and disjointed -- and I say this as a person who was self-teaching out of Saxon textbooks, which are deliberately chopped up.

Catherine Johnson said...

I thought the mantra these days is that Khan Academy will single handedly permit us to educate vast hordes of students at rock bottom prices, allowing us to fire all those evil teachers.

Again -- I haven't read the thread, but I've been planning a post on this very subject.

Education technology is supported by education reformers who want to reduce -- or at least control spending -- on public schools.

It is also supported by unions (e.g. see the 21st century learning web site).

There's a new Education Week article that probably captures the worldview of technology promoters, and I'll get it posted.

What I think has happened is that the twin strands of differentiated instruction and the ban on homogeneous grouping have merged with the century-old belief that educational technology will transform school and learning. Programs like ALEKS hold out the dream that each child actually can be "differentiated" to -- AND that this can occur outside the school and inside the child's home (relieving the teacher of the fantastic burden of cooking up 20 different curricula and 20 different assignments for 20 different students in one class) -- AND that we can now have super-tiny class sizes because students will spend the entire school day collaborating in hands-on learning, which is noisy and can't be done with too many bodies in the room.

Basically, TECHNOLOGY as it is currently configured (i.e. Khan Academy and the "flipped" classroom) offers ed schools the hope that their dream of pure hands-on learning in heterogeneous and tiny classrooms might actually come true at last -- and with good test scores to boot, 'cuz the software will see to that.

Meanwhile your random libertarian supporter of tax credits and vouchers and the like is fantasizing that the Khan Academy is going to be so "disruptive" that we'll finally have choice (or maybe just not have far-above-inflation increases in public school spending forever and ever).

Catherine Johnson said...

When I first read about Khan, I liked the guy immediately (I'm a sucker for this stuff, too, and I appreciate all his work, of course).

I also thought, "Gee, this is kind of insulting to public schools."

Then I realized public schools loved the guy and were eating his stuff up. The "technology coordinator" in my district has already attended some Khan workshop or other, and some of the administrators appear to be drooling over the prospect of sitting little kids in front of computers (when they aren't sitting in front of their Smart Boards. Hands-on!)

Obviously, Khan had struck a chord with the public education establishment.

When that happens, you have to assume there's a reason ---