kitchen table math, the sequel: "It doesn't matter how well or poorly you teach"

Monday, February 25, 2008

"It doesn't matter how well or poorly you teach"

When I was a grad student at Cal, I TA'd the first course in the CS sequence one semester (and later I taught it once in the summer session.) This was a several hundred person course, and there were numerous TAs for it. I'd TA'd several courses at other schools, as a grad and ugrad, but I'd always been the only TA, and I'd worked relatively closely with the prof. This was the first time I was one of 6, and I was shocked that the professor did not feel the need to school us or provide any guidance on what we did in section. Basically, he said, it didn't matter. No matter how well or poorly you teach, your students' grades will not be any better or worse than the other sections', he said. I was a bit taken aback by this. I loved being a TA, and I worked hard to prepare for my sections. But, in the end it was true: my class average was within a standard deviation of the mean, as was every other TA.

I heard it again from another lecturer as well, and this lecturer was famous as the best teacher in the department (he was only a lecturer, not a researcher.) He was beloved for teaching this course, and he too said it didn't matter. Actually he even said "I've straight scaled the class because it doesn't matter. The grades will fit a bell curve. They always do, have for the last 20 years."

Bracket for the moment whether these teachers have given up and are jaded, or have hit a wall, or simply have found some universal truth.

Assume for now that this attitude is the predominant one among faculty in college, at least among those who teach several hunded person courses. Assume it's the correct attitude once you average over all of the various motivations for students in such a class: this is a subject they love, this is required for the major but NOT IN their major, this fulfills a breadth requirement, they have to pass it to graduate, they excel at school, they are taking this because of parental expectations, they have no background, they've seen the material before, they go to college to carouse and do drugs, that they are full time students, they go to school on the side of full time work, etc. etc. etc.

But now do the backwards regression:

At what point is it no longer true? At what point DID TEACHING MATTER? If the college profs say it doesn't matter by the time the 17-22 year olds enter, do they think it matters for high school? Has this idea trickled down to middle school? Grammar school?

Is the in vogue idea of "student maturity" just another way of saying that middle school teachers think EXACTLY the same way that college profs of adults think about their law-of-large-numbers-averages across hundreds of students?

Is this taught in ed school? Is this such a prevalent idea in college that ed school can't undo it anyway? Because if you believe this, then WHAT IS THE POINT of going into teaching?

15 comments:

SteveH said...

"...the first course in the CS sequence one semester ..."

I assume CS means computer science. If so, then the first big course in computer programming is a killer. If it is done well, your grade is based almost completely on your ability to get the programs done correctly. This is almost entirely based on huge amounts of individual effort. No teacher support will make much difference. Students either have it or they don't. This isn't true for other courses.

Forty-two said...

I'm assuming that CS=computer science?

If that is the case, it might be specific to programming. I read a paper online a while back that claimed that
a) not everyone could learn programming
b) you can predict who can and can't succeed in learning programming via a few simple tests given prior to teaching the introductory class. One test was consisted of several series of assignments (a = 20, b = 10, a = b, what do a and b now equal - that sort of thing) with no explanation as to how it works.

The idea was that students who had the ability to learn programming would apply a consistent meaning to the assignment operator for each problem; even if they meaning they chose was incorrect, they still applied it the same way to each problem. Students that didn't use a consistent meaning for every problem, but instead applied various meanings depending on the context of the problem probably weren't able to think in the way needed to program successfully. The authors' theory was that the second group was too caught up in the idea that the assignment operator had to have some larger, outside significance, that they just weren't able to accept it was an arbitrary rule with no external significance - which is all a programming language is. Thus, why they could never learn to program - they kept looking for significance where there was none.

They showed data from an intro programming class of theirs, where they gave their programming aptitude predictor test at the beginning, middle, and end of the course, along with midterm and final grades, as well as college entrance exam scores (they were trying to show that IQ wasn't a sufficient or good predictor of programming aptitude). Most of the people who gave inconsistent answers on the first predictor test had inconsistent answers on the other two tests and had poor grades in the course. It was a small sample size, but interesting.

Of course, if CS doesn't mean computer science, all that was irrelevant ;).

Forty-two said...

Here is a link to the paper:
http://www.cs.mdx.ac.uk/research/PhDArea/saeed/

Anonymous said...

yes: CS= Computer Science, But, no, you don't understand the specific course taught here.

It's called Structures and Interpretations of Computer Programming, and it's about the farthest thing from a standard computer programming course that any CS department offers.

It's a course in how to think. If you're a mathematician, it's a course in operator theory and recursion theory. It's really not about progamming in any standard sense, because it abstracts away everything you need to know about the machine. It's taught in Scheme, a dialect of LISP. There's no syntax to learn, no types to learn, no memory management. The course doesn't teach "assignment" until the 8th week, because it doesn't believe in it. Assignment is senseless until you have a notion of State, and so they teach scope and reference frames and state first, after several weeks of teaching recursion.

But your comments bring up a different point. I had a terrible time with assignment operators denoted by equal signs, because they are absurd, and I certainly had a problem with courses where I looked for significance and couldn't find it--badly taught Differential Equations has this problem, for example. But I can chalk ALL of that up to pedagogy: in every example of a course where arbitrary rules are assigneed with no significance, this is a shortcut that is NOT NECESSARY. There are rules to explain what assignment means. There are rules to explain why you use this differential equation, not another. There is a reason you invert and then multiply to divide fractions.

Anonymous said...

---your grade is based almost completely on your ability to get the programs done correctly.

What mythical difference do you see here from normal courses? I mean, in physics, your grade is completely based on your ability to solve the problems correctly. How is that different? Same in math.

But see my followup answer. I may revise this post to address it, but this course is not a "learn some data structures in C or Java" course. It's a course in how to think.

Rudbeckia Hirta said...

I think that a very similar thing happens in non-remedial college math. If you know enough algebra and have developed enough learning strategies that work for you, then you'll be fine in calculus. If you don't know algebra, then it doesn't matter. If you never do the homework (unless you're naturally gifted in math), then it doesn't matter.

If you know the prerequisite math-facts and you have a method for learning new math-facts, you will succeed by induction.

I'm not blaming high school algebra teachers. I bet you that kids would come to college knowing more algebra if they entered high school knowing arithmetic. K-12 should be teaching the basics AND teaching how to learn things. College assumes that you already know these things.

The most successful college math teachers are the ones with the personal charisma to entice students to want to do the work. It has very, very, very little to do with clarity of explanations or the structure of the course.

Catherine Johnson said...

This is an interesting way of looking at it.

I've been attributing the situation here, where parents and tutors are doing vast amounts of teaching and reteaching, to ed-school constructivism: students are to "take responsibility for their own learning" and "construct their own meaning," etc.

But my district - most districts, I presume - believes deeply in biology and in WYSIWYG.

Differences in student performance are everywhere presumed to reflect differences in student biology.

We experienced the reductio ad absurdem of this attitude in our recent meeting with the Earth Science teacher & the science department chair. (I think I described the scene in another comment...)

We were there because C's grades in the course, which will appear on his Sacred High School Transcript, range from A to F.

When I says "A to F" I don't mean A- to F+.

I mean A to F-.

The department chair and the teacher had 3 explanations for this:

1. C. has weak inferential thinking

2. Earth Science is a "conceptual" course, whereas all other science courses in the middle school are mere facts and memorization. C. excels at memorization, falters at grasping concepts.

3. "Remember, C. didn't do that well on the CTBS."

All three of these explanations attribute variation in performance to essential attributes of the child's brain and innate ability.

Unfortunately, neither Ed nor I had the wit to say, "So when C. is getting As in the course his inferential thinking is high, and when he's getting F- his inferential thinking has vanished?"

We were unreceptive to the essentialist explanations, to put it mildly, and we had the standardized test data to win.

At that point the two teachers stopped knocking down our kid and turned their sites on me.

The reason Cs grades were all over the place in the course, the teacher said, was that I hadn't signed Interim Reports and sent them back to school. Therefore the teacher didn't know if I knew that C. had gotten a bad grade on this, that, or the other.

That turned out to be correct.

I am now the Earth Science teacher.

Since I have begun teaching Earth Science C has had 3 grades: 86, 89, 90.

He is well above the class mean on all 3 tests.

The implications of this fact will not be acknowledged by the school, however.

Those being:

a) C's wildly variable performance in the course has nothing to do with cognitive deficits

b) teaching matters

One of the things I see here is pervasive illogic.

My district attributes performance to biology.

My district also expects parents to reteach and hire tutors.

Catherine Johnson said...

If Allison's course was indeed a course in how to think it makes sense that nothing the teacher does matters.

Cognitive science has generally concluded that it's impossible to teach "thinking skills" per se.

As far as I can tell, no one really knows how thinking and logic develop. (Vicki Snider says outright that no one knows how to do this.)

One interesting clue: Daniel Willingham says that once people get to a certain level of education --an advanced degree such as a Ph.D.--they often possess critical thinking skills that can be abstracted from the field in which they are expert and applied to fields in which they are not expert.

Ed has long said that the reason to study the liberal arts is that the liberal arts develop one's ability to think.

To the extent that that is true, I suspect the reason is that each liberal arts discipline has a different mode of thought, analysis, and evaluation of evidence.

The student is learning different modes of thought; perhaps learning different modes of thought leads to the development of a general ability to think well.

Catherine Johnson said...

The most successful college math teachers are the ones with the personal charisma to entice students to want to do the work. It has very, very, very little to do with clarity of explanations or the structure of the course.

Actually, Ed isn't seeing this (though of course you're right about charisma and motivation).

He is doing an amazing job teaching his undergraduate students to write history papers, and he's doing it using the principles of formative assessment and incremental teaching.

I'll post more about that...

He's also seeing this make a big difference with his T.A.s

SteveH said...

"What mythical difference do you see here from normal courses? I mean, in physics, your grade is completely based on your ability to solve the problems correctly. How is that different? Same in math."

Lots of hard work, and the program has to be correct. With other classes, you do the best you can and get partial credit. With programming, doing the work to get the program correct is huge. If you don't get the program correct, partial credit doesn't mean much.

When I was in college (I started in the card punch days.), computer courses were unlike all of my other engineering courses. You had to do whatever it took to get the programs done. It's kind of like all or nothing. You could get partial credit, but if you didn't get many programs done, you risked flunking the course.

I taught college math and computer science courses for years back in the 80's. I could teach the concepts, but getting a good grade required far more work than many of the kids ever expected. There was nothing I could do as a teacher to help students with the hard work or the drive needed to get the program perfectly correct. (except for deadlines) You have to really like the process.

It could be a motivation issue, it could be a conceptual issue, or it could be prerequisite issue, some courses require a lot of effort. In K-12, one should never have a course that requires that kind of huge leap. College is another thing, especially for your major.

Anonymous said...

--Lots of hard work, and the program has to be correct.

This isn't how this course is structured. The wrong program got partial credit, based on specific defined elements.

--With other classes, you do the best you can and get partial credit. With programming, doing the work to get the program correct is huge.

I really don't understand the difference. Seriously, in my firs term quantum physics course, the prof gave no partial credit. Hit or miss, done. if you can't solve the differential equation, you get a zero.

--If you don't get the program correct, partial credit doesn't mean much.

There's no NECESSARY REASON for a course to be structured this way. It's simply not a requirement of the material. There's nothing about teaching data abstraction, recursion, data flow, object and state models, or metacircular evaluation that requires it. Even in other more typical CS courses, it's not necessary.

Anonymous said...

--In K-12, one should never have a course that requires that kind of huge leap. College is another thing, especially for your major.


Why? Why should a college course REQUIRE a huge leap? Where's this assumption come from?

SteveH said...

--If you don't get the program correct, partial credit doesn't mean much.

"There's no NECESSARY REASON for a course to be structured this way. It's simply not a requirement of the material."

Perhaps there is no reason for even a college course to require a huge leap, but programming does require the ability to create correct programs. All of the programming courses I ever had weighted the programs very highly, and partial credit dropped off quickly if you didn't hand in a correct program. They are structured this way because students need to be able to finish that last 5 percent of the program, which could take 25 percent of the time.

This was nothing like the problems in the differential equations course I taught. I could have graded that course as all-or-nothing, but that wasn't necessary. Besides, all or nothing on hundreds of problems during a semester is not the same as all-or-nothing on 10 large programming assignments during a semester.

Programming courses are notorious for consuming huge amounts of time. I remember the data structures course I taught. I tried all sorts of ways to explain how B-trees worked. The students still had to put in the time to get all of the links just right.

I will say that I don't like the general idea that hard working students could still flunk a course. It sounds a little too close to being a cop-out by the teacher. Does it have to be this way? Probably not, but it would take much longer. Some "get" programming. Some do not.

Anonymous said...

Steve,

here's the link to the textbook we used:

http://www-mitpress.mit.edu/sicp/full-text/book/book.html

You should look at it. You will understand why for this class, getting the last 5% of the program corect shouldn't take 25% of the time.

But honestly, "programming" is different, now, and still, computer science isn't programming. Yes, some of it is, but it's more than that, and can be taught and graded.

Writing data structures like B trees with unit tests and test harnesses at compilation, etc. just isn't that hard. If you teach your students to write the design document, and then critique it, and then teach them to write the tests first in modern languages (and using modern editors that indent and color code), and then critique that, you've given them chances for partial credit on the stuff that really takes time-- and it isn't coding. Completing the itty bitty bits of the project doesnt' take a huge fraction of time like it used to, because you're not looking for an error on punchcards on in syntax.

SteveH said...

"... and still, computer science isn't programming. Yes, some of it is, but it's more than that, and can be taught and graded."

When I was in school, the two killer courses in computer science were operating systems and compiler design. For the operating system course, we had to create a working multi-tasking, time-sliced operating system in assembler. At the end of the course, we sat with the professor who timed and graded our system. Creating a full working program is the validation of knowledge and skills in computer science.

All of the computer science courses I taught (except for the hardware design course) were dominated by programming; not just pieces, but whole programs. It's easy to write functions, but it can be incredibly difficult to write whole applications.

Having taught courses in systems analysis, I know all about design and analysis tools; top down, bottom up, and outside in, but it's the final program that really counts. I've been in jobs that used all sorts of tools like flow charts, prototyping, Warnier Orr diagrams, data flow charts, and walk throughs. By the time you get to the coding, you find that most of these documents are incredibly wrong. You don't hear much about software engineering these days.

It's a nice idea to think that "coding" is just a small mechanical part of the process, but it isn't. Coding is often where you finally figure out what you want to do. Having written about a million lines of code (currently in Visual C++) over 35 years, I have a little idea of the capabilities (and painful limitations) of modern programming techniques and tools.

Computer science departments may have evolved to where you can specialize in non-programming areas (I've heard that you can get a computer science degree without taking a course in data structures.), but for programming, the proof will always be with the correctness of the final program. You could grade the parts, but it's the final product that matters.

Turbo Pascal was a huge improvement back when I was teaching. It streamlined the process enormously, but it still didn't help the students think better. In fact, they thought less. They just went through the motions faster.

Programming is different. It requires a huge amount of time to learn to do it well. You can try to apply a divide and conquer approach to the learning process, but that only goes so far.