student evaluations & college calculus at the USAFA

In primary and secondary education, measures of teacher quality are often based on contemporaneous student performance on standardized achievement tests. In the postsecondary environment, scores on student evaluations of professors are typically used to measure teaching quality. We possess unique data that allow us to measure relative student performance in mandatory follow‐on classes. We compare metrics that capture these three different notions of instructional quality and present evidence that professors who excel at promoting contemporaneous student achievement teach in ways that improve their student evaluations but harm the follow‐on achievement of their students in more advanced classes.

[snip]

[O]ur study uses a unique panel data set from the United States Air Force Academy (USAFA) in which students are randomly assigned to professors over a wide variety of standardized core courses. The random assignment of students to professors, along with a vast amount of data on both professors and students, allows us to examine how professor quality affects student achievement free from the usual problems of self-selection. Furthermore, performance in USAFA core courses is a consistent measure of student achievement because faculty members teaching the same course use an identical syllabus and give the same exams during a common testing period.5 Finally, USAFA students are required to take and are randomly assigned to numerous follow-on courses in mathematics, humanities, basic sciences, and engineering. Performance in these mandatory follow-on courses is arguably a more persistent measurement of student learning. Thus, a distinct advantage of our data is that even if a student has a particularly poor introductory course professor, he or she still is required to take the follow-on related curriculum.6

[snip]

Our findings show that introductory calculus professors significantly affect student achievement in both the contemporaneous course being taught and the follow-on related curriculum. However, these methodologies yield very different conclusions regarding which professors are measured as high quality, depending on the outcome of interest used. We find that less experienced and less qualified professors produce students who perform significantly better in the contemporaneous course being taught, whereas more experienced and highly qualified professors produce students who perform better in the follow-on related curriculum.

[snip]

Results show that there are statistically significant and sizable differences in student achievement across introductory course professors in both contemporaneous and follow-on course achievement. However, our results indicate that professors who excel at promoting contemporaneous student achievement, on average, harm the subsequent performance of their students in more advanced classes. Academic rank, teaching experience, and terminal degree status of professors are negatively correlated with contemporaneous value-added but positively correlated with follow-on course value-added. Hence, students of less experienced instructors who do not possess a doctorate perform significantly better in the contemporaneous course but perform worse in the follow-on related curriculum.

Student evaluations are positively correlated with contemporaneous professor value-added and negatively correlated with follow-on student achievement. That is, students appear to reward higher grades in the introductory course but punish professors who increase deep learning (introductory course professor value-added in follow-on courses). Since many U.S. colleges and universities use student evaluations as a measurement of teaching quality for academic promotion and tenure decisions, this latter finding draws into question the value and accuracy
of this practice.

These findings have broad implications for how students should be assessed and teacher quality measured. Similar to elementary and secondary school teachers, who often have advance knowledge of assessment content in high-stakes testing systems, all professors teaching a given course at USAFA have an advance copy of the exam before it is given. Hence, educators in both settings must choose how much time to allocate to tasks that have great value for raising current scores but may have little value for lasting knowledge. Using our various measures of quality to rank-order professors leads to profoundly different results. As an illustration, the introductory calculus professor in our sample who ranks dead last in deep learning ranks sixth and seventh best in student evaluations and contemporaneous value-added, respectively. These findings support recent research by Barlevy and Neal (2009), who propose an incentive pay scheme that links teacher compensation to the ranks of their students within appropriately defined comparison sets and requires that new assessments consisting of entirely new questions be given at each testing date. The use of new questions eliminates incentives for teachers to coach students concerning the answers to specific questions on previous assessments.

[snip]

Students at USAFA are high achievers, with average math and verbal Scholastic Aptitude Test (SAT) scores at the 88th and 85th percentiles of the nationwide SAT distribution.8 Students are drawn from each congressional district in the United States by a highly competitive process, ensuring geographic diversity. According to the National Center for Education Statistics, 14 percent of applicants were admitted to USAFA in 2007. Approximately 17 percent of the sample is female, 5 percent is black, 7 percent is Hispanic, and 6 percent is Asian. Twenty-six percent are recruited athletes, and 20 percent attended a military preparatory school. Seven percent of students at USAFA have a parent who graduated from a service academy and 17 percent have a parent who previously served in the military.

[snip]

These findings have broad implications for how students should be assessed and teacher quality measured. Similar to elementary and secondary school teachers, who often have advance knowledge of assessment content in high-stakes testing systems, all professors teaching a given course at USAFA have an advance copy of the exam before it is given. Hence, educators in both settings must choose how much time to allocate to tasks that have great value for raising current scores but may have little value for lasting knowledge.

Does Professor Quality Matter? Evidence from Random Assignment of Students to Professors
Scott E. Carrell
University of California, Davis and National Bureau of Economic Research
James E. West
U.S. Air Force Academy

=======

6 For example, students of particularly bad Calculus I instructors must still take Calculus II and six engineering courses, even if they decide to be a humanities major.

fyi

SAT percentiles

running with the big dogs

from Work Hard. Be Nice. by Jay Mathews:

One popular slogan irritated [Harriet Ball]: “All children can learn.” That was not the right message, she thought. It ought to be “All children will learn.” The word “can” was too passive. It meant the child was capable. That was not enough. There was a big difference between capability and achievement. Many educators thought it was up to their students and their parents to summon the motivation to use their God-given talents. Ball took her responsibilities more seriously. She brought this up every time she saw the slogan: “Uh-uh, I don’t want no ‘can,’” she said. “All of us will learn. I will learn from the kids. They will learn from me. Ain’t no ‘can.’ We will all learn.”
p.37

Did you celebrate National SAT Day?

Because according to this new SAT blog by Sol Lederman it was yesterday, June 23rd. He quotes Wikipedia:

The first administration of the SAT occurred on June 23, 1926, when it was known as the Scholastic Aptitude Test.^[19][20] This test, prepared by a committee headed by Princeton psychologist Carl Campbell Brigham, had sections of definitions, arithmetic, classification, artificial language, antonyms, number series, analogies, logical inference, and paragraph reading. It was administered to over 8,000 students at over 300 test centers. Men composed 60% of the test-takers. Slightly over a quarter of males and females applied to Yale University and Smith College.^[20] The test was paced rather quickly, test-takers being given only a little over 90 minutes to answer 315 questions.^[19]

I'm guessing that there were no SAT tutors then.

the nativity gap, part 2

from Erin Johnson:

The 2007 TIMSS does break out the US results by race. In 4th grade the US Asian subgroup performs only slightly less than the top Asian countries but by 8th grade the US Asian subgroup performs significantly lower.

4th grade US Asians: 582
4th grade Singapore: 593

8th grade US Asians: 549
8th grade Singapore: 599

Highlights from TIMSS (pdf file)

Note that the math scores of Singapore stay the same from 4th to 8th grade, but the US Asian subpopulation (as well as the entire student population) declines.

If math performance was purely an IQ phenom, we would expect that those 4th and 8th grade levels would be comparable, but this is not the case.

Also, there is evidence from Whitehurst that math curricula does matter. So it is not inconceivable that differences between math in Asia vs the US might account for performance differences.

Anecdotally, having used Singapore math it is easy for me to see why Asian math programs would be significantly better at enabling kids to learn math.

severely fragmented knowledge

Looking through the SAT prep posts, I found this problem that, just 12 days ago, I couldn't begin to do.

If 20 percent of x equals 80 percent of y, which of the following expresses y in terms of x?

(A) y = 16% of x
(B) y = 25% of x
(C) y = 60% of x
(D) y = 100% of x
(E) y = 400% of x

The Official SAT Study Guide 2006
ISBN-13: 978-0874477184
p. 550

Today it seems simple and obvious.

This is a classic case of inflexible knowledge. I've practiced solving many, many literal equations over the past few years, a skill I learned and practiced in high school, too. I had no trouble doing it then, and I've had no trouble doing it now.

So, in theory, I 'know' how to do this problem.

And yet when I encountered a literal equation involving percent, I was mystified.

What fresh hell is this?


from cranberry:

Funny, I didn't need to set up a formal equation for this. If 20% of x = 80% of y, y must be four times as small as x. Thus, 25% of x.

That's the kind of thing years of doing bar models does for you.

Not that cranberry has spent years drawing bar models.

the nativity gap in a nutshell

[I]mmigrant students actually do better if they begin their American education in high school rather than in elementary or middle school.

Foreign-Born Students New to N.Y. Outshine Native Born
by Sarah Garland
New York Sun
February 11, 2008 Edition

the nativity gap

the nativity gap

from Bostonian:

Where is the evidence, adjusting for race, that American math curricula are worse than those used in Asia?

William Schmidt, director of the U.S. national Research Center for the Third International Mathematics and Science Study, is the person to read on this subject:

The Role of Curriculum
A Coherent Curriculum: The Case of Mathematics (pdf file)

and see:
The Sequence of Mathematics Topics in Top-Achieving Countries (pdf file)


nativity gap

This Stiefel, Schwartz, and Conger study, while not an analysis of curriculum, supports Schmidt's argument:

Do Immigrants Differ from Migrants? Disentangling the Impact of Mobility on High School Completion and Performance
Leanna Stiefel, Amy Ellen Schwartz, and Dylan Conger

from the New York Sun's story:

Foreign-born newcomers to New York City's public schools are performing better than native-born newcomers, a New York University study shows.

The working paper by NYU education researchers titled "Do Immigrants Differ From Migrants?" has also deflated any notions that immigrant students tend to do worse in American schools the older they are when they arrive. On the contrary, the findings demonstrated that immigrant students actually do better if they begin their American education in high school rather than in elementary or middle school.

[snip]

Ms. Schwartz, speaking at an NYU symposium last week, said she and her colleagues had hypothesized that immigrant students "who come late are the ones who are really disadvantaged," guessing that their lack of language skills, the stress of moving to a new country, and institutional differences between the schools they came from and the New York schools might hurt their graduation rate and performance on tests.

Instead, they found that the foreign born "just do remarkably better," Ms. Schwartz said.

Foreign-Born Students New to N.Y. Outshine Native Born
by Sarah Garland
New York Sun
February 11, 2008 Edition

Here is Andrew Wolf's take:

Immigrant children outperform some native-born children in New York schools, my colleague Sarah Garland reported the other day. Indeed, it seems the longer newly-arrived children attend our schools, the worse they do. These conclusions come from a new study, "Do Immigrants Differ From Migrants?"

"The foreign born are whizzing by the native born at every level," one of the researchers, Amy Ellen Schwartz, said.

[snip]

If one drives past the Bronx High School of Science any given school day morning, one will encounter a seemingly endless caravan of yellow school buses. Most of these buses come from Queens, the borough known for its immigrant population. Sixty percent of those recently offered a spot at Bronx Science as a result of their test scores come from Queens, as are 40% of the new freshmen at Stuyvesant. It [sic] it turns out that a good number of these commuter students are recent immigrants, why do the city's specialized high schools seem to have such a disproportionate number of newly arrived students?

Whizzing By
BY ANDREW WOLF
March 3, 2008

I miss the Sun.

more from Wu

from Allison:

Going forward from what Catherine posted, Wu argues that most of what teachers know about fractions, and teach to their students, is wrong in the sense that it is unsupported by what the students have previously been taught.

Wu makes clear that you must be ruthless with your self when teaching. You must not allow yourself to use any concept you've not explicitly given your students. And almost all textbooks violate this rule in almost every lesson on fractions.

Nearly all textbooks avoid defining a fraction. Since they aren't defined, they don't define how you operate on fractions in terms of any real definition. Beyond that, when they do introduce operations on fractions, the books don't define operations in terms of what students know either. They teach rules without ever teaching where the rules come from. Their "explanations" invoke properties no one taught the students, but are just asserted. That means logically, their explanations are wrong, as you can't prove a statement by assuming the statement true.

For example, how does one justify multiplication of fractions? (I'll answer how later, but for now, discuss amongst yourselves.)

Yes, you can learn to use the rule properly without understanding where it is from, but you'll only go so far, and more, you will mistrust your teachers and mistrust math. Math will seem to be an endless series of magical statements with no relationship, and sooner or later, you'll run out of time/space/effort for keeping track of them. Properly understood, math is coherent, because every new step you take follows from the ones before.

a likely story

I'm sorry.

I have to say it.

I didn't buy the grew apart after 40 years of marriage business for one second.

question: Why exactly did we all have to read earnest accounts of 'gray divorce' & suchlike for a solid week?

Off-topic, I know.

we begin

Definition of a Fraction

Recall that a number is a point on the number line (§5 of Chapter 1). This chapter deals with a special collection of numbers called fractions, which are usually denoted by m/n, where m and n are whole numbers and n ≠ 0. We begin by defining what fractions are, i.e., specifying which of the points on the number line are fractions. The definition will be both clear and simple. If you find it strange that we are making a point of giving a definition of fractions, it is because this is something thousands (if not hundreds of thousands) of teachers have been trying to get at for a long time. Most school textbooks and professional development materials do not bother to give a definition at all. A few better ones at least try, and typically what you would find is the following:

Three distinct meanings of fractions — part-whole, quotient, and ratio — are found in most elementary mathematics programs.

[snip]

Such an explanation is unsatisfactory for several reasons. To say that something you try to get to know is three things simultaneously strains one’s credulity. For instance, if I tell you I have discovered a substance that is as hard as steel, as light as air, and as transparent as glass, would you believe it? Another reason for objection is that a fraction is being explained in terms of a “ratio”, but most people don’t know what a ratio is.2 In addition, while we are used to the idea of a division a ÷ b where a is a multiple of b (see §3.4 of Chapter 1), we are not sure yet of what 2 ÷ 3 means. So to use this to explain the meaning of 2/3 does not seem to make sense. Finally, we anticipate that fractions would be added, subtracted, multiplied and divided, and it is not clear how one goes about adding, subtracting, multiplying and dividing a part-whole, or a quotient, or a ratio.

This is why we opt for a definition that is both simple and clear.

Chapter 2: Fractions (Draft) (pdf file)
H. Wu
Department of Mathematics #3840
University of California, Berkeley
Berkeley, CA 94720-3840

"KIPP schools routinely outscore many that serve middle-class"

Report finds KIPP students outscore public school peers

Again.

I remember, 5 years ago, when I first started telling people KIPP kids were twice as likely to take algebra in the 8th grade and pass Regents as our kids.

Now here it is, 2010, and I'm still telling people KIPP kids are twice as likely to take algebra in the 8th grade and pass Regents as our kids.

Same deal with Scarsdale. Last winter our Interim Director of Curriculum reported that 80% of Scarsdale students take algebra in 8th grade.

Then she added: "We're not Scarsdale. We certainly don't have the resources of Scarsdale."

'We're not Scarsdale' explained why we can't have 80% of our kids taking algebra in 8th grade. If we were Scarsdale, we could, but we're not Scarsdale so we can't.

A parent in the audience logged onto the internet, pulled the data, and reported back to the board before the night was out that in fact Irvington has exactly the same per pupil dollar spending on instructional programming as Scarsdale, so: check.

But no matter.

Tonight the board will undoubtedly approve funding to enable our Interim Director of Curriculum and a generous contingent of teachers to devote a week or two this summer to rejiggering Trailblazers in consultation with a math person from up Bedford way. Bedford School District seems to be the last Trailblazers site in the county apart from Irvington (Scarsdale adopted Singapore Math 2 years ago), so obviously we Irvington taxpayers need to pay them, too.

Check back in 2015.

nice work if you can get it

$60 - $80K

$83 million

from Saxon Math Warrior:

It was one thing for John Saxon to take on the mathematics education establishment. It was another for him to take on a special interest group whose followers were bankrolled by the federal government. With $83 million in federal tax dollars being pumped into the math education reformists’ camps during the 1990’s, any entrepreneur not in the chosen circle, who opposed reform math methods, and who was publishing his own math textbooks would have to battle more than their high-dollar funding. He would have to challenge the political ideology that guided all their decisions regarding America’s math program.

msmi 2010 Followup

msmi 2010: Institute on Fractions is now history. YAY!

I'll leave Catherine and any others to talk about their take on it. I'll have some future posts on what I think are the biggest lessons parents and teachers need to help their students understand fractions, but this is just a roundup.

In the course of 5 days, we covered: definition of a fraction, equivalent fractions, decimals, addition, subtraction, multiplication, division, decimals again, and percent. We had more to do, but we couldn't get to it.

Based on the reaction of the teachers, it was a success. I have NEVER had a class of students where so many students worked so hard. No matter what their background, everyone tried to do the problems. Their effort meant that as the week went on, the students were more engaged and more knowledgeable. The teachers also built up their camaraderie with each other.

Based on their personal comments to me and the anonymous survey I gave at the end, their overall impression was quite high. Several teachers told me that this course was the first time anyone had ever explained how to think about fractions. One told me it was "a revelation" to them, another told me this was the first time they'd see a way to visualize multiplication of fractions. Most responded to our survey saying that this material had changed how they would teach permanently. Several teachers had a different kind of revelation, too: that other people in other schools/cities/states knew and felt as they did. They were connecting the dots not only on fractions, but on the state of math education.

Not that everything was perfect. I was terribly out of practice for being a teacher--bad board technique, bad handwriting, bad short hand in my own thoughts and words, instead of being clear, specific and slow.

I made several errors in sizing up my audience too. I assumed that since I had told the principals what to expect, that they had told their teachers. I assumed that teachers, given a pointer to a web site that had, e.g. Wu's CV on it, would have read such.

The biggest complaint was that it was too much material/days too long, and not enough worked out examples. One solution to the latter is to strongly encourage the teachers to read the textbook a day ahead of time. But part of that is the nature of the beast: there is an enormous deficit of knowledge to overcome. Elementary math teachers didn't go into that field because of their stength in fractions. The breadth of math inexperience-experience even for teachers of the same grade was very large, yet being math experienced didn't quite help, because while those teachers probably followed Wu's proofs more easily, applying his ideas to actual math problems was still a new universe to them, and their skill was sometimes a hindrance, because he was asking them to think an entirely different way than they were used to.

Lastly, I'm thrilled to have met all the people involved. Wu is a delight to work with/for, and I'd do this again with him wherever we can. He was personable and charming as well as brilliant. His wife was just as delightful. CassyT, KTMer, is an exemplary woman. She's a brilliant teacher and student of human nature, and her insights into teachers saved me countless mistakes. She shared her expertise with me in countless ways, and the whole thing would have fallen apart if not for her.

So, where to go from here? First, more Wu institutes! Let's bring MSMI to your locale! Second, the really big thing is to help teachers turn what they learned here into changes in their school. That's no small undertaking. I'll talk about that more in the next post. Last, more documents for everyone: condensing of Wu for parents and teachers. I'm sure you'll see work product of that around here shortly...

more than one way to solve it

paraphrasing Wu at msmi2010:

The idea that there is always more than one way to solve it is propaganda. Sometimes you're lucky to have one way.

question: what is the opposite of fuzzy?

answer: SAT math

54 questions, 70 minutes, many working parts

no time to 'think'

no time for 'many ways to solve it'

to score well on the SAT, you need to use the fastest method, and you need to know what the fastest method is about 5 seconds after you start reading the problem

if not sooner

talk and chalk

from Have Technology and Multitasking Rewired How Students Learn? by Dan Willingham

When you encounter a new technology, try to think in abstract terms about what the technology permits that was not possible in the past. It’s also worth considering what, if anything, the technology prevents or makes inconvenient. For example, compared with a chalkboard, an overhead projector allows a teacher to (1) prepare materials in advance, (2) present a lot of information simultaneously, and (3) present photocopied diagrams or figures. These are clear advantages. However, there are also disadvantages. For instance, James Stigler and James Hiebert noted that American teachers mostly use overhead projectors when teaching mathematics, but Japanese teachers use chalkboards.³³ Why? Because Japanese teachers prefer to maintain a running history of the lesson. They don’t erase a problem or an explanation after putting it on the board. It remains, and the teacher will likely refer to it later in the lesson, to refresh students’ memories or contrast it with a new concept. That’s inconvenient at best with an overhead projector.

³³ James W. Stigler and James Hiebert, The Teaching Gap (New York: Free Press, 1999).

Having managed to follow most of Wu's lectures, I am a huge fan of this method - and a huge non-fan of the PowerPoint now-you-see-it, now-you-don't approach to teaching.

limits of working memory
working memory posts

Saxon Math Warrior

from Education News:

John Saxon was hated by the math education establishment from the time he published his first algebra book in 1981. He still is, 14 years after his death in 1996. A West Point graduate with three engineering degrees, he declared war on those he blamed for creating the “disaster in American math education.” He insisted that math leaders had overseen this debacle and there was no personal accountability being demanded for the results of their radical ideology.

In John Saxon’s Story, a genius of common sense in math education, readers learn about his battles, his strong and colorful personality, and how his historically-based traditional math program made him, much to his surprise, a multimillionaire. This was in spite of high-powered and politically-connected math leaders’ efforts to destroy him.

The author, Nakonia (Niki) Hayes, is a retired math teacher and principal who used the Saxon program. She says she wrote the biography because John Saxon made a valuable and positive impact on thousands of American children. Go to Saxon Math Warrior for more information about how to order this original biography.

wow!

MSMI2010 was amazing.

Amazing!

Still collecting my thoughts - will post - but in the meantime, Niki Hayes' John Saxon bio is out!
My copy arrived in the mail last week.

The brain hurts most when being expanded...

Professor Wu at the beginning of MSMI2010 ...

...and at the end of MSMI2010.


That's the result of 40 hours of extreme fraction action.

Notice how giddy and hard working the attendees are after 5 days!

Better sign up now for next year's institute on Geometry.