I updated "Failure of future learning"

Update here.

Retrieval practice, the bad news

I was thinking the incredible results people find with retrieval practice sounded too good to be true, and now I'm worried they are:

Failure of further learning: the limits of repeated study and retrieval practice

Wish to heck I could pull the study. All I've got for the moment is the abstract:

Abstract

Previous research has shown that little benefit is achieved through spaced study and recall of text passages after the first recall attempt, an effect that we term the failure‐of‐further‐learning. We hypothesized that the effect occurs because a situation model of the text's gist is formed when the text is first comprehended and is consolidated when recalled; it dominates later recall after verbatim memories of more recent study episodes have been lost. Experiments 1 and 2 attempted to circumvent the effect by varying the activities of participants and requiring interactive exploration. In both experiments, recall after four, weekly sessions showed little benefit beyond performance on the first recall. Experiment 3 interfered with the formation of an immediate situation model by introducing passages that were hard to comprehend without a title. Performance improved substantially across four sessions when titles were not supplied, but the standard effect was replicated when titles were given. Experiment 4 made verbatim memories available by incorporating all re‐presentations and tests into one session; as predicted, recall improved over successive tests.

Failure of further learning: Activities, structure, and meaning
Catherine O. Fritz,
Peter E. Morris,
Barbara Reid,
Roya Aghdassi,
Claire E. Naven

Unfortunately, I don't understand the final sentence in the abstract.

I see that "interactive exploration" didn't help students remember anything beyond what they recalled during their first retrieval practice. (Another nail in the coffin of hands-on, guide-on-the-side activities as the cure for limited remembering and understanding.)

I see that difficult reading passages without titles (!) helped. That finding is especially striking in light of the SQ3R approach to reading comprehension, which involves paying attention to titles. (SQ3R isn't necessarily incompatible with this finding, but still...)

It sounds like the last sentence refers to a complete, or near-complete, re-creation of the original learning-and-quizzing episode.

Did students re-read and quiz themselves, both in the same "session"?

I should go read the summary at Jung's Biology Blog.

I'm probably going to enjoy it:

In other experiments, Fritz et al. show that FOFL occurs even when ideas are presented as itemized lists on Powerpoint slides (why am I not surprised).

Update: OK, I should have read first, posted later.

The last sentence has to do with word-for-word memorization, which does improve with repeated retrieval practice.

As for the rest of it, in fact, recall does improve over time, but not by nearly as much as one would hope.

"Elaborative" study techniques -- underlining, annotating, diagramming (Make It Stick highly recommends elaboration) -- had no effect at all.

Difficult texts without titles offer no advantage I can see from the summary: students recalled less during their first retrieval practice and then, over time, continued to recall more until they reached the level of students who had read a clearly-written text with a title. Lots more pain, no gain.

At this point, I don't think this study tells me too much. The authors' theory - which I do find quite interesting - is that a student's mental model blocks further learning. Specifically, the student's mental model blocks learning of content that didn't make it into the model.

That strikes me as highly likely; I've experience a "gist" effect myself, and I think "gist effects" are a major problem in any kind of reform effort.

However, fields of study are different from a book you are trying to remember, which is what the students in this study were trying to do.

When you study a field, one of the things you are learning is the field's organization and categories.

New content gets slotted into pre-existing categories. That's why the more you know about a subject, the easier it is to learn new aspects of that subject. You have a mental 'gist' of the subject that is 'hungry' for new content, or at least wide-open to it.

Make It Stick - is 3 the magic number?

I'm reading Brown, Roediger, and McDaniel's Make It Stick in a mixed state of trepidation and fascination.

Trepidation because the wording is so incautious and, often, so imprecise that I think it's entirely possible we'll see Make It Stick touted as justification for bad teaching of every stripe.

e.g.: the preface spends a great deal of time denigrating "drill and kill," which the writers take to mean massed practice. Also, "easy" lessons are slammed on grounds that learning should be hard. (So drill and kill is too easy?) And at one point the text baldy avers, sans footnote, that "people do have multiple intelligences" while in the same breath asserting that there's no evidence learning styles have anything to do with anything. So instead of worrying about our learning styles we should all use all of our multiple intelligences all the time because you learn better when you "go wide," drawing on all of your aptitudes and resourcefulness, than when you limit instruction or experience to the style you find most amenable.

What can any of this possibly mean?

Hard is good but hard drill-and-kill is bad .... learning styles are meaningless but we should deploy all of our "intelligences" all of the time (dancing in math? math-ing in dance?) ...

And does the exhortation to "go wide" mean multisensory programs are always to be preferred? I'm completely open to that possibility myself, but there's no listing for "multisensory" in the index, so who knows?

But that's the preface.

The first chapter, on retrieval practice, is riveting.

Retrieval practice, or the testing effect, refers to the finding that taking a test -- any kind of test, in-class or a quiz you give yourself -- increases your memory of the material you are trying to learn. In other words, taking a test or a quiz is a form of practice in which you practice remembering.

What's more, simple retrieval practice is probably superior to the kind of "active," "higher-order" learning students are purported to do via concept mapping. It also appears likely that retrieval practice produces knowledge that is readily transferred to new contexts and to problem solving.

In short, memorization makes you smart. (That's not how Brown, Roediger, & McDaniel put it.)

What is shocking to me is the tiny amount of retrieval practice the middle-school students in Roediger's studies needed in order to recall the material they had covered in class: just 3 "low-stakes" (ungraded) clicker quizzes in all. One quiz at the beginning of the class (on material they were supposed to have read the night before), one quiz at the end of the class (after the teacher's lecture on the same material) and one before the unit test a few weeks later. Students scored a full letter-grade higher on quizzed than on un-quizzed material.

When Roediger expanded the study to 8th-grade science, students scored an average of 92% on quizzed material, 79% on un-quizzed material. They still remembered the quizzed material eight  months later, for the final.

After 3 ungraded quizzes no one had to study for.

Is 3 the magic number?

Here are Rawson & Dunlosky on "How Much Is Enough?"

The literature on testing effects is vast but supports surprisingly few prescriptive conclusions for how to schedule practice to achieve both durable and efficient learning. Key limitations are that few studies have examined the effects of initial learning criterion or the effects of relearning, and no prior research has examined the combined effects of these 2 factors. Across 3 experiments, 533 students learned conceptual material via retrieval practice with restudy. Items were practiced until they were correctly recalled from 1 to 4 times during an initial learning session and were then practiced again to 1 correct recall in 1–5 subsequent relearning sessions (across experiments, more than 100,000 short-answer recall responses were collected and hand-scored). Durability was measured by cued recall and rate of relearning 1–4 months after practice, and efficiency was measured by total practice trials across sessions. A consistent qualitative pattern emerged: The effects of initial learning criterion and relearning were subadditive, such that the effects of initial learning criterion were strong prior to relearning but then diminished as relearning increased. Relearning had pronounced effects on long-term retention with a relatively minimal cost in terms of additional practice trials. On the basis of the overall patterns of durability and efficiency, our prescriptive conclusion for students is to practice recalling concepts to an initial criterion of 3 correct recalls and then to relearn them 3 times at widely spaced intervals.

The chapter also says that short answer and essay tests are probably superior to flash cards and multiple choice, but flash cards and multiple choice produce superior retention, too.

Clicker quizzes work.

I need a clicker.

kids need spiraled practice, not spiraled instruction

Hainish left a link to a terrific post by a math teacher:

11th grade here = 9th grade here. In fact, Algebra 2 was such a rehash of the district's Algebra 1 course that some teachers called it "Algebra T-o-o." And really, the same point could be made about math curriculum as a whole in the U.S., since most content for any given year is a review of content from previous years. (The Common Core State Standards may help change this, but I'll believe it when I see it.)

This approach, where we touch on lots of topics each year--rather than go deep with fewer topics--and then revisit them in subsequent years is often called spiraling. But what it is for many students is stifling. And this is as true for kids who've yet to master a skill as it is for those who nailed it right away. I first noticed this when I taught 9th grade Algebra classes where every student was performing at least two years below grade level.

"Meet them where they are," fellow math teachers advised me. Makes sense, I thought, since I couldn't imagine teaching Algebra to kids who didn't know basic arithmetic. But what I soon learned is that perception matters more to students than performance. For many kids, having seen something is akin to having learned something. "Man, we already know this," students said, as I presented lesson after lesson on fractions, decimals, and percents.

Other students, meanwhile, knew they didn't understand the material, but had given up hope of ever understanding it. The implication was therefore the same for all students: encore presentations on previous years' topics were pointless. And though I was able to engage a few students when I found new ways to present old topics, one group of students was always slighted: those who really did "already know this."

[snip]

The problem, of course, goes back to the disconnect between kids seeing something and actually learning--and retaining--it. But if it didn't sink in for them the first, second, or third time a teacher presented it, why should we present it again?

We shouldn't. At some point the focus needs to be on students practicing math rather than teachers presenting it.

[snip]

[W]e should provide students spiraled practice, not spiraled instruction. When I did this in 10th grade Geometry classes, students said they learned more Algebra than they had learned in their 9th grade Algebra course. And, as a result, they were ready for more advanced math--starting with Algebra T-w-o.
Spiraled Instruction, Stifled Learning
By David Ginsburg on March 5, 2012 8:35 PM

Wow!

the myth of regurgitation

Educators rely heavily on learning activities that encourage elaborative studying, while activities that require students to practice retrieving and reconstructing knowledge are used less frequently. Here, we show that practicing retrieval produces greater gains in meaningful learning than elaborative studying with concept mapping. The advantage of retrieval practice generalized across texts identical to those commonly found in science education. The advantage of retrieval practice was observed with test questions that assessed comprehension and required students to make inferences. The advantage of retrieval practice occurred even when the criterial test involved creating concept maps. Our findings support the theory that retrieval practice enhances learning by retrieval-specific mechanisms rather than by elaborative study processes. Retrieval practice is an effective tool to promote conceptual learning about science.

Most thought on human learning is guided by a few tacit assumptions. One assumption is that learning happens primarily when people encode knowledge and experiences. A
related assumption is that retrieval—the active, cue-driven process of reconstructing knowledge—only measures the products of a prior learning experience but does not itself produce learning. Just as we assume that the act of measuring a physical object would not change the size, shape, or weight of the object, so too people often assume that the act of measuring memory does not change memory (1, 2). Thus most educational research and practice has focused on enhancing the processing that occurs when students encode knowledge – that is, getting knowledge "in memory". Far less attention has been paid to the potential importance of retrieval to the process of learning. Indeed, recent National Research Council books about how students learn in educational settings (3–5) contain no mention of retrieval processes.

It is beyond question that activities that promote effective encoding, known as elaborative study tasks, are important for learning (6). However, research in cognitive science has challenged the assumption that retrieval is neutral and
uninfluential in the learning process (7–11). Not only does retrieval produce learning, but a retrieval event may actually represent a more powerful learning activity than an encoding event. This research suggests a conceptualization of mind and learning that is different from one in which encoding places knowledge in memory and retrieval simply accesses that stored knowledge. Because each act of retrieval changes memory, the act of reconstructing knowledge must be considered essential to the process of learning. Most prior research on retrieval practice has been conducted in the verbal learning tradition of memory research (12). The materials used have often not reflected the complex information students learn in actual educational settings (13).

Most prior research has not used assessments thought to measure meaningful learning, which refers to students' abilities to make inferences and exhibit deep understanding of concepts (14, 15).

[snip]

[B]oth elaborative concept mapping and retrieval practice are active learning tasks, and our results make it clear that whether a task is considered "active" is not diagnostic of how much learning the task will produce.

The Critical Importance of Retrieval for Learning
Jeffrey D. Karpicke* and Janell R. Blunt
Sciencexpress Report
www.sciencexpress.org / 20 January 2011 / Page 1 / 10.1126/science.1199327

Although I read the research that came out a couple of years ago on learning-via-testing, I hadn't absorbed the idea that testing is a form of studying.

Years ago, I learned the distinction between recall (remembering an actor's name, say) and recognition (recognizing an actor's name when you hear it). Recall is harder.

I thought the learning-testing effect was a simple matter of working on recall as opposed to recognition.

Looks like it's not. Remembering always means reconstructing.


news flash: There's no such thing as regurgitating facts.