kitchen table math, the sequel: advanced writing in the 12th grade as assessed by NAEP

Monday, December 3, 2012

advanced writing in the 12th grade as assessed by NAEP

According to NAEP, in 2011 only 3% of high school seniors were able to write an essay as good as the one below, which is considered "Advanced."

% scoring "Advanced":
4% of white students
5% of Asian students
0% of black students
1% of Hispanic students.

5% of students whose parents had college degrees scored in the Advanced range.

5% scoring Advanced would be fine if "Advanced" meant Advanced. But it doesn't. "Advanced" on NAEP means 3% of high school seniors are able to write a coherent statement on the subject of:
  • Story or personal narrative about real/imagined difficult choice
  • Essay about technology important to student
  • Letter persuading council to build/not build convenience store
All 3 of these prompts call for opinion and the marshaling of evidence strictly (or nearly so) from the student's personal experience, and that is not at all what college writing is about. Nor is it the kind of writing one does in business or the professions.

From The Nation’s Report Card: Writing 2011:
Sample Task: Writing to explain
When writing to explain, the task of the writer is to bring together relevant information and to present this information with focus and clarity so that the topic becomes understandable to a reader. The sequence of ideas, and how ideas are arranged, must cohere and contribute to the communicative purpose.

One of the writing tasks from the twelfth-grade assessment asked students to write about a type of technology that they use in their lives and why they value that technology. The Value of Technology task began with a short video about young people’s use of technology. This video included animation and statistics about technology use. The written part of the task then specified an audience for students to address in explaining the value of a particular technology. Responses were rated using a scoring guide ranging from “Little or no skill” to “Effective.”

The sample student response shown below was rated as “Effective” in responding to the task about the value of technology. After an opening paragraph that defines video games and introduces the ideas to be developed throughout, the writer constructs the explanation primarily through use of personal experience. This approach skillfully communicates the value of video games through the use of detailed descriptions of specific games and what the writer has learned from them. Ideas are fully developed, and the rich use of explanatory details establishes a distinct voice speaking intelligently from experience. This response demonstrates skills associated with performance at the Advanced level. Twelfth-grade students at this level are able to craft responses that strategically accomplish the communicative purpose.

Student response - Grade 12 - Advanced
Videogames are a primary source of entertainment for people of all ages. Videogames are discs or cartridges that hold data; once a disc or cartridege is inserted into a gaming console, the data is read and displayed on the screen along with prompts that allow the game to be played. Games have many genres ranging from fighting to educational and can be used for than just mere entertainment. I personally have been experiencing what videogames have to offer for over five years now. Gaming is not just something that people do for fun, people can play videogames for many reasons. Videogames are an important factor in many peoples lives including mine and are a valuable type of technology.

I have been playing videogames from a very young age. Mario was the first game I was ever introduced to and it was not through playing; through sheer coincidence my mother realized that the theme music to Mario put me to sleep as a baby. Once I was old enough to hold a controller I began playing the game. Ever since that moment I have been playing videogames. Games are multi-purposed; to some it is merely a form of entertainment, but to others it could be their job. Some people argue that games are a waste of time and that they are not product. I beg to differ; games are important to me because not only do they give me something to do to pass time but they are also educational. A prime example of this is a game I was introduced to by my cousins, Runescape. When I was about thirteen I had went to see my cousins up state and I saw them playing this browser game called Runescape (a browser game is a game that can be played within an internet browser without the need to download or upload information from a disc or cartridge). Me being the person I am, I was curious as to what it was so I began to ask questions. By the end of the day I learned two things about that game, two things that to some gamers, were their favorite word. It was a MMORPG (massive multiplayer online role playing game) that was free; in essence it was a free gamethat I didn’t have to download and I could do basically whatever I wanted that was allowed in the game. Within the game you could do any of the various skills offered, quests, and even fight against other players from around the world with you’re avatar. Once I got home, I of course signed up and began to play. Throughout the few years I played that game I realized it was set in Medieval times and I learned many things about that age. I learned the process it takes to turn ore into metal, what smelting is, how leather is crafted into clothing, how clay is used, and some of the politics of Medieval civilizations throughout the quests of the game. Although I would spend hours on this game and it seemed like I was doing nothing, I infact was actually learning.

Another game my cousins introduced to me was Age of Mythology. The game was a PC game(which means it had to be bought and it contained disc which had to upload the game onto you’re computer or device and then the game could be played) and I had played it at my cousins and eventually went on to buy it. If mythology was a subject in school, this game could be the teacher. This game focuses around Greek, Egyptian and Norse mythology. You follow the antagonists (which you name) through all three civilizations chasing an evil minotaur that is attempting to end the world. You begin in a fictional Greek city and eventually move throughout the world. This game teaches any of it’s players not only how armies from all three civilazations worked but those civilazations major Gods, minor Gods, demigods and mythological creatures. Stories based on mythology or fact are also told and experienced throughout the game; such as the Trojan Horse and Ragnorak. I have never picked up a book based on mythology or ancient Gods but because of this game I have an extensive knowledge of the mythology of those three cultures. Games are important in society; they give people a hobby and peace of mind. They can also be used for educational purposes. Toddlers no longer read books to learn how to read, write, and spell, they are given toys and games to play. Games hold a high position in society and can be beneficial to those who use them if they wish to use them in that way.
This student, who is certainly a competent "personal" writer, has advanced a thesis: video games are valuable for more than entertainment.

In support of his thesis (I'm assuming the writer is male), he tells us that he learned "the process it takes to turn ore into metal, what smelting is, how leather is crafted into clothing, how clay is used, and some of the politics of Medieval civilizations" from a video game. This knowledge he acquired over a number of years and many hours of play.

In the next paragraph he tells us that although he has "never picked up a book based on mythology or ancient Gods" he nonetheless possesses "an extensive knowledge of the mythology of [Greek, Egyptian, and Norse] cultures" thanks to another video game. He provides no further detail as to what this knowledge consists of, or how long it took him to acquire it.

Essentially, the evidence this writer offers in support of his thesis boils down to: I remember stuff I saw in my video games.

The essay concludes with the assertion that "toddlers no longer read books to learn how to read, write, and spell." The writer offers no evidence to support this claim and seems not to know the meaning of the word "toddler." Toddlers have never read books, now or in the past, because toddlers are too young to read. They can't play video games, either, for that matter.

For my money, this essay is pretty much the exact opposite of what an Advanced high school senior should be able to produce in timed writing.

Very worrisome.

29 comments:

Catherine Johnson said...

I think this essay is an argument for teaching the 5-paragraph paper.

An 'advanced' 17-year old writer, using the 5-paragraph form, would know that he needed a thesis ***and*** 3 topic sentences: 'sub' ideas that develop and explain the thesis.

Allan Folz said...

Funny you mention the 5 paragrapher. Reading it, I immediately thought of that. It seems like the 5-er, if you jammed into 3. It obviously follows some sort of cookbook outline and hits many of the check-boxes that are going to be on a grading rubric.

I suspect it was written by a reasonably bright kid that's never been pushed too hard by the system and never found his school work interesting enough for him to want to push himself, at least, not interesting enough compared to video games.

Laura in AZ said...

Oh my... Me, being the person that I am, find this essay an argument for and against a LOT of things. Not the least of which is to continue to homeschool my daughter! LOL

Seriously? This was considered "advanced?" It's horrible! I expect better from my 8th grade daughter with Asperger's.

Thanks for the article Catherine - I appreciate the work you put into all of this!

Catherine Johnson said...

Hey Laura!

Catherine Johnson said...

btw, I just copy edited & commented on a paper written by Susan S's son (a paper for a college committee, no less!)

It was FANTASTIC. Absolutely great.

Susan's been 'afterschooling' her son in grammar & writing for a long time, and WOW. He's way, way, way past this paper.

SO: I conclude that it's possible to afterschool decent writing skills at home, and I wish I had done more.

(I should add that Susan's hs wasn't just teaching personal writing. I think they use a version of Toulmin argument ... but it sounds like instruction has been confusing ---- )

Glen said...

Does SusanS have any advice for those of us who would like to do the same? The more details, the better....

Catherine Johnson said...

Oh Glen - Great idea! I'm going to bug her to write a post - or at least a long comment. I know she did lots of sentence combining & old-fashioned grammar. I've got the grammar books she used - weave to check the titles.

Catherine Johnson said...

Susan used Genevieve Walberg Schaefer's books:
Steps to Good Grammar
Understanding & Using Good Grammar

I remember her getting a paragraph book from EPS (I have it) & also the Kilgallon books, but I don't know how much time she spent w/them.

I'll find out or she'll tell us---

Anonymous said...

I'll write more later today, but the biggest impact on me besides finding KTM was when I met with a homeschooler friend and she handed me her copy of Susan Bauer's The Well-Trained Mind.

I did start with another of her books, Language Lessons, when he was around kindergarten/1st grade, but by second grade, I knew they weren't going to coherently teach any grammar, any more than I was taught in the 60s.

I made up my own worksheets and just focused on 8 parts of speech and later parts of a sentence. I did use the Steps to Good Grammar book, but I simplified the sentences since they were so young. I did follow the sequence and techniques in that book, however, and it was one of the smartest things I did.

It wasn't hard. We didn't belabor things. We did the worksheets maybe 3/4 times a week for maybe 10-15 minutes. Almost Kumon-like. But by middle school when they start to actually teach this stuff, many kids struggled because they had so little exposure before that. My son was clear as a bell on all of it.

In fact, one teacher wrote on his report card, "Your son really has a 'knack' for grammar!" No, he doesn't have a knack for it--he was just taught it. It really isn't rocket science, but it won't happen naturally for most kids.

My biggest regret was dropping the ball in middle school. When I found out how bad his writing fundamentals were in the 8th grade, it became a game of catch-up. Catherine remembers that well because we were both looking for books to get us through it.

Anyway, I'll be back a little later...

SusanS

Bostonian said...

How are students supposed to supply evidence on some random topic unless they are allowed to use the library or the Internet? Maybe a better form of essay test is to give students some written material and then ask them to write a persuasive essay using that material.

Anonymous said...

That student is fluent, but he is not proficient. Today's teachers aim for fluency, so in some sense this student represents success. But he is not really using evidence so much as examples (about himself), in a context where only a huge mass of examples (i.e. data) would support a thesis.

Glen said...

Thanks, Susan. I'll be sure and tune in to the next episode of The Lone Writer, where we find out how our hero manages to escape his predicament. I can hardly wait to see how he pulls it off.

In the meantime, I have a question about Episode 1. The two Shaefer books are here. I can't tell how the green book relates to the orange. Is it the same book but with reproduction rights? Is it a different book but at the same level, offering a second round of exercises for the chapters in the orange book? The orange book claims to have exercises, too, but is it mostly tutorial with only a few example exercises, while the bulk of the exercises you really need are in the green volume? Or does the orange book contain all the exercises you need at that level, and the green book is the next level, to be used after finishing the orange?

They are pretty expensive to be buying blind, so I wish I knew what they really were. (And a note to anyone reading this who decides to save money by buying the "BRAND NEW" one offered for $19.95: it's the purple "revised edition" instead of the orange "second edition.")

Thanks.

Glen said...

And Susan, you're lucky your son has such a knack for grammar. After lots of tears, frustration, and hard work with the Singapore books and Dad, my sons eventually became lucky to be so naturally gifted at math. Now I'm hoping to find some high-quality training materials so that, with time, I can make them into born writers.

Anonymous said...

Glen,

The orange one is really all you need. I have the green one and did work through it somewhat, but the orange one starts right from the beginning.

Each chapter starts with an assessment quiz, which is always enlightening. I had to re-teach myself some things. The book seems written towards teens and adults, so it's actually a good brush up course for adults who've forgotten everything, or just need a reminder.

Each page has around 15 problems or sentences. The answers are directly opposite. Sentence diagramming is also taught from the beginning, so it builds along with the grammar. I like the fact that I had the answers right there. I didn't have to go fishing around in the back.

My sons were in early grade school when I started, so I just simplified the sentences, but kept the sequence of the book. There are also a ton of free homeschool sheets.

In middle school, it was more of a patchwork. Again, I wish I hadn't dropped the ball like I did. Catherine and I both did Megawords with our sons, Catherine more than me. Those are also good one page lessons that don't take up too much time, but cover things that aren't hardly covered in middle school (They're too busy journalling.)

Two books l liked a lot were Summarizing, Paraphrasing, and Retelling by Emily Kissner. Schools stick their noses up at the old fashioned "book reports", but I found my son actually needed direct instruction and practice performing these basic skills. I'm sure he's not alone.

Another good middle school and up one was Analyze, Organize and Write by Arthur Whimbley and Elizabeth Lynn Jenkins. I skipped around with that since it's more involved.

I also taught my son to outline by just giving him short chapters of some text to read. He was really lousy at identifying the main points and their supporting points. It took some practice, but he improved. I never found a decent outlining book.

Two others that I used a lot were Killagon (sp?) sentence combining books. I had all three and they are invaluable. I forgot their exact name, but others around here use them. They're not too expensive. I was looking over a scholarship essay of my son's this weekend, and I had several places where he could combine his sentences. Because we've done it so much, he knows exactly what I want him to do.

And finally, Catherine's fave, Writing to the Point by William J. Kerrigan. It's very wordy, but if you can get to the point of what he's saying, it is really helpful for kids starting on essays. I used his ideas a lot.

Whew. That's all I can think of for now. I know there was one other that I loved, but I can't think of it. Hope some of that helps.

SusanS

Glen said...

It helps a lot. Thanks. Is it a bit ironic that a text on getting to the point is great except for its wordiness and inability to get to the point? Just wondering.

I'm still wondering, though: How is that green book related to the orange book? Next level? Is it a doubling of the exercises at the same level? Is the orange mainly tutorial so the green book, say, quadruples the exercises at the same level? Other?

Anonymous said...

I can't find my orange book at the moment, but it looks like the green one lightly reviews a few things before going into other topics. I remember the orange book covering a lot, though. (I don't think we got all the way through it.)

In the intro of the green book it says, "The first volume, Steps to Good Grammar, covers in one year all the grammar instruction presented during elementary school through seventh grade. It provides students with a fundamental knowledge of the construction and function of the basic parts of a sentence."

On the second (green) volume, it says: "The second volume provides students with instruction that leads to a thorough understanding of the construction and function of advanced, complex grammatical elements used in English sentences. It begins with an essential, concentrated review of the material covered in the first volume plus pertinent, more advanced facts about the use of each element."

That describes it pretty well. They are both tutorials with short explanations for the "teacher" to use. Then the exercises. I am amazed looking at them again how much grammar there is to learn.

I think the green one would be hard to use if you weren't really clear on basic grammar to begin with. Both books are over 300 pages, so they're both packed with topics that build off of the last topic (usually.) The second volume also moves into clauses and phrases (gerunds, infinitives, etc.) in a way that I don't remember the first volume doing.

SusanS

SteveH said...

NAEP, 12th grade, and one grading standard. That's like taking all of the remedial, basic, college prep, honors, and AP kids and defining what is "advanced" for all of them. I stopped paying attention to NAEP (and our state tests) in K-6. CCSS suffers from the same affliction.

Since I haven't gotten my son to start practicing for the SAT essay-writing section yet, where are good examples that get top marks? That should be a better definition of "advanced" for timed writing. Then again, when will you ever need that kind of speed in the future. I would probably flunk. I take FOREVER!

Anonymous said...

Glen,

You can probably search KTM for Writing to the Point. Catherine did a few posts on it, which are actually to the point and helpful.

Steve,

Do you have big SAT prep books? I know with the ACT ones, they have sample essays with comments. I sat down with my son and had him read and critique each one. Then we read the book's critiques to see if he was on the mark or not. The low graded ones are pretty obvious, but the top three are more subtle.

For people taking the ACT there is a new way they approach the essay. They presently grade the essay on a 2-12 scale, but then combine it with the English subtest score and come up with a new composite score called the Writing/English, scored on the 1-36 scale.

So while my son's essay was pretty average, he had managed to get a 34 on the English subtest (thank you Steps to Good Grammar), and ended up dragging his W/E score up to the 30s. For a kid who hates writing, this was miraculous.

The Writing/English composite score does not affect the other composite score (with the 4 subtests), so now on the ACT you see two composite scores.

He also mentioned that the prompt was one he had seen before in one of the prep books. That definitely helped him, I'm sure.

SusanS

Glen said...

Susan, thanks for all of this. Now that I know that the orange book is essentially Level I while the green is Level II, I won't have to decide on the green one until we've finished the orange. Perfect. Thanks again.

forty-two said...

How are students supposed to supply evidence on some random topic unless they are allowed to use the library or the Internet?
The advice we were given for the TX state exams (way back when they were the TAAS tests) was to just make up evidence. The reasoning was that, since the test was to see if you could write a coherent essay (not if you had lots of pertinent facts on the arbitrary topic fortuitously stored in your head), made up facts which supported your point were more to the purpose than real facts that didn't support your point. They were awfully wide open with what constituted "acceptable evidence" in any case - personal experience qualified, as did TV shows and movies. I'm pretty sure I've read the same advice wrt the SAT essay (but graduated before that was introduced, so no firsthand experience on if it works).

Maybe a better form of essay test is to give students some written material and then ask them to write a persuasive essay using that material.
That sounds an awful lot like the AP US History DBQ: Document-Based Question. They gave you a topic on which you had to take a side, plus nine documents that, between them, provided evidence for both sides. You used the documents plus your overall history knowledge to argue your side. I didn't find them to be particularly hard, but allegedly they were a real bugaboo to many - students didn't know the difference between just retelling what the documents said versus using what the documents said to prove their point.

SATVerbalTutor. said...

I know the discussion wandered away from the "advanced" essay in question a while ago, but just wanted to point out that the kid has the pronoun/comma splice problem (not recognizing that an independent clause can start with a pronoun and so using a comma instead of a period to separate two sentences). That's a pretty basic error -- basic, of course, being exactly the *opposite* of advanced...

SteveH said...

"Do you have big SAT prep books?"

My Blue Book is currently loaned out, but we haven't even looked at the essay section yet. We've mostly focused on Erica's "The Ultimate Guide to SAT Grammar" and Mike's "PWN the SAT: Math books. I highly recommend both. They get right to the point and don't waste your time.


"He also mentioned that the prompt was one he had seen before in one of the prep books. That definitely helped him, I'm sure."

I plan on collecting many of those prompts so that he can practice the preliminary thinking and organization process. It seems as if one should start with the examples and let them determine your position. In some cases, I find better (enough?) justifications for the contrary position. If you find good examples, the essay should write itself. I find many of the prompts stupid. How much can you hijack a prompt, or at least, interpret it with a new spin?

Here is one:

"Many people admire leaders who have effectively achieved their goals. But too many leaders are dedicated to achieving their goals at the expense of caring for the people who follow and support them. Leaders should be judged on how well they treat people, not on the achievement of their goals."

"Assignment: Should leaders be judged according to how well they treat people? Plan and write an essay in which you develop your point of view on this issue. Support your position with reasoning and examples taken from your reading, studies, experience, or observations."


Can you argue with both positions? Can you argue about what judging means? Can you argue with "too many"? Can you question why they don't talk about the people who do NOT follow and support the leaders? Those elected in politics always treat their followers nicely. The real question is how they treat the rest.

In other words, can you argue that the question is stupid or wrong? In a nice way? Can you use sentences without verbs? Can you get a lower score because of your position, not your writing? I suppose you can if "they" think you are going off-topic. Who are "they"? Do "they" not appreciate us "concrete-sequential" types?

gasstationwithoutpumps said...

The SAT essay does not carry as much weight as people ascribe to it. My son just got his SAT scores back from the November SAT. He has always struggled with writing, particularly timed writing with lame prompts, so he only got 8/12 on the SAT essay. Despite that he got 730 on the writing section.

The math part of the SAT has a very low ceiling---missing one question dropped his score from 800 to 770. I'd still like him to take the ACT, to see whether it is any better at discriminating among students in the top few percent.

SteveH said...

"...so he only got 8/12 on the SAT essay. Despite that he got 730 on the writing section."

That's good to know. My son is very good with grammar. I'm not exactly sure why. His writing is unknown for this format.

Is your son a junior or a senior?


"...missing one question dropped his score from 800 to 770. "

I know! My son has no problems with the math questions except for those last very few. I wonder if the ACT is as sensitive at the top in math. Does the ACT discriminate based more on knowledge at the top end? The extra trig stuff looks very easy so I don't know if they do it as much as the SAT.

Being in major East Coast SAT territory, I don't know if it's reasonable to substitute the ACT. I had a discussion with another parent about this and we both wondered whether to believe colleges when they say it doesn't matter. The score correlation chart between ACT and SAT makes it seem like the ACT is a better bet at the top score end. My niece, who is not a math geek, got a 34 on the ACT. Perhaps the ACT is an easier test to score in the top one percent. Maybe that won't last too many years if more people start taking the ACT. I think they correlate based on percentages. It makes me wonder if there is a window of opportunity. I'll have to have my son take a sample ACT test. Just one more testing requirement I'm placing on him. He won't like that.

gasstationwithoutpumps said...

I was going to say that I doubt very much whether either test is "and easier test to score in the top one percent". After all, on both tests, 1% of students will score in the top 1%.

But then I looked at score distributions for the SAT. To get in the top 1% on SAT math, you need to get every question right—the test is so easy that missing one question drops you out of the top 1%. I don't know how many ACT math questions you need to miss to drop to 33 (the 98%ile score in math), but I suspect it might also be only one.

SteveH said...

"After all, on both tests, 1% of students will score in the top 1%."

But that's my question. Are they equal? Are the top Midwesterners as anal in test prep as the top East and West Coasters?

They must do some comparisons between ACT and SAT with the same students, but are they equally prepared when they take both tests? You can't just take someone who is very prepared for the SAT and have them take the ACT to define the correlation.

The only way for me to judge that is to do the two exams myself. In general, when I look at any math question in the SAT or the ACT, I know that I can get the answer. The question is whether I've seen it before or whether it catches me thinking in the wrong direction. I might not get the correct answer within the time limit. I don't seem to have the "tricky" question issue with the ACT.

This won't matter much for most of the test score range, but when you get to the top level scores, I think it will have an effect, especially when it comes to consistency of scores over a number of practice tests. I would guess that the variation in practice test scores of the top one percent is greater for the SAT than the ACT.

Anonymous said...

For the SAT writing, your 200-800 score is "adjusted" by your essay grade.

Generally, getting 8-9 means your 200-800 score is what you got from the multiple choice section. On some tests an 8 will drop you by 10-20 points or a 9 might boost you by the same. The "break even" point is between 8 and 9, inclusive ;-). A 9 will sometimes get a comparable boost to your score.

If you have the blue book, you can check the big scoring grids for the writing section to see where it was for those first three tests when given.

SteveH said...

Thanks. I just got my Blue Book back and I will be studying it. The scoring sounds odd if you can still get an 800 with a '9' on the essay.

Jen said...

Yup, on Tests 1,2,& 3 of the Blue Book, if you have a raw score of 49/49, you will still get an 800 with a 9/12 essay. 770 or 790 with an 8/12.

I don't really have a problem with an essay for which you read the prompt, think, and write all in 25 minutes (first thing in the early morning) not having a huge effect on your score though.

That kind of a writing sample is just not really very indicative of what you could do with a normal amount of time to think, write, and revise!