Home 
Past Issues November 2005 How NAEP labels a reader ‘proficient’
| How NAEP labels a reader ‘proficient’ |
|
|
|
|
By Susan Ohanian NAEP asserts, “To reach the goal of approximating actual reading experiences, NAEP reading passages are typical of those read by students every day. The passages are taken from authentic texts found in the environments of students in grades 4, 8, and 12. Selected passages are original sources, not simplifications, paraphrases, abridgments, or modernizations.” http://www.nagb.org/pubs /reading_ Framework/ch2.html Selections typical of those read by students every day? Ask yourself which children have subscriptions to Highlights ($26.04 a year) and read selections from William Bennett’s The Moral Compass ($35 new but available from Amazon.com for $.39). Ask yourself about children’s access to these materials: California, to name, one closed school libraries decades ago, and across the nation public libraries, when they can even remain open, are cutting back on hours and book budgets. For documentation on library disparity, take a look at Prof. Stephen Krashen’s research, as discussed on Jim Trelease’s website: http:// www.trelease -on-reading. com/rah _chpt6_p1.html NAEP declares: “The selected passages for each grade are developmentally and topically appropriate. In addition, passages are intact and meet criteria for genre, language, interest, and organization. These criteria include curricular considerations, appropriateness of the topic, style, perspective, and fairness. Documents are genuine and related to tasks that are appropriate for the grade level and experiences of the students being assessed.” http://www. nagb.org /pubs/reading_ Framework/ch2.html Fourth graders may indeed be interested in many things, even crab hunting and wombats. But a government that subjects children to testing that will be trumpeted as indicating the quality of their schools, should surely include a few items that break out of the nostalgia genre, stories and articles closer to their own backgrounds and experiences. One can make the argument that reading about astronauts, crab hunting, wombats, life in the American Colonies, and rural children and their dogs might not be appropriate in topic, interest, perspective, or fairness. For those who think a West African folktale satisfies an ethnic consideration, consider the specialized language and format of such a tale. Teachers do very well teaching this genre and the rest of the nostalgia genre, too, but what is appropriate for a classroom doesn’t give students full opportunity on a test. According to NAEP, “Passages range in difficulty from those that could be read by the least proficient readers (e.g., about second-grade level in a fourth-grade class) to those that could be read by only the most proficient readers (e.g., possibly eighth-grade level in a fourth-grade class), as determined by teachers in specific grades.” So when NAEP issues their proclamations about proficiency levels, 71 percent below proficiency nationwide, where do these eighth-grade level passages fit in? After all, fourth graders not proficient on eight-grade level passages is hardly surprising. But this little detail seems to escape media scrutiny. Always, always we must ask: When we look at NAEP proficiency levels, what are we seeing? NAEP notes that one of their categories, Examining content and structure, “requires critically evaluating, comparing and contrasting, and understanding the effect of such features as irony, humor, and organization.” Indeed. One can ask how many members of the National Assessment Governing Board (NAGB), recognize irony in literature. Or in life, as in their being put in charge of reading assessment for the nation. For fourth graders, the most tested quality is Reading for Literary Experience; 55% of the questions are in this domain, which NAEP describes thusly: “The reader brings his or her experiences and knowledge to the text in such activities as anticipating events, picturing settings, predicting consequences, analyzing actions, and considering the language of literary works. The reader thinks about the perspective of the authors and characters and considers the language and story structure when reading for literary experience. Various types of texts are associated with reading for literary experience, including novels, short stories, poems, plays, legends, biographies, myths, and folktales.” (emphasis added) It seems more than obvious to note that children who vacation at the beach, visit museums, and whose home and public libraries are well supplied with a variety of books will bring more experiences to passages about crab hunting and American colonists. And as schools pursue the relentless path of test prep to satisfy NCLB requirements on state tests, children of poverty are shortchanged even more. They don’t have cultural experiences tests value and measure, their schools rob them of intellectual experiences, and tax dollars no longer support public libraries. NAEP breaks down data on each question in every way you might imagine: sex, region of country, race/ethnicity, public nonpublic (5 categories), national school lunch eligibility, location (central city, urban fringe, rural). So you can find out that more girls than boys correctly answered the questions of where wombats live. Since students in the Northeast, South, and Midwest outscored students in the West on the question Where do wobats live, Standardistos in the West can form a wombat habitat deficit course of study and NCTE and IRA can apply to the Carnegie Foundation for money on wombat-habitat-deficit coaching models. Our professional associations have a lot to answer for. The Passages NOTE: All the released reading passages are online at the NAEP website, but one must access them through the NAEP site. You can assess story texts through the questions site. Once you pull up a question, click on “View Reading Passage.” Here is the question site: http://nces.ed.gov/ nations reportcard /ITMRLS/ searchresults. asp?Num SearchResults =1&Search Subject=Reading&SearchIndex =1&Search StartIndex=1& Questions PerPage= 20&SearchQuestionSet=0& or: http://tinyurl.com/ba426 “How the Brazilian Beetles Got Their Coats” retold by Elsie Eels” (from The Moral Compass, edited and with commentary by William J. Bennett, Simon & Schuster. 1995) offers a small example of how background knowledge and experience informs the text. It would be interesting to ask children to explain the meaning of “set” in this sentence: In Brazil the beetles have such beautifully colored, hard-shelled coats upon their backs that they are sometimes set in pins and necklaces like precious stones… Children of relative affluence are more likely to know about “setting” jewels, particularly when “set” is used differently a few paragraphs later: The bring green and gold parrot set the royal palm tree at the top of the cliff as the goal of the race. And if a fourth grader does understand the meaning, we can guarantee he will be grossed out by the concept. But I admit that for me this set variance was small potatoes compared with green mangoes with golden flushes on their cheeks lying on the ground under the mango trees. I don’t know anything about mangoes and learning they have cheeks stopped me cold in my reading. My husband didn’t know about mango cheeks either, dismissing my question with “You know what they mean.” But neither one of us quite knows what a mango looks like. An Internet search reveals I can buy frozen mango cheeks and cook a coconut, mango, and lychee terrine. I even found a picture of a mango cheek. My point here is that as a reader this term shut down my reading for about 15 minutes. Test takers can’t afford to stop and ponder. NAEP gave “Full comprehension” to just 35% of children answering this constructed response probe: Choose someone you know, have read about, or have seen in a movie or television show who is like the rat or the beetle. Explain how that person or character is like the rat or beetle. Full Comprehension: “I am like the beetle because I am well-mannered and not greedy. My friend is like the rat because she thinks she is the best at everything.” Scorer’s Comment: The student compares him/herself and a friend to both the rat and the beetle, using the appropriate character traits as the basis of comparison. Evidence of little or No Comprehension: “The hare is like the rat.” Scorer’s Comment: Response mentions a character that is like the rat but does not explain the likeness with a text-based characteristic. Now why would this child have chosen a lead character from The Tortoise and the Hare if he had little or no comprehension of the text at hand? Why didn’t he compare the beetle or rat to a character in The Velveteen Bunny or Captain Underpants? NAEP rated 71% of the responses Unacceptable on this question: Do you think that the beetle is being sneaky by not telling the rat she has wings? Use information from the story to explain why or why not? Unacceptable: “Yes I do because she is not telling the rat she has wings.” Scorer’s Commentary: Response provides an opinion but the support merely repeats the question. Sometimes scorers reward students for repeating the question; sometimes they don’t. Note that the student who declares herself well-mannered gets full credit but the one who finds a story heroine sneaky does not. A frequently-used prompt asks readers to suggest a different ending to the story. These two answers for “The River” by Yetti Frenkel received only partial credit: “Minny and/or Cory might have drowned” and “The dog might have drowned,” because, in the scorer’s words: Both responses provide an appropriate opinion of how the story might have ended differently, but neither supports the opinion with evidence from the story. Thus, although children’s answers may exhibit a certain lack of sophistication (or maybe they exhibit the virtue of Robert Frost’s matter-of-fact hard floor), even the scorer acknowledges they don’t exhibit lack of understanding of the reading selection. And so, once again, people viewing NAEP results on the reading proficiency of American fourth graders need to be aware of just what is being tested. It isn’t reading comprehension but the ability to pull off a sometimes sophisticated, sometimes formulaic integration of textual elements into one’s response. Many young responders don’t repeat elements from the text because they take the text for granted. It’s sitting there: why repeat it? A fourth grader who writes “The dog might have drowned” flunks. If he’s written “The dog might have drowned because the kids couldn’t reach him” he gets full credit. Similarly, when the prompt asks Suggest a new and different title for the story and explain why it would be a good title, the student who summarized the events of the story in his title “How the Brazilian Beetle Won the Race” and offered this justification—“because I like it”— received only partial credit because, according to the scorer, the response offers no support for why it would be a good title. Since when is a writer’s gut feeling not allowed? And again, what is being tested? Surely not whether the student understood what he read. It is clear he did. When a student fails to get full credit for this answer, someone has to admit that there’s another agenda here. What’s being tested is not reading comprehension but English major-type ability to supply textual evidence to support right answers. Anyone who teaches knows that children’s writing sophistication lags behind their reading level. NAEP ignores this, choosing to label lack of writing erudition as deficient reading comprehension. The NAEP mantra seems to be: If you can’t write about it in full detail, you don’t know it. When children were asked, Suggest another title for the story [The River]. Use examples from the story to explain why it would be a good title,” only 19% of respondents received full credit. The scorer gave only partial credit for “’Danger on the Ice’ because there is danger on the ice,” complaining that this is a circular explanation. But full credit was given to “The Icy River because the dog got stuck in the icy river.” Go figure. And remember: this kind of arbitrary and capricious judgment determines NAEP proficiency ratings, those ratings broadcast in media headlines across the nation: NAEP Finds 71% of 4th graders score below the proficient level. For New Mexico it was 80%. Does this mean their 4th graders can’t read, or does it mean they weren’t given credit for their alternate titles for “The River?” In another constructed response probe for “The River,” children are asked, “Do you think this story was exciting? Use an example from the story to explain why or why not.” Children who answered “Yes because every good story has some danger” and “Yes, because I like rescue stories” were given only partial credit. Here is the scorer’s reasoning: Both responses provide an opinion and support it with a general reference to the story. The first response claims that the story has some danger, but does not use examples to describe what is dangerous. The second response provides a general reference to the genre of the story that is not explained in terms of story events. Are we looking for pre-trial lawyer skills here or evidence of whether children understood the story? Young readers who write “because every good story has some danger” are taking it as a given that the scorer knows that they know the story has danger in it. Shouldn’t the word because be worth something here? Fourth graders aren’t used to building an evidentiary case to prove their reading savvy, and there are two reasons why schools should not go into the business of training them to do so: 1) It would mean a whole lot less time spent on reading. 2) It would kill the joy of reading for reading’s sake. On Thin Ice Another constructed response probe asked, What was one of the most important lessons that Cory and Elisa learned from their experience? Here are samples of responses that received full credit and those that didn’t: Evidence of Full Comprehension: 1)“Cory and Elisa learned that they must not give up hope and keep on trying.” 2) “One of the most important lessons they learned was that by working together they could save Minnie.” One doesn’t have to read many answers labeled “Evidence of Full Comprehension” to recognize which 10-year-olds have been trained to repeat the question in their answer and which haven’t. Teachers, even knowing this is what the Standardistos want should be very reluctant to deliver it. Teaching in our times means following Henry David Thoreau: Resist much. A surefire method of killing any joy in reading for children is to require them to grind out these answers for everything they read. Too many of the Full Comprehension answerers march along like Stepford byproducts, leader the reader to start checking penmanship: most of the “Full Comprehension” answers sound like they were written by the same person. Teachers need to come to grips with the reality that sometimes “full credit” from the testing marauders comes at too high a price. Evidence of Partial or Surface Comprehension: 1) “Never walk on thin ice.” 2) “Keep an eye on the dog.” Scorer’s Commentary: Both responses identify a superficial lesson that conveys an understanding of a trivial aspect of the story. Trivial aspect of the story! Indeed. For children who live near bodies of water that freeze in winter, the lesson about walking on ice isn’t trivial at all. I must admit I feel close to kids who insist on avoiding all frippery and cutting to the chase. They remind me of my first year of teaching—when on a test I asked students what lesson they learned from Silas Marner. I gave full credit to the student who wrote, “Hide your money in a safe place” feeling she’d put me in my place. And I confess I don’t remember any of the other answers. Standardistos will label such answers naïve and incomplete; the rest of us, recognizing there are many levels of comprehension, must be willing to give credit to students who go for such basics. It would do testocrats well to remember that only in the school world do English major-type answers hold sway. The real world is more pragmatic. And since the world needs pragmatists as well as philosophers, it does us all great harm to label pragmatists as failures. The plot of “The Box in the Barn” by Barbara Eckfeld Conner involves a young boy who lets a surprise puppy for his sister’s birthday out of a box, and the puppy runs off. He’s very worried because he’d been told not to touch the box. Fourth graders are asked, If the box had been empty when Jason opened it at the party, what would Jason most likely have said? Give examples from the story that support your answer. This student response was rated unacceptable: “I think he would say, ‘I didn’t do it.’ That’s what most people would say.” The responder even included quotation marks for the direct quote. Scorer’s Commentary: The response doesn’t use information from the story to predict Jason’s reaction. It provides dialogue that does not demonstrate understanding of Jason’s honest character. The fact is that the reader has gone beyond the story to make an observation about human nature. The scorer’s reaction seems especially bizarre when one considers another response that received full credit: “Maybe he might have just kept what he had done a secret. Because he had been trying to act normal.” Here the scorer says, The response use information from the story to predict how Jason might have reacted if the box had been empty. The response predicts that Jason might have kept it a secret, based on the part of the story when he pretended to act like everything was fine when the party began. Can we tell from these scorer judgments that one child is a more proficient reader than the other? Again, NAEP’s declarations about non-proficiency are worse than suspect. Another consideration must be whether the children don’t feel oppressed by the incessant writing tasks. This quite insipid story, 1029 words long, is weighed down by 12 questions, 7 of which require constructed responses providing evidence from the story: • Why did Jason think everyone would be angry with him when they found the puppy missing? • If the box had been empty when Jason opened it at the party, what would Jason most likely have said? Give examples from the story that support your answer. • Why do you think Jason’s father explained how he found the puppy? • Describe how Jason might have felt if the box had been empty when it was opened at the party. Explain why he might have felt that way? • Why did Jason not answer when his mom called him for dinner? • From when Jason got up in the morning until he went to bed that night, his feelings changed as different things happened. Describe three different feelings that Jason had and explain what made him have those feelings. • What did Jason’s father think that Jason had learned? Tell what in the story makes you think so. Whew! Someone take pity on these fourth graders required to deconstruct so many motives and responses to one limp story. Of Fares and Fishes: Grade 8 Reading Passages It is troubling to note how formulaic the answers receiving Full Comphrension credit are. Of course it’s too much to ask that a respondent’s ‘voice’ come through on a test, but one has visions of lots of test prep monotonous filling in of expected responses. Students who draw on their own personal experience are penalized for not referring to the poem. In some cases answers labeled “No Comprehension” might actually be indicative of a deep, personal response to the poem. One wouldn’t know without talking to the students. Talking with children about poetry usually raises more questions than answers, but NAEP is looking only for answers that fit their formula. Full Comprehension: “I liked the poem because the poet, Elizabeth Bishop, beautifully and accurately described the catching and letting go of a fish.” Partial Comphrehension: “The fish wanted to get let go and the boy saw the rainbow and let him go.” Little or No Comprehension: 1) “I did not like it because it was not interesting.” 2) 2. Which of the following best describes the person speaking in the poem? 3. Why does the person let the fish go? What in the poem makes you think so? Full comprehension: “I think he let it go because it was tired and old and it was caught many many times. What made me think that was the five hooks in its mouth and the fact it didn’t fight.” Little or no comprehension: 1) “I think she let it go because she’s an animal lover, and she’s just one of those people who won’t eat fish when they catch them or like to fish just for fun.” 2) “I think she let him go because he was making everything colors of the rainbow. 4. Choose an image from the poem and explain what it means. Little or no comprehension: “The rainbow means it is very colorful.” 5. Explain how the language used in this poem is different from the language used in everyday speech. Use a line or phrase from the poem as an example. Full Comprehension: “The dramatic reds and blacks from his shiny entrails. He uses a lot of adjectives.” Partial Comprehension: “Fresh and crisp with blood. Very different from everyday speech. Most people would of said, ‘Very slimy and bloody.’ (quotes in original) 8. Reread the lines beginning with “I admred” (line 45) and ending with “aching jaw” (line 64). What do these lines tell you about the fish’s experience? Partial: 1) “This tells me the fish was in pain and he wasn’t very happy.” 2) “That it has got caught a lot of times.” Little or no Comprehension: 1) “She thought proudly of the fish. She knew how it felt to have an aching jaw.” 2) “the hooks.” 9. Describe how the speaker’s decision to let the fish go is related to something you have experienced, red, or seen. Full Comprehension: 1) “I have seen this reaction before in many inspirational movies and I have done so too by letting a captured butterfly fly freely.” 2) “This relates to when I caught a lightning bug and felt sorry for it and let it go.” How Did This Happen? |
| < Prev | Next > |
|---|