Facebook
Edutopia on Facebook
Twitter
Edutopia on Twitter
Google+
Edutopia on Google+
Pinterest
Edutopia on Pinterest Follow Me on Pinterest
WHAT WORKS IN EDUCATION The George Lucas Educational Foundation

The Challenge of Authentic Assessment

Problem: Old-school accountability tests are crude measurements of student learning.
Solution: Build a better test.
Grace Rubenstein
Former senior producer at Edutopia
Related Tags: Assessment,All Grades

Past:

Traditional standardized tests deaden teaching and inaccurately measure student learning.

Credit: Gregory Cherin

When I was a younger education reporter in the old mill town of Lawrence, Massachusetts, the big day came when the state released scores on its school accountability tests. The Massachusetts Comprehensive Assessment System, better known and feared as the MCAS, fulfills the requirements of the federal No Child Left Behind Act through annual tests in English and math (and now additional subjects).

I scrutinized pages of numbers and wrote a story on the success and failure of nearby schools. My editors played it big on the front page because they knew parents would look anxiously at their school's results and homeowners would mentally adjust their property values based on the scores. I prodded principals and superintendents to explain their schools' leaps or stumbles.

And unwittingly, I played right into the dominant illusion that these bloodless test scores are the most definitive measure of a school's success -- and that they measure what's most important.

Cold, hard numbers have a way of seeming authoritative, but accountability tests are not the infallible and insightful report cards we (and our state governments) imagine them to be. The educational assessment tests states use today have two fundamental flaws: They encourage the sort of mind-numbing drill-and-kill teaching educators (and students) despise, and, just as important, they don't tell us much about the quality of student learning.

"We are totally for accountability, but we've got the wrong metrics," says John Bransford, a professor of education at Seattle's University of Washington who studies learning and designs assessments. "These tests are the biggest bottleneck to education reform."

Hobbled by History

Jennifer Simone, a fifth-grade teacher at Deerfield Elementary School, in Edgewood, Maryland, is acutely aware of the limitations of standardized tests. Her curriculum must emphasize subjects for which the state accountability test measures proficiency -- math, reading, and science. Social studies? Though the subject is on her master schedule, if there is a shortened school day, it gets dropped.

Moreover, Simone says, the test scores don't truly reflect her students' abilities and are too vague to help her pinpoint individual needs. She longs for an assessment that relies on more than just written problems, that could capture the more diverse skills visible in her classroom and valued in the workplace, such as artistic talent, computer savvy, and the know-how to diagnose and fix problems with mechanical devices. Simone asks, "If we differentiate our instruction to meet the needs of all the learners, why aren't we differentiating the test?"

Future:

Desiree Jerome demonstrates a chemiluminescent reaction as part of her portfolio to move up to the next grade at F.W. Parker Essential School, in Devens, Massachusetts.

Credit: Gregory Cherin

The simple, but unsatisfying, answer is history and efficiency. The tests that states use to satisfy NCLB descended from a model created in the 1920s designed to divide students into ability groups for more efficient tracking. Eighty years, a world war, and a technological revolution (or two) later, the tests remain structurally the same.

Policy makers revere the seeming objectivity of these tests, but the truth is that the exams are not adept at determining either how well teachers have taught or students have learned -- and test makers themselves will tell you so. Stephen Dunbar, an author of the influential Iowa Test of Basic Skills, explains that these tests can help illuminate statewide educational trends but are too broad a brush for the detail at the school and classroom level that NCLB demands.

Assessment tests might show the overall effectiveness of the ninth-grade curriculum, for instance, or indicate trends within large demographic groups in that grade. But Dunbar says that when you get down to measuring the ability of students at Dallas's Woodrow Wilson High School, for example, where you're comparing this year's ninth graders to last year's, accountability test scores are not very useful. "They might tell you more about idiosyncrasies in that combination of kids than the level of achievement or the quality of teaching and learning that's going on," Dunbar explains.

In other words, state governments, at the behest of the feds, are using tests to measure something they actually don't measure very well, and then penalizing schools -- and in some cases, denying students diplomas -- based on the results.

"Most of these policy makers are dirt ignorant regarding what these tests should and should not be used for," W. James Popham, professor emeritus at the University of California at Los Angeles and former president of the American Educational Research Association, told PBS's Frontline in 2001. "And the tragedy is that they set up a system in which the primary indicator of educational quality is simply wrong." (See Popham's article, "F for Assessment," April 2005.)

There are several reasons the tests are imprecise. (See "Where Standardized Tests Fail.") Some are technical: an ambiguous question, a misjudgment in setting the difficulty level, a scoring error. The National Board on Educational Testing and Public Policy, at Boston College, has documented cases when scoring errors sentenced children to summer school or caused them to miss graduation before the mistakes were discovered. Some reasons are personal: Simone, whose school narrowly dodged state intervention last year, has seen fifth graders arrive on testing day angry about personal matters; others struggled to sit still during the test or broke down in tears under the pressure.

The tests' fallibility has most to do with the very idea of measuring a year's worth of learning in a single exam. Inevitably, cramming that much coverage into a short test leads states to rely mostly on multiple-choice questions -- the fastest and cheapest means of large-scale assessment. Such brief yet weighty exams limit the ways students can show their skills, and because it's impossible to test hundreds of state standards in a few hours, they leave teachers guessing on which to emphasize. Randy Bennett, who holds the title of distinguished scientist at ETS, writes that this rigid idea of assessment yields a "narrow view of proficiency" defined by "skills needed to succeed on relatively short, and quite artificial, items."

Even when states do pony up to use open-ended essay questions and pay human scorers, these questions can encourage formulaic answers. Last school year, I watched the principal of a (high-scoring) Boston high school interrupt a test-prep session to warn students not to stray from the essay-writing formula -- main idea, evidence, analysis, linking -- lest they lose points. "Don't be creative," she said fiercely. "You've heard me rail against standardized tests, and this is why. There's one way to do this, and it's the way the assessment coordinator told you."

Equally worrisome is that today's assessments emphasize narrow skill sets such as geometry and grammar, and omit huge chunks of what educators and business leaders say is essential for modern students to learn: creative thinking, problem solving, cooperative teamwork, technological literacy, and self-direction. Yet because NCLB has made accountability tests the tail that wags the dog of the whole education system -- threatening remediation and state takeover for schools that fall short -- what's not tested often isn't taught.

In short, the American accountability system is a bastion of the past that's stifling our ability to tackle the future.

High Stakes

The good news is there's work afoot to create better tests that will challenge students to demonstrate more creative, adaptable skills -- and, in turn, encourage teachers to teach them. Some model assessments already exist; for instance, many experts tout the Programme for International Student Assessment (PISA) exam for its challenging, open-ended questions on practical topics, such as climate change or the pros and cons of graffiti. Even more advanced models, some using computer simulations, will become available in a few years -- and none too soon.

Business leaders have issued dire warnings about how hard the U.S. economy will tank if our education system doesn't get itself out of the nineteenth century, and fast. They're clamoring for creative, productive, affable employees -- not just dutiful test takers -- and they point to assessment as a crucial tool for turning the tide. Microsoft founder Bill Gates, addressing state governors, CEOs, and educators at the National Education Summit on High Schools in 2005, said, "America's high schools are obsolete. Even when they're working exactly as designed, they cannot teach our kids what they need to know today. In the international competition to have the biggest and best supply of knowledge workers, America is falling behind."

The New Commission on the Skills of the American Workforce, convened by the nonprofit National Center on Education and the Economy, issued a stark report in December 2006 predicting that our standard of living "will steadily fall" compared to other nations unless we change course. The globalized economy has created, the commission wrote, "a world in which comfort with ideas and abstractions is the passport to a good job"; what's essential, it added, is "a deep vein of creativity that is constantly renewing itself." According to the report, whatever efforts we make to modernize education, without a complete overhaul of the testing system, "nothing else will matter."

Congressman George Miller, chairman of the House Education and Labor Committee and chief House wrangler of NCLB (and a member of The George Lucas Educational Foundation's Advisory Board), understands the problem. The original law left it up to states to choose their own tests, but now he believes most states picked tests more for cost and efficiency than for educational value. "They don't truly measure what a student knows or doesn't know," he says, "or whether students have a depth of understanding so that they can apply their knowledge."

Real Solutions to Real Problems

In the past, states haven't had much choice in the kinds of large-scale assessments available, nor have they asked for much. That's about to change.

Test makers in multiple corners are creating more complex assessments, ones that, if tied more closely to curriculum and instruction, could paint a clearer picture of student learning. They're building these assessments to measure the twenty-first-century skills we so urgently need, aiming to gauge a child's readiness for the real challenges that await. If tests like these succeed, they could not only provide better information about children's readiness for real life but also give educators incentive to do what they want to do anyway: teach kids in engaging ways to be well-rounded people and lifelong learners, not drill the life out of school with dry test preparation.

A number of researchers are building tests that could be models -- or at least one piece of a larger model. The University of Washington's John Bransford and Andreas Schleicher, head of the Indicators and Analysis Division at the Organisation for Economic Cooperation and Development (OECD), maker of the PISA exam, believe students need dynamic problems to solve, ones that require real-world research and allow them to learn on the spot, not just apply prior knowledge.

A static problem, for instance, would ask test takers to say from memory how to save a certain endangered bird species. A dynamic assessment (in a real example from Bransford's lab) asks students to use available resources to learn what it would take to prevent the white-eyed vireo from becoming endangered. This is a novel question that demands students independently dig for information and know enough to ask the right questions to reach a solution.

Bransford says he doesn't believe the old trope that students must master a battery of content-specific facts before they can have a prayer of learning higher-order skills. "Just the opposite," he says: Students need to understand big concepts in each discipline, such as the relationship between a species' life cycle and its risk of extinction, but from there it's the higher-order skills that lead them to the pertinent facts.

At ETS -- which writes the SAT and Advanced Placement exams, among others, and administers fifty million tests a year -- Randy Bennett is field-testing assessments that make use of about thirty years of psychology research on how children learn. It's research that he says has been largely left out of test design. The key strategies he has found include asking students to integrate multiple skills (such as reading and making comparisons) at once, presenting questions in meaningful contexts, and using a variety of information forms, such as text, diagrams, and symbols. Eva Baker, codirector of UCLA's National Center for Research on Evaluation, Standards, and Student Testing, proposes one more: Never have someone present a solution without explaining why he or she chose it.

It's not so different from the kind of assessment Jennifer Simone would like for her students. She'd like the exam to use more formats than just writing, including visual or spoken components. "You would have to take the time to have a student interview, allow students to have an oral response," she says. "That's how we teach them reading."

Technology is what will make this revolution possible. Already, computers have enabled Bransford, Baker, and others to create interactive questions, search environments where students can find new information, and simulations to make problems more engaging and real. These tools can record students' answers as well as their thought process: what kind of information they sought, how long they spent on each Web page, and where they might have gone off track.

The British government has created a computer-literacy test that challenges teens to solve realistic problems (how to control crowds at a soccer match, for instance) using online resources. The more sophisticated these tools become, and the more adeptly test makers use them, the better assessment will be.

So, progress is coming -- in some cases, has arrived -- but as the OECD's Andreas Schleicher says, "It's a long road, and we're at the beginning." The biggest hurdles are time and money (richer tests require more of both to design and administer), and that rarely tamable beast, politics. The next version of NCLB, due later this year, could pump federal money into pilot projects to help states create richer assessments, paired with richer curriculum -- but only if that clause survives the political battle to come.

Stephen Dunbar, the Iowa test author, has doubts that more complex tests can be done on a large scale. Though the effort is worthy, he says, the cost and time to create and score open-ended questions, and make them comparable from year to year, could make it too impractical. Scary as it might sound, artificial intelligence is likely to play a big role in the scoring of such exams. If the technology becomes sophisticated enough to handle answers to trickier problems, it could make better assessment more affordable.

The ETS's Randy Bennett, on the other hand, believes the prospects of building an assessment system to match the demands of the twenty-first century are "pretty good." The key is to convince states that it's practical, affordable, and clearly better than today's exams at providing meaningful information. At least one state, West Virginia, has begun asking the test makers it contracts to emphasize more modern problems and skills. Another hurdle will be for politicians to temper their devotion to multiple-choice questions and get comfortable with a little subjectivity. "For any assessment," Schleicher says, "you have to make a trade-off between objectivity and relevance."

Jennifer Simone, for one, is depending on forward-thinking test makers and policy makers to succeed -- for the sake of her students, most of all. "That we are held accountable is a good thing. That we are doing something to measure the progress of our students is a good thing," she says. "I just disagree with the way it's being done."

Grace Rubenstein is a senior producer at Edutopia.

Comments (8)Sign in or register to postSubscribe to comments via RSS

Harry, Los Angeles's picture
Anonymous (not verified)

Having just one test at the end is a strategy for failure.

What's wrong with frequent tests? As long as the questions build on a body of knowledge, you'll have the same evaluative results as the single summative test. Furthermore, schools and teachers can pinpoint learning problems with the class and/or the student and apply remediation when it's most useful.

Marion Brady's picture
Anonymous (not verified)

It grieves me that so much faith is placed in machine-scored tests, and so little in the opinions of educated professionals with myriad hours of direct contact and interaction with learners.
Even more-sophisticated and reliable assessments would be possible if stable groups of students were taught by teams.

Folwell Dunbar's picture
Anonymous (not verified)

Here in my home state of Louisiana, LEAP and ITBS are the standard for standardized testing in K-12 education. The tests cover the BIG three traditional R's, "cultural literacy" in core subject areas, and, supposedly, critical thinking. They consist of reams of "multiple guess" questions and a handful of written responses. What's interesting about LEAP and Iowa is not what they test, but rather what they don't. When was the last time you went into a job interview and were asked to fill in bubbles using a number 2 pencil? Have you ever tried to make a sale by writing a short essay on an arbitrary prompt? In a brainstorming session, have you ever been limited to only one of four ideas. For some strange reason, the folks in Des Moines and Princeton (not to mention the State Department) don't feel that listening, speaking, and creativity are worthy enough for the new accountability. In today's changing economy, an ability to listen attentively, think out of the box, and speak "like, you know, good" are more important than ever. Unfortunately, as teachers scramble to prepare their students for the end all and be all Test, these invaluable but untested skills are being passed over.

Frank Krasicki's picture
Anonymous (not verified)

Build a better test? Why bother?

The fraudulent idea that schools should be measured and punished for not achieving NCLB standards is intellectually silly, political pandering to the neo-con right, and degrading to the generations of teachers and children who suffer under this misosophy.

Schools must relearn what children are, how they learn, and how they grow a soul - all the things abandoned for the past thirty years to entertain the madness of the standardized testing regimes who control federal funding with and iron fist and an army of bureaucratic Hitlers to enforce their national blight.

The revolutions confronting educators will never again be so trivial as tweaking the bad ideas of the last century. The uniqueness of every child and every child's gift must once again become a national priority.

These children may outlive our generation by hundreds of years, their medicine will astonish us, the integration of ubiquitous technology to everyday reality will will not only reinvent learning it will reinvent lifestyle.

You can test till you all turn blue but I can assure you you are doing little more than killing time.

- Frank Krasicki
http://region19.blogspot.com

Jim Carlson's picture
Anonymous (not verified)

This test comes out of the University of Iowa in Iowa City, not Des Moines (home of Iowa State University).

Anonymous's picture
Anonymous (not verified)

Although true that the ITBS does not come out of Des Moines, neither does Iowa State University (Ames)

Michael Griffin's picture
Anonymous (not verified)

Such heavy reliance on assessment systems will stifle creative and individual learning. Teachers teach to test and resist tangental exploration on the basis that 'its not in the syllabus'.
As Einstein said: "Not everything that counts can be counted, and not everything that can be counted, counts".
My music education materials explore involving students in self and peer evaluation leading to meta gognition and greater self awareness. See the website above for more details or feel free to contact me.
Michael Griffin

Patrick Moran's picture
Anonymous (not verified)

In many fields, American authorities are strong on specifying desired outcomes but weak on nurturing the means to achieve those outcomes. A corollary of this fact is that examinations are often used to assign blame.

Instructors will benefit their students by structuring the relationship between student and tests of any category before that kind of test is administered. Students need to know that some examinations are diagnostic (they inform the teacher where each student stands at the beginning of a course), weekly quizzes and tests of that general kind are primarily to inform students of their recent rate of progress toward achieving competencies that will be of value to them in the future, and final examinations that are primarily of value to the school for another part of its job -- certifying the level of competence of students to the outside world. After students have been socialized under the currently prevailing conditions for more than a decade, it may be very difficult to disabuse them of the idea of a college education is to get grades and then a diploma.

Intrinsic rewards which follow as a natural consequence of acquiring knowledge and competency can be vitiated by primary reliance on extrinsic rewards and punishments. Deadly dullness follows, and recovery of natural initiative and joy in learning will rarely be a short-term result of release from these noxious influences.

Education should not kill the joy of learning.

blog A New Era for Student Assessment

Last comment 6 days 21 hours ago in Performance Assessment

Discussion Completion Grading for Homework

Last comment 1 week 16 hours ago in Assessment

Discussion Teachers Throwing Out Grades

Last comment 1 week 10 hours ago in Assessment

blog Debunking Homework Myths

Last comment 1 week 10 hours ago in Assessment

Discussion What do we mean by effective marking? And how do we keep this manageable?

Last comment 3 weeks 2 days ago in Assessment

Sign in and Join the Discussion! Not a member? Register to join the discussion.