Skip to content Skip to navigation

Where Completion Goes Awry: The Metrics for “Success” Mask Mounting Problems with Quality

Tomorrow's Academy

Message Number: 

What the AAC&U board said then, however, remains true today: the intense national commitment to increased college attainment needs to be matched by an equally intense focus on quality or, more specifically, on the kinds and levels of learning that degree attainment is supposed to represent. Completion ought reliably to mean that students have demonstrated-cumulatively, over time-their acquisition of the knowledge and skills they will need for the complex and fast-changing challenges of work, citizenship, and contemporary life.



The posting below looks at the need to strive both for increased college completion rates and high quality and productivity. It is by Carol Geary Schneider, president of Association of American Colleges and Universities, and is from the Winter 2012, Vol. 98, No. 1 issue of their publication, Liberal Education Copyright © 2012 Association of American Colleges and Universities, 1818 R Street NW, Washington, DC 20009. All rights reserved, reprinted with permission.


Rick Reis

UP NEXT: Demonstrating Institutional Effectiveness



Tomorrow's Academia

---------------------------------- 2,296 words --------------------------------

Where Completion Goes Awry: The Metrics for \"Success\" Mask Mounting Problems with Quality


While the United States is still among the top five most educated nations internationally, it has slipped to fifteenth in terms of the rate of college completion for Americans aged twenty-four to thirty-five. This unhappy development-so at odds with our history of international leadership both in college-going and in world-class postsecondary institutions-has rightly been taken as a wake-up call. With vigorous leadership from policy centers and major philanthropies, educators now are intensely focused on reversing this downward trend. Completion and productivity initiatives are cascading, and new performance incentives for improved degree production are being unveiled in one state system after another.

Our nation's future does indeed depend on developing all Americans' talents to the fullest extent possible, and the Association of American Colleges and Universities (AAC&U) has been strongly committed to that goal since we first launched our Greater Expectations initiative (the forerunner of our current Liberal Education and America's Promise, or LEAP, endeavors) over a decade ago. In the opening pages of the 2002 Greater Expectations report, we noted the evidence that college has become a \"revolving door\" for entirely too many students, at high cost to their hopes for the future and to society's need for a well-educated citizenry. In that spirit, we have welcomed the intensified focus on student success and completion, which AAC&U's board of directors embraced in an official statement released in 2010.

What the AAC&U board said then, however, remains true today: the intense national commitment to increased college attainment needs to be matched by an equally intense focus on quality or, more specifically, on the kinds and levels of learning that degree attainment is supposed to represent. Completion ought reliably to mean that students have demonstrated-cumulatively, over time-their acquisition of the knowledge and skills they will need for the complex and fast-changing challenges of work, citizenship, and contemporary life.

Unfortunately, however, the completion agenda is steaming ahead without setting either goals or markers for educational quality. As the authors in this issue of Liberal Education make patently clear, when we create incentive systems for enhanced degree production, with no questions asked about the sufficiency of learning, the door is literally wide open to choices that deplete rather than build educational quality. The articles on completion we share in these pages are thoughtful-and sobering. One foundation is already sharing Debra Humphrey's lead article with its productivity grantees. I hope this issue will open a candid dialogue and invite a much needed policy \"do-over\" designed to make quality the driver for completion and productivity.

As national studies on \"high-impact practices\" make clear, higher education already has invented a raft of learning-intensive programs and pedagogies that, the evidence shows, can simultaneously lift both completion rates and student achievement of key skills such as writing, research, and analytical reasoning. There are well-tested models for deploying technology not for course delivery, but in support of high-order student inquiry and achievement. The question is whether we're willing to use the knowledge we already have to create incentives for high-quality learning as well as incentives for increased degree production. It shouldn't be an either/or choice; we can do both.

In what follows, I offer my own observations on where the completion and productivity agendas have gone awry, and on how we can actually achieve those greater expectations.

The credit hour as a proxy for quality

The term \"quality\" is scarcely even mentioned in most of the abundant policy materials that frame the nation's current efforts on completion and productivity. Even when invoked, quality is never defined. Policy statements at all levels promise solemnly that costs can be cut and graduation rates raised \"with no compromise of quality,\" of course. But absent meaningful reference points-or incentives-for the demonstrated achievement of quality, warnings against its erosion are empty at best. The real message seems to be \"more degrees, cheaper and faster\" with no questions asked about what the degree represents.

Absent meaningful markers for quality, the real proxy for progress on completion and productivity has become, de facto, that hardy but hoary invention of the early twentieth century, the course credit hour. Established over a century ago, in the heyday of the industrial assembly line, and given a significant boost forward when Carnegie Foundation for the Advancement of Teaching promoted it, the credit hour signals the purported combination of students' time spent in class plus the time spent outside of class learning course-aligned material.

Freshened up and sometimes disaggregated (by student subgroup and/or income), the credit hour is now the marker of choice for reporting progress on persistence, retention to the next level, transfer of courses from one institution to another, and, of course, degree attainment itself. It is our de facto proxy for quality. But as a proxy, it both protects and disguises subprime performance.

Employers are pleading with colleges and universities to build higher levels of American capability. Yet, credit hour production tells us only about \"efficient through-put\" and, when used in productivity analysis, the comparative costs of different degrees. Credit hours tell us absolutely nothing about what students are even doing in a course, much less about their levels of achievement. Does the course require extensive writing? Research? Projects? Team problem solving? Applied learning? Or just coasting and exam-time cramming? The federal government has recently blown the whistle on courses that meet too briefly for the credit hours awarded. But frankly, this is a small part of a much larger problem.

The credit hour is equitably awarded for high-impact learning and low-impact activity alike. It is our coin of exchange. When it comes to actual quality, however, the credit hour effectively hides rather than reveals. And yet, incentives for increased credit production-at lower cost-have become the strategy of choice for making America once again the \"first in the world.\"

The mounting evidence on the quality shortfall

As this association has insisted for a decade-and as this issue of Liberal Education insists once again-the real key to economic opportunity and advancement depends not on whether the student possesses a credential, but rather on whether students actually leave college with that rich portfolio of learning that employers seek and society urgently needs: broad knowledge, strong intellectual and practical skills, grounded commitments to personal and social responsibility, and demonstrated capacity to deal with complex challenges.

The facts are stark. As ACT annually reports, only one in four students who enroll in college is well prepared to be there. Most need help to recover from an inadequate foundation-in math and science, history, global cultures, languages, critical thinking, writing and problem solving, even in basic civics-from a school curriculum that is simply not strong enough. Or, enrolling students have been out of school for a while and need to strengthen basic skills that they may not have used in years. Or, the college students may be second-language learners and need a lot of support to make the transition to college-level learning in their new tongue.

While we would like to believe that college helps all these short-changed learners get up to speed, there is overwhelming evidence that many college seniors graduate still lacking essential capacities that they need for work, for citizenship, and for lives of continuous learning. These graduates-\"successful\" in their acquisition of the right number of course credits-possess credentials, but they have not achieved the kinds of learning that a degree should represent.

Not long ago, Margaret Spellings and entire platoons of policy leaders all were urging higher education to take responsibility for assessing students' actual learning in college. And, in fact, higher education has been doing that assessment. There now is a raft of highly creditable studies-including a synthesis of research on learning compiled by former Harvard President Derek Bok-probing different aspects of students' learning in college. Later this year, AAC&U will publish its own synthesis of \"what is known\" about students' progress on the essential learning outcomes that we have emphasized through our LEAP initiative.

What do these studies actually tell us? Colleges are \"underachieving,\" says Derek Bok. Students are making small gains or even declining on several key measures of learning outcomes, say the scholars doing a longitudinal study of student achievement through the Wabash Center of Inquiry in the Liberal Arts. In 2006 and 2007, 14,000 seniors earned an average grade of F in a test of basic civic and historical knowledge, reports the Intercollegiate Studies Institute. Graduates lack global knowledge and are weak on key competencies such as writing and critical thinking, say employers surveyed by Hart Research Associates for AAC&U. There have been few documented gains in the level of student learning since our 2005 report on Liberal Education Outcomes, AAC&U's forthcoming synthesis of the evidence concludes.

The \"preponderance of the evidence,\" as AAC&U's own director of assessment puts it, shows us that many students do indeed complete college without actually achieving the high-level capacities and complex knowledge that a liberal-and liberating-education ought to provide.

The completion and productivity juggernaut pays no attention to any of this. At best, it kindly assumes that if students gather the right number of course credits, \"success\" has been achieved by definition. At worst, productivity enthusiasts are looking for ways to accelerate \"degree production\" by focusing on narrow, technical training credentials tied to specific \"labor market signals and needs,\" while paring back or eliminating entirely the broader education and higher-order intellectual skills that have long been the key to our world leadership in postsecondary education. This may help us increase through-put, but it perversely defeats the entire project of helping Americans achieve higher levels of knowledge and capability.

What can be done to provide twenty-first-century quality markers?

While there is good reason to sound the alarm bells about both the existing quality shortfall and the policies likely to worsen it, there also is reason for hope. Just a year ago, the Lumina Foundation released in beta form-for widespread experimentation and testing-the Degree Qualifications Profile (DQP), a twenty-first-century quality framework that is intended to provide the quality markers higher education so urgently needs. Today, the DQP is being tested and evaluated in a suite of Lumina-supported projects organized by national associations, including AAC&U, and by some of the accrediting associations. Over one hundred highly diverse colleges, community colleges, and universities-public and private-already are involved in testing the DQP for a broad array of purposes, from curriculum design to assessments of students' cumulative learning to transfer. Other campuses are \"shadowing\" the grant-funded projects, using the DQP for their own purposes. The new CUNY community college, for example, has used the DQP to design its entire curriculum and, in fact, has significantly improved on the source document.

As one of the authors of the draft DQP, I am truly pleased that Lumina is inviting the entire educational community to become part of a meaningful discussion about the right reference points for quality. I am also very pleased that the current content of the DQP reflects the views of the AAC&U community about the essential aims and learning outcomes-or, in DQP terms, competencies-needed for success in the economy and to contribute to a flourishing democracy. The DQP emphasizes the importance of broad as well as specialized learning for degree earners at the associate's, bachelor's, and master's levels. It calls for students to develop intellectual skills that faculty value and employers seek. It makes preparation for democracy-civic learning-an expected component of all degree programs. It calls for students to integrate their learning across disciplines, in general education and majors, and to apply that learning to complex, unscripted questions. And it argues that the actual test of a twenty-first-century education is whether students can apply their learning to new settings and complex problems. This is liberal education in contemporary form, albeit now a \"rose by another name.\"

Many worry almost reflexively that a framework of this sort will be constraining. But, in fact, the framing of this document is intended to apply to a wide array of academic content and curricula. At the heart of the vision of the DQP is an insistence that once we know what students are supposed to accomplish, there are a multitude of ways to develop and certify their achievement.

But how is DQP achievement actually certified? What I personally like best about the DQP is that it builds directly from the evidence-based research on the value of high-effort, high-engagement practices in fostering student achievement and supporting student persistence in college. If you read the DQP carefully, you'll see that, again and again, it places students' effortful work at the very center of the assessment equation. Projects, research, writing, performances, portfolios-course-based and field-based-are the centerpiece of DQP assessment. The idea is simple. As a group, faculty plan the kind of robust assignments and tasks that will enable students to develop knowledge and competency in their fields. Faculty then evaluate, at milestone moments, whether students are demonstrating the expected level of learning. Our own Lumina project will experiment, across nine state systems and at two-year and four-year institutions, with effective ways to document graduation-level achievement and ways to report it externally that are accessible, transparent, and persuasive.

How would we tie these DQP innovations back to the completion agenda? I believe the answer is to create incentives both for students' timely completion and for evidence of students' demonstrated achievement in relation to Degree Qualifications competencies. We already know that students' engagement in high-effort, high-impact work helps keep them in college. If we can link demonstrated DQP achievement-anchored in exactly the kind of effortful student work that supports both persistence and learning-to the new completion and productivity incentives, we will produce the completion agenda \"do-over\" higher education urgently needs.