Measure Student Learning for Outcomes that Matter in Work and Life

Posted by Association of American Colleges and Universities on March 6, 2017

Not the Usual Standardized Test: VALUE Approach Shows It’s Possible to Meaningfully Measure Student Learning for Outcomes that Matter in Work and Life

New Report Describes Outcomes, Prospects for Taking Approach to Scale

A groundbreaking approach to assessing student learning created and spearheaded by the Association of American Colleges & Universities (AAC&U) shows it is possible to evaluate undergraduate students’ achievement without relying on standardized tests and by using existing material. In On Solid Ground, a report released today, AAC&U shares the results from the first two years of data collection for the VALUE (Valid Assessment of Learning in Undergraduate Education) initiative, a nationwide project that examines direct evidence of student learning. It represents the first attempt to reveal the landscape of student performance on key learning outcomes—Critical Thinking, Written Communication, and Quantitative Literacy—that educators, employers, and policy makers agree are essential for student success in the workplace and in life.

Key findings include:

• The strongest student performance was in Written Communication. The results support the effect that institutional efforts focused on improving student writing over the last few decades seem to have had on writing proficiency, although the effective use of evidence to support written arguments in various contexts or genres continues to be a challenge.
• In the area of Critical Thinking, students demonstrate strength in explaining issues and presenting evidence related to the issues. However, students have greater difficulty in drawing conclusions or making sense out of or explaining the importance of the issue studied.
• Students demonstrate strengths in calculation and interpretation in Quantitative Literacy, while showing weaker performance levels in assumptions and application of their knowledge. The results suggest that more emphasis has been placed on the mechanics of math-related skills and less attention on the “why, when, and where” of using quantitative approaches.

“This project represents the first attempt to develop a large-scale model for assessing student achievement across institutions that goes beyond testing,” said Lynn Pasquerella, President of AAC&U. “It does so by relying on assignments that students have completed throughout their classes and across disciplines. The results are encouraging, and AAC&U and our partners regard the VALUE initiative as offering promise for improving student learning while addressing issues of quality and equity in undergraduate education.”

The VALUE initiative is the result of several years of collaboration with the State Higher Education Executive Officers association (SHEEO) and the Multi-State Collaborative to Advance Quality Student Learning (MSC), the Minnesota Collaborative, the Great Lakes Colleges Association (GLCA) Collaborative, and nearly one hundred public and private, two- and four-year colleges and universities. Over the course of the initiative’s first two years of data collection, faculty from participating institutions uploaded more than 21,189 student work products to a web-based platform. More than 288 specially-trained higher education professionals from a range of disciplines then scored the student work on a scale of four to zero. A third of students’ work was scored twice to ensure consistency.

“In a world awash in data, VALUE generates evidence that points to what is working well and, critically, to where there is room for improvement,” said Terrel Rhodes, Vice President for Quality, Curriculum, and Assessment and Executive Director of VALUE at AAC&U. “It empowers faculty to take ownership of student learning and challenges them to evaluate their teaching practices to help students master the knowledge, skills, and abilities they need to be productive members of our workforce and our society.”

By design, the VALUE approach addresses the inherent complexity of the learning process by embracing the multiple moving parts that standardized tests and other assessment approaches try to control or eliminate. It includes: (1) the VALUE rubrics that provide the mechanism for scoring student work; (2) the faculty trained as scorers who use their expert judgment to evaluate student work products and assign a score based on the rubric dimensions and performance levels; and (3) the student work products generated in response to a faculty-designed assignment from an actual college course.

Released by AAC&U in 2009, the VALUE rubrics are the backbone of the VALUE approach. Developed to assess students’ most motivated, best work, the VALUE rubrics help faculty evaluators assess the level of proficiency represented in a student work product (paper, performance, community service project, etc.) done in real classes.

http://www.aacu.org/value-rubrics

“The VALUE project is a bold, ambitious, unprecedented effort to demonstrate the viability of a national assessment system that documents authentic student learning using a consistent process at multiple colleges and universities,” said George D. Kuh, Founding Director of the National Institute for Learning Outcomes Assessment, Adjunct Professor at the University of Illinois, and the Chancellor’s Professor Emeritus at Indiana University.

Establishing the methodological soundness of the VALUE approach was and remains a key priority. The report details initial work on validity and reliability, further demonstrating that the VALUE approach to assessment stands on solid methodological ground. Perhaps the most powerful testament to the validity and reliability of the VALUE approach to assessment comes from campuses that have paired participation in the VALUE initiative with rich, local assessment of student work using the same VALUE rubrics. Many participating institutions are using data from the VALUE initiative as a validation of their own local scoring of the same student work, thereby adding a more sophisticated, robust methodological element to campus-based assessment.

REPORT: http://www.aacu.org/OnSolidGroundVALUE

“A key feature of our assessment strategy is the scoring of authentic student work using a common rubric, which the AAC&U VALUE rubrics provide,” said David Switzer, Faculty Fellow for Assessment and Associate Professor of Economics at St. Cloud State University. “Our participation in the MSC and [Minnesota] Collaborative has given us both the knowledge and the capacity to assess student work from all across the university, and shed light on how to assess student learning in co-curricular programs.”

Moving forward, AAC&U is focusing on the disaggregation of results through the collection of robust demographic data associated with each student work product to discover any trends or patterns in learning across select student populations (e.g., low-income students). As George Pernsteiner, President of SHEEO, reported, “There is a hunger among policy leaders at the state level to make sure that our students are actually learning what they need in order to succeed in life and work. That’s motivated, I think, because of what employers are saying, but part of it is motivated by a more collective view of what is right for society in a given state.”

In addition, scores generated for each learning outcome are being examined in relation to key variables such as faculty members’ specific disciplines. The ongoing VALUE initiative puts learning outcomes quality and improvement in the hands of state and institutional leaders, faculty, and students—exactly where it needs to be if educators and policy makers are serious about preparing graduates for success beyond the first job and in their personal, civic, and social lives, regardless of what type of institution they attend.

VALUE has been funded by the Bill & Melinda Gates Foundation, the Spencer Foundation, the Sherman Fairchild Foundation, Lumina Foundation, the Fund for the Improvement of Postsecondary Education, the US Department of Education, and the State Farm Companies Foundation.

For additional information about the VALUE initiative and to download a free version of On Solid Ground, visit http://www.aacu.org/value. For more information about SHEEO and the Multi-State Collaborative, visit http://www.sheeo.org.