Key Topics

• An instrument of the neoliberal university
• The assessment bureaucracy
• Is assessment benign?
• Assessment in the ambiguity of the classroom
• Critiques of Method
• QuantCrit Critiques
• Has SLO improved student learning?
• What to do?

Introduction

To this point, the book has emphasized assessment and evaluation as good and healthy processes and focused on the “how to” do different types of assessment well. Although there is widespread agreement about the benefits of assessment and evaluation, student learning outcomes assessment in particular has no shortage of critics. These critiques question whether it is a productive and useful activity and one that reflects adequate consideration of the diversity of today’s students. These are relevant questions with respect to using large scale student data for institutional decision-making as well. This chapter addresses critiques that apply most specifically to assessment of SLOs but includes some reference to critiques of using large scale data to measure or encourage student success. As current or future administrators in institutions in which SLO assessment is the main form of assessment and evaluation activity, it is important that you be aware of these critiques and do what you can to counter them.

Higher education, like other goods and services, has been bitten by the assessment bug, pushed in no small way by accrediting organizations. In their urgency to demonstrate that a college education is worth the increasing cost, to demonstrate worth of programs in budget strapped institutions, and to comply with accreditation mandates, the tendency is to simply get on the bandwagon and assess, assess, and assess some more. This seemingly frantic behavior is what got me thinking about this chapter in the first place. After teaching the course associated with this text for a number of years, I came away with an image of higher education administrators, especially in student affairs, busily scurrying around handing out pretest and posttest surveys for every activity they offer, laboring to write learning outcomes for activities for which traditional learning outcomes can’t be easily identified or assessed, searching for learning outcomes in every single activity they do, accumulating mounds of paper and electronic surveys, and pre and post test data that sit in file cabinets unanalyzed. You likely do not have to look too hard to identify examples of such situations on your campus on in your unit.

There is nothing inherently wrong with planning a program with goals and outcomes in mind or with routinely assessing whether programs are achieving desired outcomes. One would not normally begin a trip without having a destination in mind and a plan for how to get there. That said, the critiques raise valid points that should be considered. The purpose of this chapter is to introduce and consider some of these critiques. The purpose of this discussion is not that you should unilaterally and totally “resist” assessment as Bennett and Brady (2014) recommend. That attitude would not be very popular neither is it justified. Rather, the goal is to get beyond the frenzy and “hot mess” (Lederman, 2019), to resist the nonsensical and performative aspects of the activity, and to encourage purposeful activities that will generate meaningful and useful data to help programs improve.

Some of the critiques apply generally to any evaluation activity, but most apply more specifically to student learning outcomes assessment. For each, think about not only whether the critique is a valid one but also consider assessment practices that might counter or reduce some of the problems noted. The language of the critiques is often colorful and clever. That does not make the critiques any more or less valid, but it does make them fun to read and consider.

I have organized the critiques into several categories: Assessment as an instrument of the neoliberal state, assessment as bureaucratic task, assessment as an allegedly benign activity, the challenge of assessing higher education’s more ephemeral outcomes, methodological questions, critical perspectives on large scale data, and finally, and perhaps most importantly, has SLO assessemt made a difference in student learning.

An Instrument of the Neoliberal University

Much of the critique of student learning outcomes assessment lies in a fundamental concern about the the neoliberal state of higher education in the 21st century. The argument has several faces. Neoliberalism is a term often used derisively by critics to describe a political and economic system that prioritizes free market capitalism, deregulation, privatization, and competition. In higher education, the neoliberal condition is manifested by declining public investments in higher education which force even public colleges and universities to adopt strategies of the corporate world. Prospective students are both markets to be captured and consumers to be appeased. Faculty members are rewarded for monetizing and commodifying the fruits of their labors. In this scenario, the goals of a higher education degree are closely tied to the economy and job market; outcomes assessment leads colleges and universities to identify narrowly defined outcomes that are measurable and tied to jobs. By implication, if an outcome is not measurable (e.g., wisdom), it may not be as valued. One of the frequent criticisms of assessment in K-12 schools is that curricula have changed to reflect what is tested rather than what a well-educated child should learn. It would be very challenging for testing regimes to do this in higher education so the question becomes how to ward off more centralized control of college outcomes.

Second, accountability plays a central role in the “neoliberal” state model. As Campbell (2015) notes, “With increased costs, increased enrollments, and questions of quality (i.e., high investment and low trust), higher education is ripe for the creation of metrics and calls for transparency” (p. 527).  Student learning outcomes assessment can be seen as providing one form of such “metrics.” Student learning outcomes assessment is often construed as holding colleges and universities to account for what they claim to do—educate students. Campbell (2015) argues, however, that SLO assessment fails as a measure of accountability because the wealth of data created on student learning is never seen by the public. (Given the rather uneven nature of outcome data, it may be a good thing that SLO assessment data are not generally public!)

Additionally, the neoliberal view argues that the assessment movement renders education an unworthy investment if institutions can’t demonstrate that students “get what they pay for” (Pontuso & Thornton, 2008). From this perspective, learning outcomes are thus trying to serve two processes: Accountability to the public and to demonstrate return on investment. Some argue that this is a fundamental flaw: the two cannot and should not be served by the same processes (Campbell, 2015; Shireman, 2016a).

The neoliberal view is based on a set of assumptions that likely clash with those of some alternative world views, such as those rooted in Indigenous ways of knowing for example. Although much can be done by thoughtful people to make assessment less the tool of an oppressive, neoliberal state, more equity oriented, and more responsive to new populations of students, it should be clear that the task of assessment and evaluation, as currently conceived, are fundamentally products of, and embedded in, a worldview tied to the neoliberal state in the 21st century, that may clash with alternative worldviews and methods.

“Save us from the Earnest”

This section heading, one of my favorites, is credited to Pontuso and Thornton’s critique in which they ask whether outcomes assessment is actually hurting higher education (2008, p. 66). One of the more salient critiques of student learning outcomes assessment is that it has become a “bureaucratic behemoth” and a “ballooning industry” that takes time, labor, and money away from the core function of colleges and universities (Worthen, 2018). This concern has led Pontuso and Thornton (2008) to pose the question: “Could it really have been the intention of architects of ‘outcomes assessment’ to distract educators from their primary responsibilities” (p. 61)? They go on to cite Michael Crozier’s argument that

bureaucracies are burdensome precisely because their talented members… want to reform the world. In the case of assessment, such people assume more and more responsibility, convinced that they are improving something – the students’ learning, the professors’ commitment to quality instruction, and the institutions’ reputations. In doing so, they give bureaucracy more and more work, its staff greater authority, and teachers more inane paperwork. (p. 66)

A few years ago, the University of Kansas was scolded by its accreditor, the Higher Learning Commission (HLC), for deficiencies in its student learning outcome assessment program. One of the HLC recommendations was precisely that the university establish a clear structure—a bureaucracy—to govern assessment, which includes putting a person in charge!  Critic Erik Gilbert (2019) describes the “assessment project” as having “armies of consultants, software vendors, journals, foundations, institutes, and organizations….” (p. 3). Although Gilbert’s observation is a bit hyperbolic, it is accurate to say that nearly every college and university has hired professional assessment staff and for every full-time assessment professional hired, another faculty or staff member is not hired. That said, if assessment professionals and assessment activities actually do help colleges and universities improve the teaching and learning process, they are worth the investment. This is a big “if” as discussed later in this chapter.

The effects of increasing bureaucracy have another, more subtle, effect.  Assessment can become a “managerial tool for quality assurance” (Erikson & Erikson, 2018). This is no more apparent than in the emphasis that regional accrediting bodies have put on institutions to perform in the area of outcomes assessment. Failure to do so in the current political context risks an institution’s accreditation status and thus ability to receive federal financial aid dollars, federal grants, etc.

Accreditors force institutions and their faculties to develop learning outcomes at the course and program levels, to collect data, and to use it to improve learning. And, they expect to see organizational resources devoted to the process. This has resulted in most large universities devoting considerable resources to create an assessment infrastructure consisting of people and software, which, of course, means that resources are coming from some other place to fund assessment.

A second aspect of the “managerial” critique is that, depending on how prescriptive institutions are, assessment staff can exert an influence on how faculty teach and on the curriculum itself. This is especially true for online programs and courses that are especially outcomes driven in every aspect of course construction and implementation. I should note that accreditors are typically not overly prescriptive in what an institution’s outcome assessment program ought to look like. That, however, does not stop leaders of the bureaucracy from using accreditation as a rationale for increased managerial influence over curricular and teaching processes that were once the sole purview of the faculty.

From an administrative perspective, having an effective SLO assessment program is a delicate balancing act between complying with accreditor’s demands for a well-organized, centralized assessment system that follows best practice and taking a more decentralized faculty/staff oriented, bottom-up approach hoping that later adopters will follow the early adopters because the benefits are perceived to be so great. The argument in support of the latter is that assessment will be better if voluntarily done by faculty and staff who want to do it.

Beware of What Appears Benign

A proponent of outcomes assessment might ask, “Well, what’s the harm?” Outcomes assessment does not risk life or limb and could actually have positive effects. Bennett and Brady (2014) argue that this is precisely the problem. Rather than being innocuous, as it seems, this argument assumes that outcomes assessment “provides an ideological smokescreen acting as a distraction from the real problems with U.S. higher education” (p. 147). When outcomes assessment moves away from the classroom and an individual professor to the level of a program, so this critique goes, it becomes a tool for “monitoring and auditing educators” (Bennett & Brady, 2014; Hussey & Smith, 2003), which is related to the previous critique. The real danger is that assessment puts blame on institutions and their faculties and staffs while overlooking crucial external social and structural issues and their consequences for colleges and universities (Worthen, 2018). Bennett and Brady call this “a classic liberal’s blindness to the ideological and socioeconomic contexts in which a practice exists” (p. 149). This is an important point.

The classic liberal position assumes a level playing field based on merit while critics like Bennet and Brady argue that the underlying assumptions of the assessment movement are rooted in an “undemocratic quest for social efficiency (Bennet & Brady, 2014, p. 149).”  They go on to argue that while touted as a progressive concern for student learning, the original roots stem from a very conservative effort to promote scientific management (Bennet & Brady, 2014). (Read Barr and Tagg’s (1995)  seminal article, “From teaching to learning” and you will see how Bennet and Brady could plausibly come to this position.)

Assessment, according to this argument, serves to increase the gap between rich and poor rather than diminish it. In this case rich and poor refers to institutions that tend to attract lower income versus higher income students (Bennet & Brady, 2014; Worthen, 2018). Assessment is particularly problematic if, as Worthen (2018) claims, it occupies a more central place at less selective regional universities that are likely to attract less academically prepared students than at the elite institutions. This raises the question of whether regional, less selective, colleges and universities are likely to spend a greater percentage of their limited budgets on the assessment bureaucracy than elite public or private universities? My personal observation is that less prestigious institutions do, on average, take accreditation much more seriously than bigger, more prestigious ones. The less prestigious spend time, money, and human resources engaging in accreditation related activities such as attending workshops and conferences, and even hiring consultants to assist in writing self-studies. Whether they spend more money on accreditation and assessment on a per capita student basis than research universities is an open question. Less selective institutions are, however, more susceptible to negative outcomes in that they have fewer resources to address inadequacies necessary to remove sanctions.

The Constructive Ambiguity of the Classroom

The previous critiques have focused on social, contextual, and administrative aspects of assessment rather than on assessment as an educational process. There are, as one might imagine, many critiques of the educational shortcomings of assessment in the classroom. Critiques in this bucket(s) include some of the following ideas: There are multiple types of learning outcomes, assessment drives learning rather than learning driving assessment, some outcomes of higher education are not easily reduced to simple outcome statements, and learning is more complex and sophisticated than outcome statements make it seem (Hussey & Smith, 2003). As Lampert (1985) notes the classroom is a site of “constructive ambiguity.” This raises the question of whether it is it possible to capture some of higher education’s most important outcomes accurately in simple measures and while students are in college, ABCD model or not.

Several authors, for example Erikson and Erikson (2108), zero in on the ambiguity of the “verbs” that form the heart of outcomes statements. In fact, the outcomes movement has directed a laser like focus on Bloom’s taxonomy of verbs and newer taxonomies of learning outcomes. Google the term “Bloom’s Taxonomy” and a long list of university websites pops up. Bloom’s list of verbs is undoubtedly more famous now than it was when first published! As Erikson and Erikson (2018) demonstrate, these hallowed verbs such as describe, analyze, and demonstrate, can, and do have very different meanings in very different contexts, and yet they are typically treated as having universal meaning to everyone and in any context and that everyone is in agreement on meaning. For example, Bloom’s taxonomy conceptualizes description as a lower learning goal than analysis or understanding. Erikson and Erikson dispute this interpretation arguing that these terms only have meaning within disciplinary contexts. For example, careful observation and description are key to the scientific method. Darwin’s theory of evolution was based, in part, on careful observation and description of what he saw and yet Bloom portrays description as a lower-level task. Focus on the verbs, of course, assumes that some educational goals, such as critical thinking, can even be stated in formal learning outcomes statements at all. To this point, Erikson and Erikson (2018) argue that learning outcomes statements focused on critical thinking imply it is a uniform outcome with one clearly agreed on definition, which seems antithetical to the concept of critical thinking  (p.17).

Proponents of student learning outcomes assessment take the importance of verbs and outcome statements one step further arguing that they communicate clearly to students what they are expected to know and be able to do and the levels at which they should achieve them. The assumption is that the information provided in clear outcome statements will help  students to do better by clearly informing them of what is expected at each level of performance. (A similar argument is made about grading rubrics.) Critics soundly disagree, arguing that if faculty members don’t uniformly understand outcomes or use them to mean the same thing (likely accurate), then the same ambiguity of meaning is magnified by the literally hundreds of students who take a course or earn a major, assuming they even know the intended outcomes. Moreover, compilers of tables that attempt to help educators with the task of choosing verbs for outcomes statements list the same verbs for multiple levels of cognitive complexity suggesting that meaning of the verbs is not as cut and dried as implied. Various authors also caution that learning outcomes can become ceilings for learning and thresholds or hurdles that need to be cleared (Erikson & Erikson, 2018; Hussey & Smith, 2003).

Additionally, Hussey and Smith (2003) note that not all learning outcomes can be intentionally stated at the outset of an educational activity and thus, as currently framed, don’t represent the realities of teaching and learning. Hussey and Smith (2003) introduce the notion of “emergent learning” outcomes (p. 362) that arise from activities in a classroom or in a curriculum. Thus, not all important learning outcomes can be identified ahead of time (nor should they be). These emergent outcomes are analogous to unintended consequences. Ultimately, Hussey and Smith (2003) argue for a broader, more flexible conception of outcomes that take these realities into consideration.

At least one recent assessment text—Henning and Roberts (2024)—attempts to counter some of these arguments by proposing alternative learning frameworks that might provide alternative verbs or ways of thinking about assessment. However, if one accepts Bennet and Brady’s (2014) argument that assessment is inherently a conservative process to advance scientific management, finding new verbs based on different learning frameworks does little to challenge an entire system; it merely tinkers around the edges.

Method: A Spurious Sense of Precision?

When I do accreditation visits for HLC, I find that institutions often spend more time talking about their efforts tinkering with the method by which they do assessment than they do talking about how they use outcome data to improve student learning. Surprisingly, then, use of weak methods is not a more prevalent critique. Rather, the critiques seem to assume that if the assumptions underlying assessment are faulty, then one can’t expect the methods to be anything but faulty as well. More “valid” methods would require more time and money only to serve, in their minds, an invalid purpose (Pontuso & Thornton, 2008). Hussey and Smith (2003) conclude that student learning outcomes are “imbued with a spurious sense of precision and clarity that are insensitive to different contexts and disciplines” (p. 358). In other words, the mere act of writing outcomes statements with Bloom’s verbs along with expected levels of performance and use of supposedly carefully constructed research designs, implies a methodological rigor and precision that simply do not and cannot exist in the “constructive ambiguity” of the classroom. If anything, the ambiguity in student affairs, student success, and co-curricular programming is even greater than in the classroom. This false sense of precision is precisely one of my concerns about over and inappropriate use of pre/posttests. Their use can imply a false sense of precision and causation to the user.

Robert Shireman, Senior Fellow at the Century Foundation, (2016a and 2016b) has been one of the more vocal of the U.S. critics of outcomes assessment. Among his general criticisms is the charge that assessment is merely a misguided effort to prove that policy makers (namely accreditors) care about student learning by implementing “inane counting exercises involving meaningless phantom creatures they call student learning outcomes, or SLO’s” (2016b). He sees the outcomes assessment movement as a product of efforts by well-meaning reformers (the earnest) who truly believed that teaching and learning in college could be improved by refocusing not on subject matter, but rather on learning outcomes and the typical assessment cycle: identify outcomes, collect data, make changes. Shireman argues that this supposedly rational approach to improving learning has not worked out as intended. In his view, starting with learning outcomes has resulted in lists of topics for what a course should include and efforts to demarcate levels of learning that are “gibberish,” and that convey a false sense of scientific precision around which assessment systems are built. As a consequence,  with the result being “worthless bean counting” rather than deep and meaningful learning (2016a). One might argue that the big stick of accreditation has exacerbated the bean counting problem. Instead of focusing on mindless bean counting, he suggests using authentic work as indicators of student learning. I will return to this this below.

Each of the traditional methods of data collection, including institutionally collected and reported data, self-report surveys, assignments analyzed using rubrics, and externally developed tests, can also be critiqued (Campbell, 2015). Years ago, there was an emphasis on using relatively expensive nationally normed “tests” developed by the Educational Testing Service and other testing groups. Those fell out of favor because the tests were found to measure high school general learning and were not tailored to a college’s specific curriculum. Besides, they were expensive and student motivation to do well on them was lacking. Self-report surveys, such as the NSSE  have been particular targets of attack as measures of learning. Porter (2013), especially, argues that students’ estimates of their own learning are more likely a function of their overall perceptions of the campus environment and their pre-college characteristics and experiences than their learning. (Porter’s critique is focused on the NSSE, but his point is relevant for other self-assessment surveys as well.) Others counter that self-report data are better than no data, a point with which I agree.

Focusing solely on methodological limitations risks “assessment paralysis” while ignoring the criticisms may result in piles of meaningless data. Certainly, there are calls to employ more rigorous externally derived and validated assessment instruments and to use more rigorous research designs (e.g., Banta & Baich, 2011; Dowd & Tong, 2007). I call them the “If only we had more and better data” arguments. Shireman rejects the “if only” argument out of hand. He wants different data. His response to what he sees as the inane focus on outcomes using contrived assessments is summed up in the following quote with a call to use authentic work:

A student-work [authentic, direct assessments] approach serves as a check on whether colleges are engaging students in projects that intrigue and challenge them, a function of both the learning experiences that instructors design, and the campus supports, services and environments that help students focus on school and stay on track through those 3,446 tasks [the homework, assignments, tests, presentations, etc., a student does to graduate]. An accountability approach that starts with the artifacts of student engagement stands the best chance of prompting institutional redesigns that will increase low-income students’ likelihood of graduation with a high-quality degree (Shireman, 2016, p. 14. Text in brackets added by Twombly.)

    It is fair to point out that, as noted below, many institutions would say that they do begin with, and rely heavily on, a student-work, direct assessment, approach, which is much more difficult in student affairs, student success, and in some co-curricular programs. Certainly, organizations such as National Institute for Learning Outcomes (NILOA) and the American Association of Universities and Colleges (AACU) argue for use of authentic assessments although maybe not in the way Shireman intends. It perhaps goes without saying that authentic assessments of direct learning is easier done in academic units than in student affairs and success programs and perhaps in co-curricular as well. And, to insist on more rigorous methods begs the question of whether the focus would shift to improving methods rather than student learning.

A “Quantcrit” Perspective

An emerging critique is that introduced by Gillborn, Warmington, and Demack (2018) who advocate what they call a “Quantcrit” critique of using large data to make decisions about humans to include predictive or learning analytics in higher education. These are the programs that advising units use to supposedly improve advising by relying on extensive data on past student performance to predict current student performance and to recommend options. Rooted in Critical Race Theory,  QuantCrit attacks the seeming primacy of quantitative, numeric data to make all sorts of educational policy decisions. Gilborn et al.’s argument is that data are 1) presumed to be objective and are 2) unbiased and not racist. For Gillborn et al., the mere fact that numbers are labeled objective is part of the problem because assumed objectivity gives the conclusions or recommendations from quantitative data a presumed and, perhaps false, respectability. Although numbers themselves are not biased, all research (this is as true of surveys and questionnaires as it is of predictive analytics systems) that generate the data on which decisions are based is a product of humans and all of their preconceptions, assumptions, and biases. Use of predictive analytics, which relies on historical data to inform recommendations for current students, has been a particular focus of critique for its potential to negatively affect underrepresented populations.

Two additional concerns with predictive analytics must be recognized. First, they do not account for individual motivation or other individual factors. Second, they are based on algorithms made up of variables selected by someone whether that be system operators, consultants, or college officials. These algorithms are often proprietary and not public. It is crucially important for administrators to understand what variables are in the models being used and to determine whether they appropriate to apply to the students at your institution.

To reject use of “big quantitative data,” and predictive analysis systems, out of hand for its potential biases ignores the reality that humans are likely even more biased. (See Yuval Noah Harari’s discourse on this topic in 21 lessons for the 21st century.) The point is that all human research is laden with assumptions and biases whether based on quantitative or qualitative data. These implicit beliefs should be made explicit and considered to the extent possible. Colleges and universities ought not hide behind the presumed objectivity of quantitative data to reinforce inequities.

In the field of teacher education there is an effort to promote culturally responsive teacher evaluation. It seems that the concept could also be adapted and applied to assessment of student learning outcomes, although doing so would be challenging when involving “big data” for predictive puproses. What might culturally responsive SLO assessment look like? Some thoughts go back to Kitchner’s argument about doing no harm, ensuring instruments are not systematically and knowingly biased, collecting and analyzing data responsibly, being sensitive to a need to understand outcomes for students from diverse backgrounds without essentializing students or labeling them as deficient. Although there are no clear answers, the important starting point begins with asking the question. See Henning and Roberts’s 2024 version of their assessment book for a chapter-length discussion of various approaches to culturally responsive assessment in student affairs.

Does SLO Assessment Improve Student Learning?

Now to the most important question and criticism of all. Has assessment of student learning outcomes made a difference in student learning? In other words, has student learning improved in light of 30+ years of doing SLO assessment? Improving student learning is presumably the stated goal of assessment. Most of the critics either implicitly or explicitly, point out that there is little evidence that the millions of dollars and person hours spent on assessment has yielded any discernable effects on student learning or institutional decision making (Banta & Baich, 2011; Cox, et al., 2017). This statement was reinforced by a panel of assessment and accreditation experts at a 2019 meeting of the Western Association of Schools and Colleges. They described assessment as a “hot mess” (Lederman, 2019). The outcomes of higher education are clear at one level as one presenter at the meeting noted: students who attend college and graduate earn more over a lifetime than those who do not (although this earnings gap has declined over time). However, when attempts are made to assess what students learn in classrooms, the effects are small (Lederman, 2019). These conclusions are supported by research by Banta & Baich (2011) who reported that only six percent of 146 institutions they examined showed any evidence of improved student learning. This is partly because, according to Pontuso and Thornton (2008), “the dirty little secret” is “that teachers pay almost no attention to assessment outcomes” (p. 63).  If the latter is, in fact, true, then units should ask themselves why their data is not being used and reorient to get useful data.

Suskie (2018) rightly points out that assessment itself can’t improve learning. It is rather the insights assessment offers into curriculum and teaching and what is done with those insights that can lead to improved learning. That said, she largely agrees that aside from one study done in engineering back in 2006, there is little evidence that assessment has led to improved learning although college and university officials believe that assessment has led to better teaching. This is concerning for a large scale effort requiring significant attention and resources.

This critique brings us full circle back to the origins and purpose of the student learning outcomes assessment movement—to use SLO assessment data to improve learning (p. 63). More than 25 years and millions of dollars and person hours after Barr and Tagg (1995) made their appealing argument that increased attention to assessment of student learning would lead to improved quality of higher education, it is not clear it has done so. And, if this is true for student learning outcomes assessment in academic programs, one can surely argue it is no less true for assessment of learning in student affairs, student success, and co and extracurricular programs.

On the more hopeful side of the debate, are those who make some version of the “if only” argument: if only there was more, better, or different data. If only assessment were part of the faculty reward system, or if only more assessment resources or knowledge were available, assessment would yield significant increases in student learning. Scholars, such as Campbell (2105) and Dowd and Tong (2007) are in this “if only” camp.  For them, the answer is more, rather than less, evidenced based, scientific research methods involving multiple sources of data. Likewise, Banta and Baich (2011), while recognizing genuine obstacles to using more rigorous methods involving faculty (or staff) members in assessment (e.g., lack of time, resources, and knowledge) remain hopeful. They are hopeful that “if only” good practices are followed and institutions use the resulting data to continually inform decisions, fundamental changes will result. This argument follows the classic liberal belief in progress resulting from rational processes.

Jankowski, Timmer, Kinzie, and Kuh (2018), in a report for the National Institute for Learning Outcomes Assessment (NILOA), used results from a 2017 survey of chief academic officers about assessment practices, to argue that assessment practices are actually trending toward what Shireman recommends—use of authentic measures—which presumably  provides “better data.” NILOA’s conclusion portrays the assessment enterprise as increasingly using authentic measures of student learning that meet institutional needs, that is integrating assessment initiatives across the institution, and is using results to inform course and program-level improvements.

The authors of the NILOA report agree that many challenges remain: communicating results, documenting improvements in student learning, adopting equity as a focus for data use, providing enhanced professional development regarding use of assessment results, technology use, and integrating assessment efforts across an institution. Many of these challenges are even greater for student affairs, success, and co-curricular programs. NILOA, however, is not a disinterested party. As part of the assessment bureaucracy, theirs is essentially a “believer’s argument” like that of Banta and Baich (2011) and Suskie (2018). NILOA exists to promote assessment and therefore believes that if it is done better, the results will be better.  As an HLC reviewer, I am far less convinced that institutions have the time, will or capacity to do assessment really well and to sustain such efforts over the long haul.

Cox, et al. (2017) and Taylor (2020) are far less optimistic. Both authors used large scale quantitative analyses to look at assessment outcomes. Although most institutions in the Cox et al., study used some form of data driven decision making, the authors did not find much support for the assumption that data on student learning outcomes is used to “mediate relationships between students’ incoming characteristics and their engagement in educationally effective practices” (p. 855) or to create rich learning environments that uniquely affect students’ learning. In fact, Cox et al., challenge the notion that “more assessment will improve student outcomes” (p. 856), especially if done for symbolic reasons. Taylor (2020) extends this argument and adds that use of quantitative data to inform data-driven decision making actually subverts student success efforts by forcing student success units to adopt economic logics and to commodify their work. These practices result in “hegemonic data-use cultures, wherein the dominance of economic logics was introduced and maintained through systemic and cultural shifts in data use practices” (p. 1090). According to Taylor, those who disagree with or who do not use data to drive their decision-making risk loss of resources, if not their positions.

The empirical work of Cox et al., Taylor, and experiences of others (e.g., Basko, 2021), are evidence that critique of institutional reliance on student-related data to drive institutional decisions is growing but still largely unheeded. Despite a significant effort to significantly raise the expectations for SLO assessment on my campus, I have seen little mention of any concerns raised about any of these critiques. In fact, most assessment experts double down on the “If only we have more outcomes, better maps, etc.,” assessment will be better and maybe learning will be enhanced.”

The critiques suggest that colleges and universities ought to reflect on and evaluate the effectiveness of their student learning outcomes processes. In other words, they ought to apply assessment and evaluation principles to the SLO process itself. For example, Taylor (2020) argues that data-driven decision involving assessment making can actually subvert student success efforts while Basko (2021) opines that higher education may be asking the wrong questions about student success and therefore using the wrong data to support its claims. An assessment of assessment as a process might reveal other challenges but also open paths for improvement.

What to Do?

Erik Gilbert (2019) an historian, and frequent assessment critic, brings us back to the ultimate question: Has assessment of SLO has made colleges and universities better? Hs it enhanced learning? Does it cause harm by diverting resources and shifting focus away from the real problems of declining budgets, rising tuition (forcing students to work more) and student debt loads (Gilbert, 2019; Worthen, 2018)? There is little support for the claim that parents and students use SLO assessment data from academic or student affairs and cocurricular programs to make college choice decisions. Gilbert is especially cynical because he thinks that climbing walls are a more important determinant in college selection than whether a college produces learning outcomes. Given all of the time, effort, and resources devoted to student learning outcomes assessment, there is unfortunately little evidence of improved learning. That finding is disturbing given the attention paid to SLO assessment.

These critiques are compelling raise important questions. They should not be dismissed as merely the complaints of lazy, whiney faculty members who simply do not want to be held accountable for what they do (a common rebuttal to the critiques). Careful consideration of the critiques can be an important aspect of figuring out how to make assessment practices better and more meaningful, leading to better student and institutional outcomes.

This seems particularly crucial in student affairs, student success, and co-curricular areas where the frenzy to engage in student outcomes assessment seems to have reached a fevered pitch, where the outcomes are even more difficult to capture in Bloom’s verbs, meaningful curricula hard to implement, participation is variable, and the learning spaces even more constructively ambiguous. The problem is exacerbated in these units if SLO assessment data, ostensibly collected as the basis for making program improvements, is then used for accountability and program justification purposes. It is also made worse if SLO assessment follows a model ill suited for the outcomes produced. Data that are useful for the former may not be for the latter and vice versa. The former asks you to take risks to make student learning better and build on your mistakes. The latter may punish you if you can’t demonstrate productivity or even admit weaknesses in any area.

Questions such as what should be assessed, when, how, how often, and for what purpose are critical. Merely participating in the charade to demonstrate program worth by collecting minimally useful data on anything and everything a program or unit does using questionable instruments does no one any good and it takes already taxed staff away from their main duties. I don’t have the answers, but I do know that the first step is knowing how to do assessment that is meaningful and thoughtful, to be clear about why one is doing it, and knowing when, where and how often to do it. Moreover, one must know how findings will be used. Thinking about the critiques should help you avoid some of the pitfalls.

Summary

In sum, outcomes assessment (and evaluation) are important activities or ways of thinking about what you do. This chapter has reviewed some of the major critiques of and concerns about SLO assessment as an organized, institutional effort. Responsible administrators and faculty members, as well as accrediting agencies, need to examine their assessment programs to ensure they are yielding useful information that will lead to the promised improvements in student learning and student success.The need to do SLO will continue.