i. Standard Eight – Educational Effectiveness

In the text that follows, we describe the range of processes we use to collect information about the experiences of Wesleyan students as they relate to retention, graduation, and learning. Much of what we describe below bears on themes in President Roth’s December 2016 “look back” on Wesleyan 2020. There he highlights our open curriculum, which asks students to own their educational experience by choosing their own academic itinerary, making their own selections among Wesleyan’s myriad opportunities to learn in the classroom and across disciplines — as well as through co-curricular activities. In line with our mission to provide “a liberal education characterized by boldness, rigor, and practical idealism,” we hope that all students who attend Wesleyan graduate on time with minimal debt, having participated in educational experiences intrinsically rewarding and beneficial to them beyond the university.

Retention and Graduation Rates and Other Student Experiences

Wesleyan continues to successfully retain and graduate its undergraduate and graduate students. The percentage of first-time students who return to the university for a second year has held steady around 95%, while about 92% of incoming first-time students go on to graduate from Wesleyan within six years of their arrival (see Standard 8 data form). Since 2012, graduation rates in Wesleyan’s Master’s programs have been strong, (91% to 100%), with the exception of one cohort where this rate fell to 64%.^[1] Among doctoral students, first-year retention is very strong (90% and above), while completion rates have typically fallen between 70% and 79%, with the exception of the “3-years prior” group identified in our Data First Forms, where the completion rate fell to 50%.

The stability we see in the retention and graduation rates for undergraduates overall is generally but not always reflected across all subgroups (see figure below). For example, the graduation rates of first-generation African American students who matriculated in 2006 or 2007 had atypically low graduation rates (79% and 76%, respectively). However, since then, graduation rates for Wesleyan’s African American undergraduates have rebounded and have ranged from 89% to 93%.

The graduation rate of students with the highest financial need trails that of other students by three percentage points, while graduation rates for first-generation college students or Pell recipients trails that of other students by two points. Conversely, students who were actively recruited to play a sport at Wesleyan graduate at a rate two points above students who were not recruited (regardless of athletic participation status), and graduation rates show very little variation by the score used to rate students’ academic backgrounds as part of the admission process. While differences do prompt various interventions across the university, the small magnitude of these differences provides some reassurances about the success of those students who could be considered most at risk of leaving Wesleyan before earning a degree.

Graduation Rates for Select Student Characteristics, Fall 2005 to 2011 First-Year Cohorts

[Insert graph]

The university continues to pursue pilot programs designed to support the success of first-generation students and students from underrepresented groups. First Things First is a two-day pre-orientation for thirty incoming first-generation students to meet peers while learning about available campus resources and benefiting from the experiences and knowledge of older first generation students. WesMaSS, described in the section on advising, is a program for increasing the success of underrepresented students in the sciences. 

The University has responded to increased concerns about equity and inclusion on campus by improving coordination across the Office for Equity & Inclusion, Student Affairs, and Academic Affairs and implementing strategies aimed at increasing inclusion, sense of belonging and success. As mentioned in Standard 5, one of these strategies is the planning for a new Resource Center that will open in Fall 2017. The vice president for equity and inclusion sits on cabinet, and the University has hired a full-time dean for equity and inclusion and increased staffing in the office of counseling and psychological services with a special emphasis placed on cultural competency.

Assessment of Student Learning

Since 2012, we have tried to bolster our system for assessing student learning. We have continually reviewed and sought to improve established processes for evaluating students’ educational experience and curricular coherence. Recent examples include the recent revamping of the teaching evaluation system and our annual monitoring of students’ completion of the General Education Expectations. We have also continued to examine data from surveys of incoming and current Wesleyan students and of Wesleyan alumni. Through our regular suite of surveys, we ask our current and former students about their development of select knowledge and skill sets at four points in time: (1) as they arrive on campus (Survey of New Students), (2) while they are enrolled (Enrolled Students Survey), (3) when they are graduating (Senior Survey), (4) after they have graduated (Alumni Survey). While we do not administer each of these surveys every year nor to all students, and while response rates for each vary,^[2] the surveys provide us with an important starting point for identifying the strengths and weaknesses of the Wesleyan curriculum and providing direction as we begin to design direct assessment projects.

We have generated new opportunities for pedagogical and curricular innovation, as well as more nuanced assessment work at the course, department, and university level. We have accomplished this through the creation of new centers, like the Center for Pedagogical Innovation (CPI) (described in Standard 6) and the Center for Global Studies (CGS), and the expansion of older institutional structures, like the Quantitative Analysis Center (QAC) and the First Year Seminars (FYS). We have also created new positions: a Director of Instructional Design, an Associate Director of Assessment, an Associate Director of Fellowships, Internships, and Exchanges, and an Assistant Director of Language and Intercultural Learning. These positions sit within in the CPI, Institutional Research, and the CGS. We have also formalized the position of Director of Academic Writing, which supports the teaching of writing in the FYSs and in the curriculum more generally.

We note that these new positions and centers have been created through Wesleyan’s new money request process. This is a yearly procedure that allows groups to request funding for pilot programs, typically for an initial three year period. That process has always focused heavily on assessment of success, with interim and final reports required before a decision is made on permanent funding. Thus we do not yet have permanent funding for all of these additions, but we hope their creation will facilitate our unrolling of a new, flexible competency framework to guide projects related to teaching and learning and provide the support required for high quality, feasible and scalable assessment work.

Teaching Evaluations

Wesleyan faculty and administrators pay significant attention to student evaluations of teaching. Teaching evaluation results are used as an indirect measure of the educational experiences of Wesleyan students and as an important source of information considered during reviews of faculty for tenure, promotion, and annual review. Shortly before Wesleyan’s last NEASC self-study, an ad hoc committee commissioned by the Educational Policy Committee had reviewed the state of our teaching evaluations and provided a recommendation for improvement. Following up on that, we have developed, tested, and implemented a completely new teaching evaluation questionnaire and data system.

Before committing to the new form – something the faculty approved with a vote in April 2016 – the university conducted pilot testing with about 40 faculty volunteers. We compared response rates (as well as the provision and length of text in open-ended items) in the old and the new evaluation systems. Additionally, we surveyed faculty and students who had participated in our pilot of the new systems to ensure no significant concerns had emerged. In these surveys, 71% of student respondents preferred the new form, and nearly all faculty reported liking or accepting it.

We implemented the new system and new evaluation questionnaire for all tenured and non-tenure track faculty in the fall of 2016. Tenure-track faculty were each provided the option to decide whether to remain on the old form (for consistency) or move to the new form (for the provision of increased detail) until the term in which he or she stands for tenure.

This new evaluation instrument differs markedly from our old questionnaire. By replacing the old form’s three questions with a total of 12 items, the new evaluation system garners more detailed nuanced feedback by asking students to rate additional dimensions of teaching, courses, and their experiences interacting with faculty. For example, in the old instrument, we simply asked students to rate the “overall quality” of both the course and the teaching they experienced in the course. In the new instrument, we ask students to respond to a series of statements, including:

• The assignments were a useful part of the course.
• There was a clear connection between instruction and assessment.
• The instructor communicated knowledge effectively.
• The instructor was accessible outside of class.
• The understanding/skills grew as a result of this course.

In addition to covering a broader array of topics, we also improved the granularity of information collected by replacing the former system’s four-point ordinal response scales with nine-point scales, allowing for increased differentiation and the use of a broader array of statistics. Finally, the new system allows for flexible reporting and benchmarking (e.g., against departmental or divisional averages, statistics for classes of a similar size, etc.).

General Education Expectations

Wesleyan takes pride in its open curriculum, which gives students the opportunity to direct their own education. At the same time, we trust that students will achieve educational breadth by adhering to our General Education Expectations. Students are expected (not required) to complete two course credits in each of Wesleyan’s three Divisions by the end of their sophomore year (Stage I) and an additional credit in each Division by the time they graduate (Stage II). If completed, the Expectations total nine credits of a student’s required 32.

Wesleyan monitors compliance with the General Education Expectations. Trend data show that about 75% to 80% of our students graduate from Wesleyan having completed both stages, a rate with which the university is satisfied, especially considering that this level of compliance occurs without a strict requirement in place.^[3]

Among students who do not comply with the Expectations, 67% had not completed the three credits in Division III (STEM). Patterns in non-compliance with the Expectations parallel patterns in students’ self-assessments of what they have gained regarding broad content knowledge associated with the University’s three Divisions. Only 47% percent of students who responded to the 2015 Senior Survey reported that Wesleyan has contributed “quite a bit” or “very much” to their understanding of “the process of science and experimentation.” Likewise, only 54% reported that the university had contributed “quite a bit” or “very much” to their understanding of “the role of science and technology in society.” These numbers are low relative to what the same students report for their ability to understand other perspectives, a pattern that is echoed in data from the 2013 Alumni Survey (see figure below).

While Wesleyan is not planning any major efforts to raise the rate of compliance with the General Education Expectations, it is worth noting that the recent changes described in the Response to NEASC Area of Emphasis: Advising may prove beneficial here. By giving faculty advisors more time to speak with their advisees about the goal of General Education Expectations and by making information about General Education courses more accessible, more students may see the advantages of broadening their curricular explorations.

[Insert graph]

Four New Competencies

In its 2012 self-study, Wesleyan explained why it had decided to move away from the ten Essential Capabilities adopted by the faculty in 2005. The hope originally was that these ten capabilities would complement the General Education Expectations in providing curricular coherence for students navigating the open curriculum. Over the years, however, it became clear that the capabilities lacked the practical appeal necessary to guide students’ academic planning. Studies showed no correlation between course selection and students’ reported growth in these capabilities. Moreover, students who had written essays about the relationship between the capabilities and their courses did not rate their advising sessions more favorably than students who had not written such essays. The self-study concluded that “the General Education Expectations and the essential capabilities may serve as guides for some, but what is essential at Wesleyan is that advisors—pre-major as well as in the major—and students work together to define a coherent program in relation to each student’s aspirations and capacities.”

In the summer of 2016, Academic Affairs adopted a simpler, competency-based framework that we believe better represents what is distinctive about the liberal arts education Wesleyan offers. The hope is that this new framework will be of more appeal to faculty and students in advising sessions as well as helping to guide future pedagogical experiments and assessment projects. The new number of competencies is four. The second, third and fourth can be found on AAC&U’s list of essential learning outcomes tied to high-impact practices, and the first, we find, bears particularly on the distinctiveness of Wesleyan’s brand of liberal education:

• Mapping = navigating complex environments (NCE)
• Expressing = writing, expressing, communication (WEC)
• Mining = quantitative analysis and interpretation (QAI)
• Engaging = negotiating intercultural differences (NID)

This framework is meant to be both flexible and inclusive. It is not intended to supersede the competencies that specific departments and disciplines deem primary, and it may well change with feedback from faculty, staff, and students about its utility.

In Fall of 2016, the Provost established an Assessment Task Force to work on:

1. Determining whether Wesleyan could benefit from creating an ongoing assessment advisory committee and who might serve on it
2. Designing strategies for publicizing the four new competencies described above
3. Crafting exercises for the assessment of general education, using the competencies as a guideline, and a timeline for when such exercises might take place
4. Considering how to triage requests for assistance with assessment work throughout campus.

We are also looking outside the university — to the New England Assessment Support Network (NEASN) — for advice in institutionalizing these competencies. This group, composed of representatives from small liberal arts colleges throughout New England, meets once a year to share information about how they have normalized assessment processes among faculty on their campuses, to provide examples of successful general education and departmental assessment projects that examine both content knowledge and competencies, and to discuss how to prepare decennial and five-year reports for NEASC. We hope membership in NEASN will assist us as we institutionalize the four-competency framework on our campus.

In what follows, we examine our student survey data for early clues about whether students have progressed in these competencies during their time at Wesleyan. We also describe how the four competencies bear on Wesleyan’ creation of new centers, programs, and positions that we trust will ultimately support the assessment of student learning as well as supporting student learning itself.

[Insert graph]

Expressing = writing, expressing, communicating. In recent years, Wesleyan has invested in students’ writing in a number of ways. In its 2012 self-study, Wesleyan noted that it was working to formalize the First Year Seminar (FYS) program around the teaching of writing. This change was in response to the perception among faculty that first-year students’ writing was a serious area of concern. We now ask that faculty who teach an FYS require 20 pages of disciplinary-specific, expository writing and encourage them to have their students engage in some sort of revision process. While Wesleyan does not require that students take an FYS, the majority of students do (e.g., 73% of the Class of 2020 first-year cohort), thanks to increasing the number of seminars offered (from 24 to 51, as described in Standard 4) and first-year advisors emphasizing the importance of taking an FYS.

With the reimagined FYS program in its fifth year, we would now like to explore whether students’ self-assessment of their writing ability affects their enrollment in an FYS. The majority (77% to 90%) of graduating seniors and alumni/ae who responded to our surveys responded positively to the question about Wesleyan’s contribution to their writing skills (see figure above). But about 25% of Wesleyan’s incoming first-year students report that they are only “somewhat prepared” or “unprepared” to “write clearly and effectively.^[4]” We might benefit from learning about how these students who report not feeling very well prepared for college-level writing navigate the open curriculum. Do they seek out courses where they will develop their writing skills or avoid them? We also are interested in determining whether students who take an FYS complete it having developed writing habits (e.g. freewriting, multiple drafts, peer review, etc.) and awareness (e.g. about different writing genres, disciplinary approaches and purposes, etc.) that lead to better writing performance in later courses. Finally, we would like to begin documenting the kinds of writing that FYS faculty assign and the kinds of writing processes they introduce to students. While the response and completion rate was low,^[5] a pilot survey sent to students who had completed an FYS in the Fall of 2015 has given us a general sense of how our FYS faculty teach writing. Of those who responded to the survey, 86% reported having received feedback from their faculty on drafts, and another 58% reported having received feedback from peers. While these students reported that commenting on papers dominated faculty members’ instructional style, other pedagogical strategies were common as well (see table below).

[Insert table]

Learning more about what kinds of lessons, readings, and exercises faculty incorporate into their teaching of writing will help us to determine what most contributes to students’ growth in written expression and best prepares students for the kinds of writing they will encounter later in their Wesleyan career. It will also encourage faculty members to share successful pedagogical strategies with one another.^[6] We hope our new faculty appointment, Director of Academic Writing, will help to facilitate the sharing of such strategies, as well as help in the examination of the relationship between writing pedagogies and the student experience.

“Expression” takes many forms, not just writing. Currently, there is no easy way for students to locate courses where they can deliberately cultivate their oral communication skills. It is also not clear how often students receive explicit instruction on how to prepare for a formal presentation (in a class, as part of poster presentation session, for a thesis) or feedback about how they performed in one. One recent proposal is to introduce sophomore seminars focused on public speaking the way the FYS program focuses on writing. Furthermore, while many courses at Wesleyan encourage class discussion, participating in class discussions may not automatically produce students with sophisticated, durable skills in oral argumentation. We do not know whether our students could benefit from more intentional instruction and feedback in such courses, nor much about what they may already be receiving. About 72% of students who responded to the 2015 Senior Survey reported that Wesleyan contributed “quite a bit” or “very much” to their ability to “communicate well orally,” a percentage that generally confirms what we see in Alumni Survey data (see figure above). It is not clear, however, that students are in the best position to assess how much they have actually improved, and our survey questions are not nuanced enough to tell us much about what aspects of their oral communication skills improved the most or what caused that improvement.

We should also note that our “expression” competency can extend far beyond traditional notions of written and oral communication. As noted in Standard 6, one project sponsored by the CPI’s Pilot Programs division is creating structures to support the teaching of digital storytelling as a way to bring active learning opportunities to students taking introductory courses. Courses that replace traditional papers with digital storytelling could provide us with a different kind of assessable artifact, particularly for evidence of how well students link images and audio and edit each to tell compelling stories. Furthermore, students taking courses or participating in programs offered through the Center for the Arts have many opportunities to develop their skills in artistic expression. These programs exist throughout the curriculum, thanks to Wesleyan’s longstanding Creative Campus Initiative, Mellon-funded at first and now supported entirely by Wesleyan. It will take some careful thought to develop assessment projects that allow us to capture how students develop in this regard during their time at Wesleyan that honor the methodological and epistemological approaches used in these disciplines.

Engaging = negotiating intercultural differences. About 64% of students in the 2015 Senior Survey told us that Wesleyan had contributed “very much” or “quite a bit” to their developing global awareness. Data from the 2013 Alumni Survey show that younger alumni/ae are more likely to say this than older alumni/ae (see figure above), suggesting that this is an area in which Wesleyan has changed over time. The newly created Center for Global Studies’ (CGS) provides strategic support for mainstreaming language learning and study abroad in Wesleyan’s liberal arts curriculum and it may therefore contribute to how we think about this competency and how we develop it in our students. The creation of the CGS involved bringing together two previously existing offices — Office of Study Abroad and the Language Resource and Technology area — and adding two new positions: Associate Director of Fellowships, Internships, and Exchanges, and the Assistant Director of Language and Intercultural Learning. In part by uniting these four functions under one organizational umbrella, the CGS will facilitate our students’ intercultural engagement with the world, especially in relation to study and/or internships abroad. It is also poised to help monitor and track their acquisition of knowledge and competencies in this domain.

Among students who responded to our 2015 Senior Survey, 36% reported that the university contributed “very much” or “quite a bit” to their ability to read or speak in a foreign language, while another 36% reported that Wesleyan had contributed “very little or none.” Relative to other skills we ask our students to describe, reported skill development in this area is low, but does appear to be correlated with studying abroad. Among those who studied abroad, 54% reported “very much” or “quite a bit” relative to only 21% of those who did not. Conversely, 46% of those who did not study abroad reported “very little or none,” compared to 24% of those who did. We know less about what differentiates self-reported language acquisition within these two groups (e.g. variation among study abroad programs, languages studied, and on-campus course enrollment), and whether self-reported language gains refer to reading or speaking proficiencies. We also do not know whether students’ self-assessment of language is confirmed by more objective measures, including our language faculty’s evaluations of those students’ written coursework and performance in oral proficiency interviews. Direct assessment projects in which we match students’ self-assessments on this survey to curricular choices and language performance as evaluated by our language faculty might be instructive. While we are early in thinking about these issues and exactly what research questions we should be asking, we are hoping our new Assistant Director of Language and Intercultural Learning will help us as we continue to think about the relationship between language acquisition and study abroad.

We are also interested in the relationship between studying abroad and students’ development of intercultural competencies. We need to determine what tools we can use to capture what happens to students once they study abroad in a way that it is organizationally feasible and that builds upon pre-established administrative processes. For example, Wesleyan students already must write a pre-departure essay describing their goals for studying abroad. These essays could provide evidence about students’ baseline levels of intercultural competency that we could compare to their levels upon their return.

Studying abroad is not the only way that Wesleyan students have the opportunity to engage in ways that cause them to “engage,” something our survey data confirms. When we asked students in the Class of 2015 how much Wesleyan had contributed to their ability to “relate well to people of different races, nations, and religions,” 77% of respondents said “very much or “quite a bit.” This percentage was the same, regardless of whether the students had reported studying abroad or not. That study abroad should help with learning a new language but not with intercultural competency bears consideration. We would like to think that this is because our campus provides multiple institutional structures intended to cultivate this skill: through our course offerings, the programming offered by the Allbritton Center for the Study of Public Life, the Office of Religious and Spiritual Life, and the other, many robust extracurricular options that have the potential to bring students from different backgrounds into contact with one another. We still have work to do, however, in determining how well these opportunities are accessed and received by all students and how successfully they foster students’ ability to negotiate intercultural differences.

Mining = quantitative analysis and interpretation. Wesleyan’s Quantitative Analysis Center has expanded its curriculum and cocurricular programming substantially since 2012. It now sponsors a certificate in applied data science, a data analysis minor, and an initiative supporting the exploration of computational thinking through unique course offerings and lectures. These supplement its well-established summer apprenticeship program, statistical and GIS tutoring services, and history of promoting project-based learning. To supplement the quantitative courses offered inside specific majors, the QAC also now offers an increasingly wide array of both partial- and full-credit courses, from Working with Excel and VBA, to the Economics of Big Data to Introduction to Statistical Consulting. We clearly list these courses in the online catalog and on the webpages of both the QAC and Gordon Career Center. We are confident that if students want to work on their quantitative (particularly statistical) skills, they know where to look for courses and have many options from which to choose. From fall 2012 to fall 2016, QAC enrollment increased nearly four-fold (from 78 to 297), and the number of distinct offerings increased from three to 26.

We also have some evidence to suggest that the QAC succeeds in attracting students who might otherwise not enroll in a course that explicitly teaches quantitative skills. Research conducted by QAC faculty show that its project-based courses, relative to statistics courses that rely on more traditional pedagogy, attract and retain students from a broader range of majors and diverse backgrounds.^[7] QAC faculty have also found that underrepresented minority (URM) students at Wesleyan who took these courses were as likely as non-URM students to have gained confidence in their statistical skills and were twice as likely as non-URM students to exhibit an increased interest in conducting research.^[8] We know less about how successfully students transfer what they learn in QAC courses to subsequent courses and capstone projects or how enrollment in a QAC course early in one’s college career might shape subsequent trajectories in terms of research interests, course enrollment, and even extracurricular activities. We would also like to know about what motivates students to take QAC courses: Do they do so to fulfill a major requirement; to supplement other coursework in the major; to satisfy a General Education Expectation; and/or to prepare for the job market?

While the QAC’s expansion is an important curricular development at Wesleyan in regards to quantitative course offerings, it is also the case that departments in Division II and Division III have always offered their own sets of courses in which they develop a broad range of quantitative skills in their students. While we expect that students who complete majors in these divisions — or who have taken courses in these divisions to satisfy General Education Expectations or their own intellectual interests — may have stronger quantitative skills than students who do not, we have not yet tested this assumption with survey data or direct assessment measures. Further, for those students who do not take QAC courses – or quantitative courses offered by the majors – we do not know whether or how they develop “basic” quantitative literacy. How often are the kinds of assignments that might promote these skills embedded into what appear, at least by name, to be non-quantitative courses, and does students’ quantitative literacy improve in significant ways after having completed them?

Almost 40% of students in the 2015 Senior Survey reported that Wesleyan had contributed only “some” or “very little or none” to their ability to “understand and use quantitative reasoning and methods.” While this is higher than we might like to see, it is also much lower than what we see for those who responded to the 2013 Alumni Survey question about being prepared to “use quantitative tools” (see figure above). In fact, our most recent survey data suggests that the percentage of students reporting that Wesleyan had contributed “very much” or “quite a bit” to their quantitative skills may be trending upwards, though it will take a few more years of data collection to determine whether this is a sign of a durable change.

[Insert graph]

That recent graduates are more likely to acknowledge Wesleyan as contributing to enhanced quantitative literacy skills suggests that Wesleyan may have effectively broadened exposure to quantitative methods for Wesleyan’s student body in recent years (see figure above). This may have occurred because of the QAC curricular expansion but also because the percentage of students completing majors in Division III has grown from 19% in 2000 in 42% in 2016.^[9] But what about the students who do not acknowledge such a contribution? Who are these students and what happens to them? Do students with no relationship with the QAC or a quantitatively-inclined major leave Wesleyan with the quantitative literacy skills we would hope for all liberal arts graduates? Do they develop these skills in other ways (leadership in campus organizations, internships, etc.)? Assessment projects that strive to answer these questions might inform new pedagogical initiatives and shape advising practices.

Mapping = navigating complex environments. This competency bears on Wesleyan’s distinctive combination of strong curricula/programming in the creative arts and high-level opportunities to conduct research in the STEM fields. By “mapping” we mean the ability to:

• Examine the relationship of objects and spaces in the material and imagined world
• Develop tools to create, manipulate, and navigate constructed and natural environments
• Chart movement through and interactions with space and its consequences.

Courses in the arts (e.g., dance, studio art, and art history), the natural sciences, and mathematics are potential starting places for students wishing to work on this competency — as is our new Digital Design Studio. Courses offered by the new College of Integrative Sciences (in experiential design and in engineering) and the Center for the Study of Public Life (in design thinking and human-centered design) may also contribute to these mapping skills. Mapping is first on the list of our competencies, and we discuss it here last, not to downplay its importance, but to highlight its novelty. But with no self-reported data to begin our inquiry, we will need to look to more qualitative measures to inspire our assessment work.

E-Portfolios

Wesleyan is exploring the use of e-portfolios in pursuing a number of important goals, including: assessment of student-learning outcomes; promotion of integrative learning; and facilitating impactful, holistic discussions between advisors and advisees that link students’ intellectual and career goals with course and major selection. The current strategy is to support the faculty and students who already use e-portfolios while seeking other opportunities to use e-portfolios to meet pedagogical, curricular, and assessment needs. For example, Romance Languages faculty have for many years asked students to build e-portfolios in Mahara to document language learning within specific courses. This use of e-portfolios — documenting growth in language skills and encouraging student metacognition — could be emulated in any number of subject areas to document progress in other skills (e.g. writing, intercultural, etc.).

The Associate Director of Assessment is proposing an e-portfolio course that students could take in conjunction with their other courses and activities. This course would afford sophomores, juniors, and seniors the time to review what they have achieved both inside and outside of the classroom. Guided by the AAC&U’s LEAP rubric for integrative learning, this course would appeal to faculty and offices that want their students to prepare for (or debrief after) high-impact educational experiences (e.g. service learning, internships, comprehensive exams). These e-portfolio tools would be helpful to students with respect to writing personal narratives, making connections between courses, and metacognition. They also could be helpful to the university in better understanding what is happening to students: What kinds of assignments do they receive and what do they learn from doing them? How do assignments in one course relate to assignments they complete in another? How are co-curricular and extracurricular experiences interwoven with academic ones? What knowledge and skills do students wish to acquire, and how do these align with what they have acquired? How do these align with the four competencies Wesleyan is now promoting? How well do students perform the sort of integrative learning Wesleyan hopes for all of its graduates? While the pilot phase of this e-portfolio course would focus on designing a pedagogical structure that is most helpful to students, future iterations might contribute in crucial ways to knowledge about Wesleyan and its students.

Grading

As part of a larger focus on assessment and learning outcomes, the provost has commenced an ongoing conversation with faculty about grade inflation and compression at Wesleyan. The first of these conversations in Fall 2016 introduced the growth in the use of “A” grades over time. The second conversation in Spring 2017 served to provide faculty with more a detailed breakdown of grade distributions at the university. Nearly two-thirds of all undergraduate grades assigned between Spring 2014 and Fall 2016 fell in the A-range (64%), while only 3% fell below a C+. There was wide variation by department (A-range grades comprised between 45% and 89% of all grades), division, and faculty type.

As part of this effort, Academic Affairs provided each department chair with grade distribution information specific to their academic unit. Faculty found these data illuminating and were receptive to the conversation. Several voiced agreement with the provost’s framing of the concern as a collective action problem both within and outside the institution. Others highlighted the challenge of generating a wide grading distribution at highly selective institutions. Is it surprising that Wesleyan’s bright, motivated, and well-prepared students perform well in their classes, especially when these classes are small and taught by faculty who carry a fairly low teaching load?

We will continue to explore grading in more depth in the 2017-2018 academic year. As this process unfolds, we may need to investigate the extent to which Wesleyan students are grade-oriented, and how this orientation, should it prove to be widespread, may hinder efforts to reduce grade inflation and impede learning. We have heard anecdotes about the problem of “grade-grubbing” but the scope of this problem is unclear. We must also monitor possible unintended consequences to a change in grading practices (such as student admission to graduate programs and effects on evaluations of course instruction).

Departmental Assessment^[10]

In 2012-2013, Academic Affairs added a section to the template for department annual reports that asks chairs to describe how they assess student-learning outcomes. Incorporating this new section into an established process began to normalize university expectations about the assessment of student learning at the department level. It has also allowed for the continual collection of information about how departments shape assessment processes to fit within their discipline’s theoretical, methodological, and practical frame.

A review of the recent annual report submissions (from 2014-2015 and 2015-2016) reveal that departments use a variety of methods to assess their students’ learning and experience in the major. It is common for Wesleyan students to participate in a culminating, high-impact educational exercise during their junior or senior year in which they draw upon their work in other courses. As mentioned in Standard 4, all majors offer some form of capstone option, often a 2-semester thesis or 1-semester essay; half require their students to complete one. Data gathered through the major certification form show that 76% of all students across all majors in the Class of 2016 complete a capstone.

In their reports, fifteen departments referenced their capstones as a main mechanism for assessing student-learning outcomes. Seven reported using multiple readers and/or external readers to assess student work in capstones — the advantage being that grades earned by students reflect more than one individual’s perspective. Eight describe requiring all their majors or a subset of majors to participate in a final presentation or performance as part of the capstone or in place of it, allowing for the collective viewing by faculty of student work. Some departments also report sharing their evaluations of capstones with others for the purpose of better gauging the contributions of a broader curriculum to those students’ learning.

As documented in the E-Series, departments rely on other formative and summative assessment mechanisms as well. For example, German Studies uses American Council on the Teaching of Foreign Languages (ACTFL) standards to administer oral exams for students who have just completed Beginning German. Most students then place into the European A2-level courses in Germany, and a few place even higher. After completing GRST 211 Intermediate German, students usually place into B1-level courses in Germany. These independent placement norms have confirmed for the department that their introductory courses prepare students well. The department also monitors its students when they study abroad, usually after completion of GRST 211, and has found that Wesleyan’s language sequence successfully prepares them for living and studying in a German-speaking environment.

Molecular Biology & Biochemistry began administering an Assessment Exercise (AE), a test evaluating students’ knowledge and analytical skills, after they had taken MB&B 208 as sophomores and then again as seniors about to graduate within the major. The test revealed a significant improvement in scores between sophomores and seniors and identified three courses—MB&B 383, 394, and 395—as most strongly correlated with this improvement. To increase the response rate associated with the AE, the department has since instituted a policy requiring senior majors to take it; the policy is reinforced by its placement on the major certification form, which ensures a 100% response rate. The department has also changed the timing of the first administration, now asking for volunteers from first-year students who have just enrolled in MB&B 181, since response rates are generally higher for first-year students at the university. With the higher response rate, the department hopes to gather better statistics for the assessment of learning within the MB&B major, as it follows individual students as they progress from first to senior year. The department points out that one challenge with this research design is that it cannot track the later performance of students who took MB&B 181 but end up not majoring in MB&B.

The multidisciplinary majors of the College of Social Studies (CSS) and the College of Letters (COL) have a long history of requiring their students to take a comprehensive exam in the sophomore and junior years, respectively. For example, in the COL, the written and oral components are evaluated by outside examiners, and these evaluations are made available to the COL director. One trend that emerged in recent years was that students were performing less well on the oral exam than on the written portion. This led the COL faculty to begin emphasizing oral presentations in classes, often by asking students to take on some of the actual teaching of class sessions. A second trend — that students’ historical reasoning was weaker than their reasoning in literature and philosophy — led the department to re-sequence its colloquia so that they begin with the colloquium on antiquity colloquium and end with the colloquium on the twentieth and twenty-first centuries. Faculty report that, following this change, they have seen COL students become more sensitive to influence and change over time.

Some departments utilize regular department processes as an opportunity to collect information about students’ intellectual interests, academic goals, and self-assessments of relevant knowledge and skills. For instance, American Studies asks its rising juniors to submit a one-page description of the concentration they will pursue and how they expect that concentration to meet both their own educational goals and those the department has for them. Second-semester seniors in American Studies are then required to submit a one-page assessment that is retrospective, describing how well they have met these goals. One recent finding that emerged from this effort is that seniors in the major reported that the AMST 200 course on Colonialism and Its Consequences provides a solid foundation for other AMST courses that students take later.

For its part, English sends a four-question survey to students as they declare the major and as seniors. The survey asks them to self-assess their development in reading, writing, and critical thinking, as well as their overall progress in the major. Responses to this survey are reviewed annually by all faculty in the department. These responses inform ongoing decisions about course design, assignments, and the like, and they have also led the department to provide additional advising resources for students. In the 2016-2017, the department introduced a day-long “advising check-in” in both the fall and spring semesters during which pre-majors and majors can drop in for advising prior to registering for classes. The department saw 47 students drop by the “check-in” in the fall and 57 in the spring.

Some departments, like Biology, hold informal focus groups (at celebratory lunches or dinners they host for seniors) in which they ask majors for feedback about curricular and learning experiences. In the case of Biology, students in recent years have asked for more applied statistics courses in the department and more access to labs. The latter are understood by students to be valuable educational experiences, but are often difficult to get into. Likewise, teams of faculty in American Studies hold focus groups over lunch in which they speak with students about their paired statements from junior and senior year (described above).

Other departments have formalized discussions about their goals for student learning and how curriculum and pedagogy within a department align with those goals. For example, Art History meets each spring to exchange pedagogical strategies used across course levels to promote students’ achievement of the department’s eight learning goals. The meeting allows for deeper discussion of those senior majors who do not write theses. From this discussion, the department is now focused on (1) improving such majors’ abilities to analyze formal composition in works of art in different media; (2) assessing relationships between their foreign language requirement and student research using foreign languages; and (3) improving majors’ ability to conduct research, dedicating more time to instruction in formulating research questions and methods for finding appropriate sources. They have also considered the effectiveness of 100-level courses as a foundational experience for majors, and the current mix of course requirements for both Art History majors and minors. A recent result of this annual conversation is to consider creating new courses for majors focused on the use of Wesleyan’s Davison Art Center collection. The new courses would allow students to demonstrate their acquisition of multiple departmental learning goals, including expository writing and research using both primary and secondary documents. In the spring of 2017, one professor offered a pilot of an exhibition-based course with the curator of the Davison collection. At the end of the semester, the department plans to meet to discuss their experience and to reflect on whether such a course could serve as a required proseminar for all majors.

While most departments complete the section on assessment of student-learning outcomes when submitting their annual report, there is still much work for them to do: (1) projects should be tied to stated learning goals for both the majors and non-majors; (2) students’ self-assessments should generate more research questions which can be answered through the collective review of student work; and (3) there should be more department- and university-wide discussions about the relationship between student learning, the curriculum, and pedagogy. Providing more consistent support to departments—through the sharing of ideas, tools, staffing, and expertise—will be key. Chairs have requested this support, having found it difficult to get their own assessment projects off the ground when faced with so many competing demands.

Academic Affairs has a number of ideas about how to help departments move forward with their assessments of student learning in the major:

1. Academic Affairs should include questions on the annual report template that encourage departments to describe specifically both the strengths and weaknesses they see in student performance as those are tied to stated learning goals for the major and trends over time.
2. The new Associate Director of Assessment should establish an annual timeline for connecting with departmental chairs to offer support as they prepare their annual report.
3. The new Associate Director of Assessment should design a menu of services (e.g. interviews, focus groups, research design planning, example assessment plans from departments, curricular mapping support) from which departments can select what they most need. (This might be coordinated with what CPI offers with respect to specific courses.)
4. The Assessment Task Force should create a structure for the sharing of departmental resources that are useful for both indirect and direct assessment projects (e.g. surveys questions, interview protocols, skills tests, rubrics for evaluating student works and performances).
5. The Assessment Task Force should work to design some assessment projects that link the goals of an academic department to an office that works in service of those goals (Center for Global Studies [study abroad and internships], the QAC, the Library, Student Affairs, etc.).

 

What Students Gain

Undergraduate programs

The Gordon Career Center now collects data on the “first destinations” of graduating Wesleyan students, following National Association for College and Employers (NACE) standards for data collection and reporting the data online. Of the 73% about which we have information for the Class of 2016, 71% are employed, 13% are continuing their education, 8% are seeking employment, and the remainder (9%) are either seeking educational opportunities, not seeking employment, or engaged in other activities. Among those who are employed, the most common industries in which they are employed include tech/engineering/sciences (15%), media and communications (13%), financial services (12%), leisure/arts/entertainment (11%), and consulting (10%). The Center also posts data on alumni in the health professions. The acceptance rate of alumni attempting to enter into medical, dental, and veterinary schools between 2011 and 2015 was between 65% and 75% — well above the national average of 43%.

In 2015, the university created the Office of Fellowships, Internships and Exchanges (FIE), housed within the new Center for Global Studies. The FIE offers streamlined advising services for those (graduating students and alumni alike) seeking one of 17 post-graduate fellowships and tracks applications and acceptances. In 2015-2016, there were 82 applications for these fellowships and nine final recipients (including a Fulbright, a Keasbey, and a Watson).

While such data can give a sense of what happens to some of our students immediately upon graduation, the most important tool here is the Alumni Survey. This survey is administered every four years (at least) to the alumni 10 years out from graduation; these alumni can be expected to be in positions where the long-term impact of Wesleyan’s unique liberal arts education will be visible. The most recent of these surveys was conducted in 2013 and showed that roughly two-thirds of the Class of 2003 reported having enrolled in some sort of graduate program. Education (Higher Ed more so than K-12) was the most common field of employment; also notable were “politics, public policy, and advocacy” and “media, journalism, and publishing.” (University records kept on alumni suggest that even more alumni are working in the field of education than the Alumni Survey suggests.)

However, it is important to note that this survey had a response rate of only 35% and certainly suffers from response bias. Analysis of Senior Survey responses for the same class indicates that we know less about the post-graduate employment experience of those who were dissatisfied with their undergraduate experience.

Graduate programs

As noted in the Response to NEASC Area of Emphasis: Graduate Programs, in 2014-2015, the Graduate Program began to administer an exit survey to students completing their graduate work. This survey asks about graduate student satisfaction with general aspects of the program experience (e.g. instruction, advising, examinations, availability of degree requirements, etc.) as well as more targeted questions about the research experience (related to writing and defending the thesis or dissertation) and related outcomes (whether single-authored or co-authored research has been submitted/accepted for publication, job placement within academia or elsewhere, or placement in another graduate program). Since we have such a small number of graduate students, we waited to analyze the survey results until we had collected three years of data collected. (Insert more here when this is analyzed.)

Summary

Since 2012, Wesleyan has worked to improve how it collects and digests information about the educational effectiveness of the institution and the success of both our undergraduate and graduate students. We recognize that we still have much work to do in this area in terms of conceptualizing and implementing both ongoing and periodic assessment projects that are both feasible and provide credible and usable insights to multiple campus constituencies, particularly about student learning outcomes. However, we are excited about the new organizational structures we have in place to assist us as we look to supplement our traditional data sources and assessment methods with new ones that honor the mission and character of our institution, that create spaces for collaboration with faculty and staff members across campus, and that allow us to better understand the experiences and needs of our students.

^[1] Since Master’s programs are only two years in duration, we do not calculate retention rates.

^[2] Response rates for recent administrations of these surveys are as follows: 2017 Survey of New Students (74% of entire first-year cohort), 2015 Enrolled Students Survey (42% of entire undergraduate student body), 2015 and 2016 Senior Survey (59% and 60% of entire senior cohort, respectively), 2013 Alumni Survey (35% of 1973, 1983, 1993, 2003, and 2008 cohorts).

^[3] Typically, students who wish to receive honors in a particular major are required to complete both Stage I and Stage II of the General Education Expectations.

^[4] 2016 Survey of New Students.

^[5] Approximately 32% of those invited to take the survey (about 157 students of the 485 who enrolled in Fall 2015) began and completed the survey.

^[6] One striking pattern that emerged in students’ comments in this survey was their desire for assistance in learning how to write a research paper. While this desire extends far beyond the “expression” competency, it confirms what we have heard anecdotally from upperclassmen beginning to work on capstone projects: that they wish they had learned about library resources earlier in their college careers. The library is trying to work with faculty to integrate more formally library instruction into lower-level courses.

 

^[7] Dierker, Lisa, Jennifer Cooper, Jalen Alexander, Arielle Selya, and Jennifer Rose. 2015. “Evaluating Access: A Comparison of Demographic and Disciplinary Characteristics of Students Enrolled in a Traditional Introductory Statistics Course vs. a Multidisciplinary, Project-Based Course.” Journal of Interdisciplinary Studies in Education 4 (1): 22–37.

^[8] Dierker, Lisa, Jalen Alexander, Jennifer L. Cooper, Arielle Selya, Jennifer Rose, and Nilanjana Dasgupta. 2016. “Engaging Diverse Students in Statistical Inquiry: A Comparison of Learning Experiences and Outcomes of Under-Represented and Non-Underrepresented Students Enrolled in a Multidisciplinary Project-Based Statistics Course.” International Journal for the Scholarship of Teaching and Learning 10 (1): 1–9. doi:10.20429/ijsotl.2016.100102.

^[9] This percentage does not include students majoring in psychology.

^[10] For the sake of readability, we refer to all departments, programs, and colleges at Wesleyan as “departments” in the section that follows.