TBers: The posting below looks at the contributions capstone courses can make to student assessment. It is by Catherine White Berheide, professor of sociology, Skidmore College. The article is from Peer Review, Spring 2007, Vol. 9, No. 2 http://www.aacu.org/peerreview/. Peer Review is a publication of the Association of American Colleges and Universities [www.aacu.org/peerreview] Copyright © 2007, all rights reserved. Reprinted with permission. Enjoy and hope to see you at our next Teaching Breakfast - Nov. 7, 2007 at 8 am in Morris Hall Doing Less Work, Collecting Better Data: Using Capstone Courses to Assess Learning By Catherine White Berheide, professor of sociology, Skidmore College I have never met a faculty member who was excited about doing assessment, although rumor has it they exist. In fact, most have been resistant if not downright hostile to the notion. I fall in the resistant category. I have too much work to do to welcome any new task. Surprisingly, the wrong reason--minimizing the additional work--has led to the right way to do program assessment. Analyzing work students produce in the capstone is simply easier than most other assessment options. Fortunately, it also provides better measures of student learning. Since I am not the only faculty member to have come to this conclusion, capstones are becoming central components of assessment plans. Using Capstones to Assess Undergraduate Education The capstone course provides a venue for "assessing how successfully the major has attained the overall goals" (Wagenaar 1993, 214). Indeed, according to Rowles et al. (2004), assessment is the primary organizing principle of some capstones. As Black and Hundley note, when students look back on their four years of college in a capstone course, they "provide invaluable information to faculty about the quality of instruction and of programs" (2004, 3). Many programs are taking advantage of this rich source of data (Berheide 2001; Brock 2004; Forest and Keith 2004). National surveys of departments reveal that in political science as well as in sociology, capstones are the most common assessment (Kelly and Klunk 2003; Spalter-Roth and Erskine 2003). Henscheid (2000) finds that almost half of 707 regionally accredited colleges and universities use capstones as part of their institution's assessment program. While Henscheid also finds that smaller colleges and universities are more likely to use capstones for assessment than larger ones, at the University of Washington, about 60 percent of the departments use "some kind of senior experience--including capstone courses, design courses, and senior seminars--to evaluate student's learning in the majors" (Beyer 2001, 1). At Valdosta State University, nineteen of twenty-four academic units evaluate performance in capstone courses as a method of assessment, making it the third most frequently used method behind final exams and evaluation of course presentations (Yates 2004). Similarly, at Seton Hall, twenty-two out of thirty-three academic units use capstone courses as part of their assessment programs. Across disciplines, private institutions are more likely than public ones to use products from capstone courses to assess undergraduate education. Assessing Capstone Products Currently departments use capstone products to assess their majors in a variety of ways, ranging from rudimentary to rigorous. Beginning at the most basic level, some departments require students to publicly present their work as an exhibition, performance, poster, etc. (Bachand et al. 2006, 21). These displays "provide the most direct and most unfiltered picture of students' capabilities" (Hartmann 1992, 128). When these presentations are judged in some way, the assessment process has moved to the next stage. For example, some institutions--including Saginaw Valley State University and Skidmore College, where I teach--submit projects for presentation at conferences or to undergraduate paper contests, providing external validation of the quality of student work. Some programs, including the engineering programs at Saginaw Valley State University, even use external evaluators to "grade" the projects. Best practice, though, involves going a step further to analyze the projects systematically for the evidence they provide about program quality and to use that evidence to make curricular improvements. For example, the sociology department at the University of Wisconsin--Milwaukee uses five Likert scale items to assess how well the capstone papers demonstrate achievement of the department's learning goals (2006). A more elaborate approach involves applying an existing rubric, such as Primary Trait Analysis (Jervis and Hartley 2005), or a locally developed one (Cappell and Kamens 2002) to capstone products. This more systematic approach can provide useful insight into the strengths and weaknesses of the curriculum. A Case Study Having dragged our feet as long as we could, my departmental colleagues and I finally were forced to conduct an assessment in spring 2003. We reluctantly agreed to use senior seminar papers for our program assessment because all the other alternatives looked like more work. We chose the theory goal because we were already concerned about the issue. The two sociologists teaching the required theory course examined one strong, one average, and one weak paper. This first stab at assessment led to three main conclusions: 1. All three papers, including the weakest one, demonstrated "basic facility with many of the crucial concepts in social theory." 2. The theory goal needed to be revised. 3. The department needed to teach the connection between theory and methods not only in the theory and senior seminar courses, but also in the introductory, methods, and at least some elective courses. (Brueggemann 2003) The following year, the sociologists who teach statistics and research methods evaluated how three more papers achieve our methodological goal--concluding that "students generally succeed in achieving our methodological goals" (Fox and Karp 2004, 7). They made several recommendations "to strengthen further an already effective program," including suggesting that the program revise its goals. In the third year, the sociologists decided to look at how well students could articulate how the discipline contributes to understanding social life, concluding that "senior sociology majors, at all levels of ability, are applying sociological perspectives to issues of concern to them" (Berheide and Walzer 2005, 4). The 2005 assessment identifies two general areas for improvement: 1. Encourage students to be even more explicit in linking their specific concerns with implications for sociological theory and knowledge. 2. Help students to improve their ability to move from simply cataloguing findings to writing about them in prose that reflects more synthesis. (Berheide and Walzer 2005, 4) Overall, with relatively little effort, my department has learned a remarkable amount about what our students know and can do after majoring in sociology. First, we have learned that at least on these three goals, we are doing a good job. Second, we have learned that our theory and methods goals need some revision. Third, we have learned that we need to create greater "sequencing" within the major, especially around theory and methods. Even our minimal approach to assessment has provided vastly better data than we typically draw upon for making curricular decisions. In short, faculty do not have to spend a lot of time and effort to get very useful data. Other Examples A wide range of disciplines have used capstone products to assess the majors with favorable results. Some departments, such as industrial engineering and aeronautics at the University of Washington, have capstone projects evaluated by industry experts; others, such as sociology at Bowling Green State University, have them evaluated by both department members and outside experts. The sociology department at Bowling Green has found that the outside evaluator usually, but not always, agrees with inside evaluators (Bowling Green University 2007). Capstones are not just used to assess majors; they can also be used to assess general education. Some institutions, such as Millikin University and Portland State University, have interdisciplinary general education capstone requirements (e.g., Brooks, Benton-Kupper, and Slayton 2004; Rhodes and Agre-Kippenhan 2004). At Southeast Missouri State University, sixty senior seminar faculty analyzed over three hundred capstone products to assess general education goals related to information, thinking, and communication skills. They concluded that student achievement on these three learning objectives ranged from performances in which students were unable to formulate a thesis, produce an edited writing sample, or cite source material accurately to artifacts that demonstrated clear mastery of the ability to locate and use relevant source material, evaluate others' arguments and construct their own, and produce polished pieces of writing. (Blattner and Frazier 2004, 5). As a result of this assessment, faculty "have begun to redesign the writing assignments they give to students by requiring more than a single draft of papers and by specifying requirements for citation of sources and inclusion of reference lists" (Blattner and Frazier 2004, 6). Capstone experiences in the disciplines can also be used to assess general education goals. A senior thesis assessment project at my college revealed that at the draft stage before their thesis advisers have provided feedback, students have trouble specifying the question guiding their thesis, defining key concepts, and organizing it. Simon et al. also conclude that "students in the sciences and social sciences who have experience with research come to the senior thesis better prepared than in those disciplines that do not reinforce research skills" (2006, 1). According to Weiss (2002), sociology department chairs rate work in the capstone course as the second most valuable assessment tool. Moriarty (2006) finds that 51 percent of criminal justice programs consider capstones a very effective assessment instrument. One reason for the effectiveness of capstone products for assessment is that they are a direct measure of student learning. Other assessment experts (e.g., Angelo and Cross 1993; Banta et al. 1996) consider direct methods of assessment the best way to measure student learning. Capstone products are also authentic embedded assessment methods, since they are created as part of normal classroom activities. Finally, capstone products are an efficient assessment method, since they take advantage of an existing source of data. In short, capstones courses provide a venue for assessing how successful a curriculum is in achieving its learning objectives. Making Change The final step is to use the data collected about student performance to improve the major. Yates (2004) finds that, at Valdosta State University, capstone-based assessment led most frequently to the addition of new courses and other changes in curriculum as well as changes in pedagogy or course format. For example, performance in capstone courses as well as on final exams, and pass rates of licensing exams, portfolios, and juried exhibitions, led the art department to include visual assessment, analysis, and writing projects in one of its courses. Similarly, the University of Indianapolis Department of Communications has found the capstone to be an excellent mechanism for assessing the quality of its academic program. As is the case in my department, evaluating senior projects has raised concerns about the connections between the capstone and the rest of the student's course of study. According to Catchings, "the issues of alignment among curriculum, learning, and the capstone have prompted concerted efforts to improve the quality of both the curriculum and the capstone," including "redesign of department core curriculum courses in order to reinforce expectations in writing and oral communication" (2004, 7). After five years of assessing the capstone, Leach and Lang report that the department of anthropology at the University of North Dakota has added methods and theory courses to the curriculum because "our students have provided relatively weak evidence of their understanding of how theory affects observation and interpretation in scientific and humanistic research." They also report "an improvement in the clarity and strength of written and oral communication, as a result of assessment recommendations" (2006, 5). As these examples demonstrate, departments that have used capstones to assess their majors have found that it leads to improved student learning and can actually make faculty work lives easier. Assessment, therefore, is not an end in and of itself, but rather a means to an end. The end is the improvement of student learning at the individual, program, and institutional levels. Analyzing capstone projects is an efficient and effective approach to achieving that end. References Angelo, T. A., and K. P. Cross. 1993. Classroom assessment techniques: A handbook for college teachers. San Francisco, CA: Jossey-Bass. Bachand, D. J., D. Huntley, M. Hedberg, C. Dorne, J. Boye-Beaman, and M. Thorns. 2006. A monitoring report to the Higher Learning Commission on program assessment, general education assessment, and diversity. www.svsu.edu/emplibrary/ HLC%20Final%20Report%202006 %20-Web.pdf. Banta, T. W., J. P. Lund, K. E. Black, and F. W. Oblander. 1996. Assessment in practice. San Francisco, CA: Jossey-Bass. Berheide, C. W. 2001. Using the capstone course for assessment of learning in the sociology major. In Assessing student learning in sociology, 2d ed., ed. C. F. Hohm and W. S. Johnson, 164--76. Washington, DC: American Sociological Association. Berheide, C. W., and S. Walzer. 2005. Sociology assessment: Contributions of sociology to understanding social life. Saratoga Springs, NY: Skidmore College. Beyer, C. H. 2001. Assessment in the majors, 2000. www.washington.edu/oea/pdfs/ reports/OEAReport0103.pdf Black, K. E., and S. P. Hundley. 2004. Capping off the curriculum. Assessment Update 16 (1): 3. Blattner, N. H., and C. L. Frazier. 2004. Assessing general education core objectives. Assessment Update 16 (4): 4--6. Bowling Green State University. 2007. Assessment reports. www.bgsu.edu/offices/ assessment/Reports.htm. Brock, P. A. 2004. From capstones to touchstones: Preparative assessment and its use in teacher education. Assessment Update 16 (1): 8--9. Brooks, R., J. Benton-Kupper, and D. Slayton. 2004. Curricular aims: Assessment of a university capstone course. The Journal of General Education 53:275--87. Brueggemann, J. 2003. Assessment in sociology: Theory component. Saratoga Springs, NY: Skidmore College. Cappell, C. L., and D. H. Kamens. 2002. Curriculum assessment: A case study in sociology. Teaching Sociology 30:467--94. Catchings, B. 2004. Capstones and quality: The culminating experience as assessment. Assessment Update 16 (1): 6--7. Forest, J., and B. Keith. 2004. Assessing proficiency in engineering and technology within a multidisciplinary curriculum. Assessment Update 16 (4): 9--10. Fox, W., and D. Karp. 2004. Sociology assessment: Undergraduate training in research methods and statistics. Saratoga Springs, NY: Skidmore College. Hartmann, D. J. 1992. Program assessment in sociology: The case for the bachelor's paper. Teaching Sociology 20:25--28. Henscheid, J. M. 2000. Professing the disciplines: An analysis of senior seminars and capstone courses. Columbia, SC: University of South Carolina. Jervis, K. J., and C. A. Hartley. 2005. Learning to design and teach an accounting capstone. Issues in Accounting Education 20:311--39. Kelly, M., and B. E. Klunk. 2003. Learning assessment in political science departments: Survey results. PS: Political Science and Politics 36:451--55. Leach, M., and G. C. Lang. 2006. The not-sostony path to program assessment and, along the way, transforming a senior capstone seminar in anthropology. University of North Dakota Assessment Committee Newsletter (November): 1--7. www.und.nodak.edu/dept/datacol/ assessment/newsletter/2006nov_anth.pdf. Moriarty, L. J. 2006. Investing in quality: The current state of assessment in criminal justice programs. Justice Quarterly 23:409--27. Rhodes, T. L., and S. Agre-Kippenhan. 2004. A multiplicity of learning: Capstones at Portland State University. Assessment Update 16 (1): 4--5, 12. Rowles, C. J., D. C. Koch, S. P. Hundley, and S. J. Hamilton. 2004. Toward a model for capstone experiences: Mountaintops, magnates, and mandates. Assessment Update 16 (1): 1--15. Simon, L., D. Curley, M. A. Foley, R. Ginsberg, M. Hockenos, and D. Smith. 2006. Senior thesis assessment workshop. Saratoga Springs, NY: Skidmore College. Spalter-Roth, R. M., and W. B. Erskine. 2003. How does your department compare? A peer analysis from the AY 2001--2002 Survey of Baccalaureate and Graduate Programs in Sociology. Washington DC: American Sociological Association. University of Wisconsin--Milwaukee Department of Sociology. 2006. Assessment of undergraduate and graduate programs in sociology. University of Wisconsin--Milwaukee. www.uwm.edu/letsci/assessplans/ SociolAssess.pdf. Wagenaar, T. C. 1993. The capstone course. Teaching Sociology 21:209--14. Weiss, G. L. 2002. The current status of assessment in sociology departments. Teaching Sociology 30:391--402. Yates, C. B. C. 2004. Executive summary of the Report on Assessment Methods at Valdosta State University. www.valdosta.edu/sra/ documents/Assessment_Project_Exec_ Summary.pdf.