TBers, Because the College has recently piloted online student evaluations, I have been following a discussion about his issue on the Professional Organizational Development (POD) listserv with interest. Posted below is a recent thread from that discussion that you might find interesting. I was particularly interested in the post from Mike Dabney highlighted in red) about building a culture of trust in your classrooms by using student evaluations. If anyone wants to try this using existing survey technologies let me know and I can show you how to access and use them. Mr. James B. Greenberg Director Teaching, Learning and Technology Center Milne Library SUNY College at Oneonta Oneonta, New York 13820 blog: The 32nd Square at http://32ndsquare.blogspot.com wiki: The 32nd Square at http://32ndsquare.wikidot.com email: [log in to unmask] phone: 607-436-2701 fax: 607-436-3081 IM: oneontatltc Twitter: greenbjb Yammer: https://www.yammer.com/users/greenbjb Identi.ca: http://identi.ca/jimg/all Plurk: greenbjb Myspace: greenbjb Pownce: greenbjb Friendfeed: greenbjb Spoink: greenbjb orkut:Jim Greenberg Twine: greenbjb "Ignorance is curable, stupidity lasts forever" P Think before you print! Please consider the environment before printing this email ------ Forwarded Message From: Mike Theall <[log in to unmask]> Reply-To: <[log in to unmask]> Date: Tue, 21 Oct 2008 15:39:15 -0400 To: <[log in to unmask]> Subject: Re: [POD] Improving Completion Rates of End-of-Semester Student Evaluations Mick, RE "The instructor relates to me as an individual.": The question itself is reasonable since relationships can influence learning and related factors such as motivation, effort, interest in the subject, further study, etc. From that perspective, the question has some validity. The data from this question will more than likely be reliable either indicating the strength of relationships (good to bad) or alternatively, contextual factors (e.g., class size) that impact on responses. The problem, of course, is in the way in which data are analyzed, reported, interpreted, and used. The assumption that this question can be used to compare individuals is flawed. Even when interpretive guides are used and a factor such as size in considered, there remain issues such as the implicit theories of the decision makers. Let's assume that the ratings on this item are reported. One who values relationships and sees strong scores from your class of 210 on this item, may consider your teaching exceptionally effective. "Wow, look what he was able to do in a class this big!" One who does not acknowledge the power of relationships may ignore the score entirely or may, perversely, infer that your strong ratings reflect some debasement of the discipline or pandering to lowest-common-denominator student desire for entertainment. You must be "POPULAR!!!!" Egad! In any case, the issue is less a matter of the reliability and validity of the item or the instrument, than it is the reliability of the decision makers and the validity of the bases for their decisions. I think we agree on this anyhow, so I pose the above only because I think the validity of practice deserves much more emphasis than it gets. Cheers, mike -----Original Message----- From: Professional & Organization Development Network in Higher Education [mailto:[log in to unmask]] On Behalf Of La Lopa, Joseph M Sent: Thursday, October 16, 2008 2:01 PM To: [log in to unmask] Subject: Re: [POD] Improving Completion Rates of End-of-Semester Student Evaluations I love your approach to what should be asked. Here is why. At Purdue there are two questions that all professors must ask on evaluations, regardless of what or how they teach, discipline, etc., using a scale from 5 (SA) to 1 (SD). 1. Overall, I would rate this course as: 2. Overall, I would rate this instructor as: In one course same semester I was rated 4.0, 4.6. In another same semester it was 4.1, 4.1. In another same semester it was 4.4, 4.8. The problems with these two basic questions as the standard by which to judge all faculty are so numerous I would not even know where to begin without writing a book on the subject. It gets better. My department has a one-size-fits-all set of questions to evaluate teaching of faculty regardless of class size, subject, pedagogy, etc. Here is a good example of one that really helps me....NOT...in my class of 210 students! "The instructor relates to me as an individual." Please do not fall out of your chairs laughing......................... So your questions, similar to those asked in a SGID, have high utility and to be commended as far as I am concerned. Mick -----Original Message----- From: Professional & Organization Development Network in Higher Education [mailto:[log in to unmask]] On Behalf Of Michael Dabney Sent: Thursday, October 16, 2008 2:14 PM To: [log in to unmask] Subject: Re: [POD] Improving Completion Rates of End-of-Semester Student Evaluations To Linda's helpful notes, I add one suggestion, whether you use online evaluations or continue with paper. It is my belief, from classroom experience of my own and others (unpublished) that intentionally building a culture of student trust in the evaluation process is critical to completion rates. This can never happen if only end-of-term instruments are used. Students need evidence that evaluation data is used to make meaningful adjustments to teaching practice. It would be easy for a professor to conduct a simple early-in-term formative assessment of teaching, with paper or a free online tool like Survey Monkey or Zoomerang: What am I doing that helps you learn, and that I should CONTINUE? What am I doing that makes learning difficult, and that I should STOP? What am I not doing that would help you and that I should START? Then demonstrate commitment by two critical trust-building steps: (a) share all results back to the class; (b) respond to every point. Make! changes as appropriate. Sometimes syllabus commitments were made that can not be changed. Sometimes a teacher needs time to think through the implications of a suggestion before making a commitment to change. And sometimes, students offer creative ideas a teacher can implement immediately. How often to do these formative assessments? Once, twice, several times . . . understanding that you are building a culture of trust and that by making appropriate adjustments to practice, you are sending the strong messages (a) that your students have a stake in how you conduct the class and (b) that you care deeply about their experience and their success. In this process, an added benefit is that students discover from the shared feedback that peers learn differently and value different teaching strategies: what helps me learn well is very challenging for other students. Empathy for classroom peers, trust in the assessment process, and deeper understanding of the professor's challenges, must surely be a! mong the results. Mike Michael W. Dabney Director, Teaching and Learning Center Hawaii Pacific University 1188 Fort Street, #139 Honolulu, Hawaii 96813 808-543-8048 Explore TLC's new FAQ database at http://www.hpu.edu/FacultyFAQ and TLC's web site at http://www.hpu.edu/index.cfm?contentID=9473&siteID=1 From: Professional & Organization Development Network in Higher Education [mailto:[log in to unmask]] On Behalf Of Linda Nilson Sent: Thursday, October 16, 2008 4:56 AM To: [log in to unmask] Subject: Re: [POD] Improving Completion Rates of End-of-Semester Student Evaluations Paul -- Clemson went to exclusively online student evaluations a couple of years ago. By the time we were preparing for this change, Jossey-Bass had already published Lynn Sorenson & Trav Johnson's New Directions for Teaching and Learning, No. 96 (Winter 2003), Online Student Ratings of Instruction, so I went there first and recommend you do, too. The idea from this volume that has worked well on this campus is for faculty to promise students some kind of reward for a obtaining a high response (e.g., 90%) before the last class meeting, such as food the last day or a couple of extra points (not a meaningful number) for everyone. Call it bribery, call it what you will--it works. Some faculty get 100% response with it in medium-size and ever larger classes. Since Clemson is a laptop campus, some faculty tell their students to be sure to bring their laptops to class on evaluations day, and students fill out the online form in class. Of course, a few students either forget their laptop or bring a non-functioning laptop. But this alternative appeals to those who are opposed to the "bribery" option above. Promotions, encouragement, and the like alone have not raised the response rate high enough. All the best, Linda Linda B. Nilson, Ph.D., Director Office of Teaching Effectiveness and Innovation 445 Brackett Hall, Clemson University Clemson, SC 29634 U.S.A. Tel: 864.656.4542 * Fax: 864.656.0750 [log in to unmask] * www.clemson.edu/OTEI From: Professional & Organization Development Network in Higher Education [mailto:[log in to unmask]] On Behalf Of Paul Quick Sent: Thursday, October 16, 2008 9:59 AM To: [log in to unmask] Subject: [POD] Improving Completion Rates of End-of-Semester Student Evaluations As UGA has moved away from paper-based end-of-semester student course evaluations and has adopted online evaluations, we have experienced a dramatic decrease in completion rates, mainly because students have been asked to go online out of class to do the evaluations. Though research indicates that low response rates do not necessarily have an effect on the quality and nature of the feedback that instructors receive from students, our administration would like us to investigate what other schools are doing to improve completion rates of online course evaluations. What are institutions doing to improve (or require) online student feedback on courses? I've heard that some schools require all students to go to a website where they must either submit comments or deliberately opt out of submitting comments in order to get their semester grades. Is this true? Is your institution doing this? What has been the effect? Pros/Cons? Benefits/Problems? How else do schools get substantial (perhaps not substantive) feedback? Please email me personally at [log in to unmask] Once I get an idea of what institutions are doing, I'll report back to the list. Thanks! Paul Quick, PhD Director of TA Programs Center for Teaching and Learning University of Georgia 30606 706.542.0534 ************************************************************************* You are subscribed to the POD mailing list. To Unsubscribe, change your subscription options, or access list archives, visit http://listserv.nd.edu/archives/pod.html For information about the POD Network visit http://podnetwork.org Hosted by the John A. Kaneb Center for Teaching and Learning and the Office of Information Technologies at the University of Notre Dame. ************************************************************************* ************************************************************************* You are subscribed to the POD mailing list. To Unsubscribe, change your subscription options, or access list archives, visit http://listserv.nd.edu/archives/pod.html For information about the POD Network visit http://podnetwork.org Hosted by the John A. Kaneb Center for Teaching and Learning and the Office of Information Technologies at the University of Notre Dame. ************************************************************************* ************************************************************************* You are subscribed to the POD mailing list. To Unsubscribe, change your subscription options, or access list archives, visit http://listserv.nd.edu/archives/pod.html For information about the POD Network visit http://podnetwork.org Hosted by the John A. Kaneb Center for Teaching and Learning and the Office of Information Technologies at the University of Notre Dame. ************************************************************************* ************************************************************************* You are subscribed to the POD mailing list. To Unsubscribe, change your subscription options, or access list archives, visit http://listserv.nd.edu/archives/pod.html For information about the POD Network visit http://podnetwork.org Hosted by the John A. Kaneb Center for Teaching and Learning and the Office of Information Technologies at the University of Notre Dame. ************************************************************************* ************************************************************************* You are subscribed to the POD mailing list. To Unsubscribe, change your subscription options, or access list archives, visit http://listserv.nd.edu/archives/pod.html For information about the POD Network visit http://podnetwork.org Hosted by the John A. Kaneb Center for Teaching and Learning and the Office of Information Technologies at the University of Notre Dame. ************************************************************************* ------ End of Forwarded Message