I agree! One aspect to consider is how important the type of question is to your content area. When the questions get "tricky" (back to Jennifer's original example of using "not" in the question") or complicated, sometimes students are unable to demonstrate the content area knowledge that they have because they get lost in the reading comprehension piece versus the content. If one chooses to use questions like these, explicit instruction prior to test day with an example or two would probably be helpful. Kim E. Griswold, Learning Specialist and Academic Coach The Center for Academic Development and Enrichment 224 Alumni Hall S.U.N.Y. Oneonta Oneonta, N.Y. 13820 (607) 436-3071 From: Teaching Breakfast List [mailto:[log in to unmask]] On Behalf Of Pence, Harry Sent: Saturday, March 11, 2017 9:49 AM To: [log in to unmask] Subject: Re: Multiple Choice Questions- query This thread reminds me of a colleague who used to give multiple choice questions with 17 options (I think it was). In addition to A, B, C, and D, there was all of the about, none of the above, A and B but not C and D, A and C, but not B and D, etc. I found them to be useful only as an exercise in who was the best test taker, but useless in measuring what the student knew. Harry Harry E. Pence SUNY Distinguished Teaching Professor Emeritus SUNY Oneonta ________________________________ From: Teaching Breakfast List <[log in to unmask]<mailto:[log in to unmask]>> on behalf of Wilkerson, Bill <[log in to unmask]<mailto:[log in to unmask]>> Sent: Friday, March 10, 2017 1:47:40 PM To: [log in to unmask]<mailto:[log in to unmask]> Subject: Re: Multiple Choice Questions- query Jennifer, On the which is incorrect questions. I used to car pool with an education psychologist who studied test construction. He was not a fan of these questions, nor of answers like "all of the above" and "A and B". I don't remember all of the details, but he said that are confusing to students and requires students to shift gears in a way that is harder than we might guess. I have enjoyed the thread. More generally I have found the learningscientist.org blog and resources useful in my teaching and as a resource for students. Bill ________________________________ From: Teaching Breakfast List <[log in to unmask]<mailto:[log in to unmask]>> on behalf of Crouse-Powers, Amy <[log in to unmask]<mailto:[log in to unmask]>> Sent: Friday, March 10, 2017 8:31:03 AM To: [log in to unmask]<mailto:[log in to unmask]> Subject: Re: Multiple Choice Questions- query Jennifer: In the study skills courses at CADE and in our tutor training workshops, we talk about studying for the type of test that will be given. When some students are given terms to learn, I find and the tutors find, that some (esp. freshmen) often study as though the professor is going to give them a matching test or a fill-in-the-blank with a word bank test. They need to study in a way that will help them recall and use the information in a variety of contexts. A tell-tale sign for bad studying is those infernal flash cards with a term on the front and a simple definition on the back. At the very least, we try to get them to increase the complexity of the information on those cards (I've attached a little handout we give them with a pack of a few large blank note cards to get started). The tutors are trained to help them look at the information to apply a variety of levels of thinking to it to consider the variety of questions that might be asked. I tell the tutors to ask the students to take notes in the Cornell style (with the 2.5" margin on the left) and bring the notes to tutoring and work with them on making up questions about that material. (The students go straight to "Define X" as the question they think will be asked, but the tutor is trained to help them combine multiple ideas, compare and contrast, etc.) I'll also put in a plug for academic coaching. The CADE professional staff can work with students on these skills. They can either sign up for a regular appointment that will occur over several weeks or they can do a one-time drop-in with an academic coach (this semester, that's offered in the library Rm 102 on Thursdays from 3-5 p.m.) The poster I attached for academic coaching is set for large-sized printing, but you can at least see what we are offering. Amy Crouse-Powers On Mar 7, 2017, at 11:31 AM, Withington, Jennifer <[log in to unmask]<mailto:[log in to unmask]>> wrote: Dear Colleagues, I was going over the most-missed questions on my last freshman exam and a few thoughts struck me. I am sharing because I think that many of you have run into similar outcomes. I have noticed that a question asking students in one fashion or another for the wrong/incorrect answer is more likely to be missed by a majority of the students (e.g.: which would you fail to find?, which statement is incorrect?, which is not _____?, etc.). I feel that this is due to a combination of (1) failure to study well, (2) studying in a fashion which does not help them think these through, and (3) loosing track of the fact the question asked for the incorrect answer when they are so used to looking for the correct one. The other kind of question they have great difficulty with are the ones that ask them to reason through information I have provided in the question, couple it to facts they were to have learned for the exam, and by forward thinking as well as elimination of incorrect answers get to the correct answer. In this case reasons 1 and 2 from above are coupled with reason (4) getting tied up in thoughts of "did we have this in our notes?", "was this something I should have studied?", etc. However, I am also starting to wonder if brain development is also a factor. I am sure many of us have heard that upper-level reasoning and thought is not fully developed in brains until people are ~20 (say junior year). So are these types of questions really above the ability of most of my freshmen students? What are your thoughts? Perhaps the Ed, Psych, and EdPsych faculty have some insights that would be useful? Appreciating your insights, Jennifer Withington Assoc. Professor of Biology SUNY Oneonta 116 Science I Oneonta, NY 13820 607-436-3421