Folks:

The posting below gives some excellent tips on writing a variety of examination questions. It is from Chapter 8 Exams: Asking Questions That Provide the Answers You Need, in the book, Meaningful and Manageable Program Assessment: A How-To Guide for Higher Education Faculty, by Laura J. Massa and Margaret Kasimatis. Published by Stylus Publishing, LLC 22883 Quicksilver Drive Sterling, Virginia 20166-2102. https://sty.presswarehouse.com/books/features.aspx Copyright © 2017 by Stylus Publishing, LLC. All rights reserved. Reprinted with permission.

Regards, 

Rick Reis

[log in to unmask]

UP NEXT: Universities Grapple with Donor Behavior

Tomorrow’s Teaching and Learning

---------- 2,943 words ----------

Test-Writing Tips

Before we get into tips for writing specific types of exam items, like true-false items, we want to share four general tips for writing your exam that apply regardless of the types of exam items you choose to include. The first is to write exam items that are clear and to the point. This will help students to understand exactly what you are asking so that they can give you their best answer. Items that are written clearly and directly will also help to ensure that you are testing what students know, rather than their ability to keep track of all the information in a question. As humans we are limited in the amount of information we can keep track of and process at any given time. This limited capacity is a central component of the way our memory system functions (Baddeley & Hitch, 1974). When we load up our system with an attempt to keep track of the numerous details in a complex problem, we tend to have very little capacity left to manipulate that information in order to solve the problem. This is known as cognitive overload (Sweller, 1998). Simple ways to reduce the cognitive load of an exam question include cutting out extraneous details, like unnecessary information or visuals; providing diagrams for spatially organized information; and providing cues or signals that focus attention on necessary information, like the use of bold font to indicate a key word or phrase (Clark, Nguyen, & Sweller, 2006). When you ask faculty and/or students to review your exam items, you can ask them to identify any items that appear to include unnecessary or distracting details and use that feedback to clarify the phrasing of your exam items. 

Second, when you are writing your items, especially when you are focusing on writing succinctly, it can become tempting to ask students only for basic recall of information, such as by asking students to provide the date for a particular historical event or to provide the correct term when you give them the definition. This is especially tempting to do when writing closed-ended items like multiple-choice questions. If the learning outcome you are testing specifies that this level of learning (i.e., recognition or recall) is what students need to demonstrate, then this type of item is perfectly appropriate. If, however, you have a learning outcome that specifies students will be able to do something more complex with their learning, such as apply it to new situations or use it to evaluate new information, then you will want to write test items that ask students to do these things. These types of items require what is known as problem-solving transfer (Mayer & Wittrock, 1996). Most of the time, faculty tend to think of this type of transfer test as being limited to open-ended essay questions, but it is not. You can also ask students to demonstrate transfer, or that deeper learning, in closed-ended questions. For example, in a multiple-choice test item you can provide a description of a situation and ask students to identify the theory that is illustrated in the situation. In a label-a-diagram type of item you could provide a diagram of a system students have learned, describe a problem that has occurred in the output of the system, and then ask students to indicate on the diagram what component (or components) of the system are the most likely culprits of the described problem, and even to explain why they believe this to be the case. The point is not to limit yourself to only asking students to demonstrate the most basic levels of learning when you use an exam for assessment. 

The third thing to consider when you design your exam is that each item should stand alone. After all, each item should be designed to test a unique component of your student learning outcome. If the items are set up so that the answer to one is the key to answering the next, and so on, then a student who makes an initial mistake will have that mistake perpetuated over the course of all related items. If the exam is graded, then this will unfairly penalize the student repeatedly for the initial error. Beyond that, for your assessment purposes, it will become difficult to know if students have not learned a certain component of your learning outcome or if they just made an error on an early item that prevents you from seeing their abilities on subsequent items. If you feel that the only way to design an exam for your outcome is to set it up so that the items build on each other, then consider either setting it up or scoring it a little differently. A modified setup for this type of item is to tell students for each item to assume a specific answer to the previous question (obviously, you would not give them the correct answers). This would guarantee that all of your students would start each problem from the same place so that you could focus on scoring their final answers. Alternatively, modified scoring would require looking at the process followed to solve the problems in addition to whether students get the correct answers. This way you will be able to determine if students are not capable of doing what you ask or if they just made an error somewhere along the way in the exam. 

Fourth, it is essential to include simple, succinct directions in your exam. Your directions should indicate how to record answers (e.g., “Using capital letters, write your answer on the line provided”), and whether to show work on problems (e.g., when solving a math problem). If your questions require students to write more than a simple letter when responding, then you might ask them to write neatly. Our favorite instruction on short-answer and essay questions is, “Please write neatly. If I cannot read your answer, I will assume it is wrong.” We find that this instruction encourages students to write clearly enough that their answers can be read. In addition, assuming your exam is something students will earn a grade on, your instructions should indicate the point value of each item. On a graded exam, students can use this information to determine which items to focus their efforts on given their time and knowledge constraints. For assessment purposes, you might consider using the point values of items to reflect the importance of tested concepts. For example, if an essay item on your exam is written to capture understanding of an essential major theory, then that item should be worth more points than a true-false item that captures knowledge of a single fact. 

If you are going to use multiple types of items on your exam, such as a mixture of multiple-choice, matching, and essay items, then consider grouping those item types into sections and placing instructions at the start of each section. A possible benefit of using multiple item types is that some outcome elements may be best tested using one type of item, such as short answer, while other outcome elements may be best suited to another item type, like multiple-choice questions. In addition, students often feel that they are better at answering one type of item over another, so by including multiple types of items, you are potentially reducing student perception that their abilities relevant to a particular type of test question are influencing their test performance. 

Best Practices for Selected-Response Items

Selected-response items are those where students select a response from a set of options. These include multiple-choice items, true-false items, and matching items. These types of items are most commonly used to ask students to demonstrate their mastery of factual information, but they can be used to ask students to demonstrate much more complex thinking. For example, you can give students a set of data and ask them to make inferences about it, provide a problem to solve and ask students to select the correct answer, describe a causal situation and ask students to determine a possible effect, or provide them with a graph and ask them to conclude what it reveals. The limits of selected response items are really up to your imagination and creativity. 

A major benefit of using selected-response items is that they can be scored objectively. This can save a lot of time in your assessment process. In fact, if your institution has the resources, you can even use machines to do the scoring. A limitation of selected-response items is that writing them well is not so simple. Here we provide a few guidelines for writing the three major categories of selected-response items. 

Multiple Choice 

Multiple-choice items include a stem (i.e., the question or incomplete statement) and a set of alternatives (i.e., the response options) to choose from. Good multiple-choice items are generally more difficult to write than other types of test items, as you have to come up with both the stem and a set of plausible alternatives, which takes a bit of time and skill. 

When writing the stem for a multiple-choice item, you want to write both concisely and directly, describing a single situation or problem. If you use words like best or sometimes, then you want to make sure to highlight those words, such as by using italic or bold font or all capital letters, to help make sure students see the essential word. As a general rule, you want to avoid negatively worded stems. Negatively worded stems include words like non, never, not or except, such as “All of the following are correct except.”  Negative stems can be very tempting to write as they require you to come up with only one incorrect alternative response option. When a negatively worded stem tempts you, it may help to remember that asking students to identify an incorrect answer is not the same thing as asking them to identify a correct answer from among plausible alternatives. Another possible problem that can arise when using a negative stem is the double negative. That is, if your stem is negative, and any of your response options are negative, then you have created a confusing test item that becomes difficult to answer. In general, any item that requires more reasoning about the logic of the test item than it requires outcome-related knowledge is a poorly written item. 

Most of the time multiple-choice items include a set of four possible alternatives, although this is not a hard-and-fast rule. You could decide to include any reasonable number of alternative response options, likely between three and six given what we know about the limits of our cognitive load capacity. When you write the alternative response options for a multiple-choice item, you want to include only plausibly correct options. Anytime you include a nonplausible response option, you reduce the difficulty of the item. For example, let’s say your items include four alternatives each. This means students, if they were merely guessing, have a one-in-four chance of selecting the correct answer. If one of your alternatives is obviously nonplausible, such as listing a famous cartoon character as one of the four alternatives when asking students which theorist would have best predicted an example in the stem, then students will immediately eliminate the cartoon character as a choice and will have a one-in-three chance of selecting the correct answer on that item. Your nonplausible alternative means that students have a much better chance of guessing the correct answer, and your exam items are no longer consistent in their level of difficulty. 

Other guidelines for writing alternative response options are really about making sure that you are testing achievement of the student learning outcome rather than your students’ test-taking ability. These include making sure that your alternatives are grammatically consistent with the ending of your stem. For example, if your stem ends in a or an then this is a clue to which alternatives can be eliminated as options. Students know that if your stem ends in an incomplete sentence ending with the word a then the correct answer must begin with a consonant, and they should eliminate any alternatives beginning with vowels as plausible alternatives. The same kind of thing can happen when a stem ends in is or are. The obvious solution here is to end the stem in a way that captures multiple possibilities, such as a/an or is/are. 

Another guideline for writing alternatives is that they should be roughly consistent in length. Most students have been taught that alternatives for an item that are considerably longer or shorter than the others tend to be the correct choice, and in our experience this is usually right. 

Finally, we advise you to avoid “None of the above,” “All of the above,” or combined responses (e.g., “Both A and B are correct”), as these tend to test partial knowledge and reasoning ability. Typically, you do not have to know all of the information represented by the alternatives to get the correct answer when these types of response options are included. 

True-False

For true-false items the student is to indicate whether a statement is true or false. When writing true-false items most of us are tempted to focus on small, factual details from material that was covered in one class or mentioned in a textbook. If these kinds of details are not critical to the student learning outcome you are assessing, then it is inappropriate to include them as test items for assessment. 

Another thing to avoid when writing true-false test items is the negative statement. A negatively phrased true-false item is incredibly confusing as it requires the test taker to consider a double-negative situation. This is very difficult for students to try to reason through, which means you are not so much testing their knowledge of the outcome but are testing reasoning abilities. In addition, most of the time in this situation, it’s possible to construct a convincing argument for a true answer and for a false answer, which means you will need to eliminate the item. 

Absolute statements, like “always” or “never,” should also be avoided in writing true-false statements. Students are generally aware that nothing is true all or none of the time, making these absolute statements a dead give-away as false. On the flip side, use qualifiers, like “usually” or “seldom,” liberally. Many students have been taught that these types of qualifiers mean that a statement is true, and that will be such a student’s default answer if he or she does not know the information being tested. By using qualifiers in both true and false statements you can reduce students’ ability to use these clues as ways to determine the correct answer and be better able to test their actual knowledge of concepts. 

It is also important to avoid including more than one idea in a true-false item. As students are allowed to provide only one answer per item, then all of the ideas in the statement are either true or false. This means that students have to know only one of the ideas in order to answer the item correctly. If they know that one part is true, then the answer must be true, and vice versa. This means you are not testing knowledge of both pieces of the statement. 

Finally, consider balancing the number of true and false statements and presenting them in a random way. This will reduce any potential benefit of guessing. 

Matching

A matching item consists of two parallel lists of words or phrases that require students to match items on one list with items on the second list. It is very tempting when writing these to make one giant matching problem out of all of the things that you want to test students on, such as by listing all of the outcome-related terms in the first list and their definitions in the second list. This approach can create a potential cognitive overload for students by asking them to keep track of a great deal of information as they read through the lists. We have four tips that you can use to improve your matching items. 

First, when you construct matching items you want to include no more than about seven items in each list. This will help prevent cognitive overload, as students should be able to keep track of what’s in each list as they work through the problem. 

Second, make sure that the entire item fits onto one page. If the matching item splits over multiple pages, then students have to turn pages back and forth to see all of the listed items, which unnecessarily taxes the memory system and makes it difficult to answer the item. Remember, your goal is to test learning, not the cognitive load capacity of your students. 

Third, make sure all components included in a particular matching item are related or come from the same subcategory of the knowledge being tested. You want to set up the problem so that students have to really understand the differences between the listed items in order to select the correct response. If the listed items come from different categories, then students only have to roughly know which items belong to which category in order to select the correct responses. 

Finally, consider including an imperfect match between the number of items on one list and the possible number of responses on the other list. This will prevent students from using partial knowledge and the process of elimination to determine correct answers. If you use this approach, be sure to specify in your instructions that you can use responses more than once, and that some responses may not be used at all. 

References 

Baddeley, A.D., & Hitch, G. (1974). Working memory. Psychology of Learning and Motivation, 8, 47-89, doi:10.1016/S0079-7421(08)60452-1 

Clark, R.C., Nguyen, F., & Sweller, J. (2006). Efficiency in learning: Evidence based guidelines to manage cognitive load. San Francisco, CA: John Wiley. 

Mayer, R.E., & Wittrock, M.E. (1996). Problem-solving transfer. In D.C. Berliner & R.C. Calfee (Eds.), Handbook of educational psychology (pp. 47-62). New York, NY: Macmillan. 

Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12, 257-285, doi:10.1207/s15516709cog1202_4