MEASUREMENT RESEARCH ASSOCIATES
TEST INSIGHTS
March 2009
Greetings,
 
Certification examinations are usually meant to test the capability of a candidate to practice safely and effectively in their chosen field.  While knowledge is the foundation for practice, items that test the ability to apply that knowledge through interpretation and problem solving provide face and content validity.  

Tara McNaughton
Manager, Test Development and Analysis

Multiple Choice Items: Cognitive Skills Tested

The purpose of a multiple choice item is to measure candidate ability with regard to a specific content area.  A multiple choice item has a stem which asks a question, describes data or presents a situation.  The responses include a keyed correct response and three or four distractors or foils. The way the item is framed and the type of response required determines whether the item is Recall, Interpretation, or Problem Solving.

Recall items simply ask candidates to recall or recognize a fact.  Interpretive items require candidates to use their base of knowledge to interpret data or other information and come to some conclusion.  Problem solving items require the candidate to assess a situation, synthesize with information from their base of knowledge, and then correctly solve a problem or make a decision.  The distribution of items among the three item types is an issue that bears some consideration.  

First, most certification examinations are intended to evaluate the capability of the candidate to practice.  Is this measured better through asking questions that test specific bits of knowledge or is it more useful to structure items so that they require the candidate to apply their knowledge in a specific scenario?  Second, problem solving items are often longer than recall items, consequently requiring more time to read, interpret and answer.  Therefore, a test of predominately problem solving items could require more testing time.  Third, it appears that problem solving items are more difficult to write because there is more likely to be disagreement concerning the correct response, or there actually is more than one way to go about solving that problem.  Consequently, to insure clarity, problem solving items take more time and effort to write.  Fourth, there is a fine line between how much is enough information and how much is too much.  Problem solving items must be written succinctly, yet provide sufficient information to answer the questions.  This often involves a great deal of thought and input from more than one person.

Interpretation items are often easier to write in association with a visual.  For example, many medical and dental specialties require that the candidates be able to read x-rays of various types.  Thus, it is fairly easy to find a good x-ray and ask the candidate to interpret the x-ray and reach a diagnosis.  If the treatment plan is requested in addition to interpreting the x-ray, the item becomes problem solving.

The usual recommendation is that items be divided as follows: 33% recall; 33% interpretation; and 33% problem solving.  There is no data to support the validity of this distribution.  However, logic suggests that when items are distributed this way, candidates are measured on more than just the ability to recall facts.  They also must apply these facts in appropriate circumstances. This causes the items in the test to be viewed as closer to practice, but within the context of the multiple choice exam.  One interesting observation is that problem solving items are usually no more difficult statistically than recall items overall.

Measurement Research Associates, Inc.
505 North Lake Shore Dr., Suite 1304
Chicago, IL  60611
Phone: (312) 822-9648     Fax: (312) 822-9650



Coming Rasch-related Events
Jan. 25 - March 8, 2023, Wed..-Wed. On-line course: Introductory Rasch Analysis (M. Horton, RUMM2030), medicinehealth.leeds.ac.uk
Apr. 11-12, 2023, Tue.-Wed. International Objective Measurement Workshop (IOMW) 2023, Chicago, IL. iomw.net
June 23 - July 21, 2023, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
Aug. 11 - Sept. 8, 2023, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com