MEASUREMENT RESEARCH ASSOCIATES
TEST INSIGHTS
March 2009
Greetings,
 
Certification examinations are usually meant to test the capability of a candidate to practice safely and effectively in their chosen field.  While knowledge is the foundation for practice, items that test the ability to apply that knowledge through interpretation and problem solving provide face and content validity.  

Tara McNaughton
Manager, Test Development and Analysis

Multiple Choice Items: Cognitive Skills Tested

The purpose of a multiple choice item is to measure candidate ability with regard to a specific content area.  A multiple choice item has a stem which asks a question, describes data or presents a situation.  The responses include a keyed correct response and three or four distractors or foils. The way the item is framed and the type of response required determines whether the item is Recall, Interpretation, or Problem Solving.

Recall items simply ask candidates to recall or recognize a fact.  Interpretive items require candidates to use their base of knowledge to interpret data or other information and come to some conclusion.  Problem solving items require the candidate to assess a situation, synthesize with information from their base of knowledge, and then correctly solve a problem or make a decision.  The distribution of items among the three item types is an issue that bears some consideration.  

First, most certification examinations are intended to evaluate the capability of the candidate to practice.  Is this measured better through asking questions that test specific bits of knowledge or is it more useful to structure items so that they require the candidate to apply their knowledge in a specific scenario?  Second, problem solving items are often longer than recall items, consequently requiring more time to read, interpret and answer.  Therefore, a test of predominately problem solving items could require more testing time.  Third, it appears that problem solving items are more difficult to write because there is more likely to be disagreement concerning the correct response, or there actually is more than one way to go about solving that problem.  Consequently, to insure clarity, problem solving items take more time and effort to write.  Fourth, there is a fine line between how much is enough information and how much is too much.  Problem solving items must be written succinctly, yet provide sufficient information to answer the questions.  This often involves a great deal of thought and input from more than one person.

Interpretation items are often easier to write in association with a visual.  For example, many medical and dental specialties require that the candidates be able to read x-rays of various types.  Thus, it is fairly easy to find a good x-ray and ask the candidate to interpret the x-ray and reach a diagnosis.  If the treatment plan is requested in addition to interpreting the x-ray, the item becomes problem solving.

The usual recommendation is that items be divided as follows: 33% recall; 33% interpretation; and 33% problem solving.  There is no data to support the validity of this distribution.  However, logic suggests that when items are distributed this way, candidates are measured on more than just the ability to recall facts.  They also must apply these facts in appropriate circumstances. This causes the items in the test to be viewed as closer to practice, but within the context of the multiple choice exam.  One interesting observation is that problem solving items are usually no more difficult statistically than recall items overall.

Measurement Research Associates, Inc.
505 North Lake Shore Dr., Suite 1304
Chicago, IL  60611
Phone: (312) 822-9648     Fax: (312) 822-9650

Please help with Standard Dataset 4: Andrich Rating Scale Model



Rasch Publications
Rasch Measurement Transactions (free, online) Rasch Measurement research papers (free, online) Probabilistic Models for Some Intelligence and Attainment Tests, Georg Rasch Applying the Rasch Model 3rd. Ed., Bond & Fox Best Test Design, Wright & Stone
Rating Scale Analysis, Wright & Masters Introduction to Rasch Measurement, E. Smith & R. Smith Introduction to Many-Facet Rasch Measurement, Thomas Eckes Invariant Measurement: Using Rasch Models in the Social, Behavioral, and Health Sciences, George Engelhard, Jr. Statistical Analyses for Language Testers, Rita Green
Rasch Models: Foundations, Recent Developments, and Applications, Fischer & Molenaar Journal of Applied Measurement Rasch models for measurement, David Andrich Constructing Measures, Mark Wilson Rasch Analysis in the Human Sciences, Boone, Stave, Yale
in Spanish: Análisis de Rasch para todos, Agustín Tristán Mediciones, Posicionamientos y Diagnósticos Competitivos, Juan Ramón Oreja Rodríguez

To be emailed about new material on www.rasch.org
please enter your email address here:

I want to Subscribe: & click below
I want to Unsubscribe: & click below

Please set your SPAM filter to accept emails from Rasch.org

www.rasch.org welcomes your comments:
Please email inquiries about Rasch books to books \at/ rasch.org

Your email address (if you want us to reply):

 

FORUMRasch Measurement Forum to discuss any Rasch-related topic

Coming Rasch-related Events
April 26-30, 2017, Wed.-Sun. NCME, San Antonio, TX, www.ncme.org - April 29: Ben Wright book
April 27 - May 1, 2017, Thur.-Mon. AERA, San Antonio, TX, www.aera.net
April 29, 2017, Sat., 16:35 to 18:05. NCME Presidents Invitational Symposium: a new book commemorating Ben Wright's life and career, 16:35 to 18:05, San Antonio, TX, www.ncme.org
May 26 - June 23, 2017, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 30 - July 29, 2017, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
July 31 - Aug. 3, 2017, Mon.-Thurs. Joint IMEKO TC1-TC7-TC13 Symposium 2017: Measurement Science challenges in Natural and Social Sciences, Rio de Janeiro, Brazil, imeko-tc7-rio.org.br
Aug. 7-9, 2017, Mon-Wed. In-person workshop and research coloquium: Effect size of family and school indexes in writing competence using TERCE data (C. Pardo, A. Atorressi, Winsteps), Bariloche Argentina. Carlos Pardo, Universidad Catòlica de Colombia
Aug. 7-9, 2017, Mon-Wed. PROMS 2017: Pacific Rim Objective Measurement Symposium, Sabah, Borneo, Malaysia, proms.promsociety.org/2017/
Aug. 10, 2017, Thurs. In-person Winsteps Training Workshop (M. Linacre, Winsteps), Sydney, Australia. www.winsteps.com/sydneyws.htm
Aug. 11 - Sept. 8, 2017, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com
Aug. 18-21, 2017, Fri.-Mon. IACAT 2017: International Association for Computerized Adaptive Testing, Niigata, Japan, iacat.org
Sept. 15-16, 2017, Fri.-Sat. IOMC 2017: International Outcome Measurement Conference, Chicago, jampress.org/iomc2017.htm
Oct. 13 - Nov. 10, 2017, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
Jan. 5 - Feb. 2, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
Jan. 10-16, 2018, Wed.-Tues. In-person workshop: Advanced Course in Rasch Measurement Theory and the application of RUMM2030, Perth, Australia (D. Andrich), Announcement
Jan. 17-19, 2018, Wed.-Fri. Rasch Conference: Seventh International Conference on Probabilistic Models for Measurement, Matilda Bay Club, Perth, Australia, Website
May 25 - June 22, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 29 - July 27, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
Aug. 10 - Sept. 7, 2018, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com
Oct. 12 - Nov. 9, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
The HTML to add "Coming Rasch-related Events" to your webpage is:
<script type="text/javascript" src="http://www.rasch.org/events.txt"></script>