MEASUREMENT RESEARCH ASSOCIATES
TEST INSIGHTS
March 2009
Greetings,
 
Certification examinations are usually meant to test the capability of a candidate to practice safely and effectively in their chosen field.  While knowledge is the foundation for practice, items that test the ability to apply that knowledge through interpretation and problem solving provide face and content validity.  

Tara McNaughton
Manager, Test Development and Analysis

Multiple Choice Items: Cognitive Skills Tested

The purpose of a multiple choice item is to measure candidate ability with regard to a specific content area.  A multiple choice item has a stem which asks a question, describes data or presents a situation.  The responses include a keyed correct response and three or four distractors or foils. The way the item is framed and the type of response required determines whether the item is Recall, Interpretation, or Problem Solving.

Recall items simply ask candidates to recall or recognize a fact.  Interpretive items require candidates to use their base of knowledge to interpret data or other information and come to some conclusion.  Problem solving items require the candidate to assess a situation, synthesize with information from their base of knowledge, and then correctly solve a problem or make a decision.  The distribution of items among the three item types is an issue that bears some consideration.  

First, most certification examinations are intended to evaluate the capability of the candidate to practice.  Is this measured better through asking questions that test specific bits of knowledge or is it more useful to structure items so that they require the candidate to apply their knowledge in a specific scenario?  Second, problem solving items are often longer than recall items, consequently requiring more time to read, interpret and answer.  Therefore, a test of predominately problem solving items could require more testing time.  Third, it appears that problem solving items are more difficult to write because there is more likely to be disagreement concerning the correct response, or there actually is more than one way to go about solving that problem.  Consequently, to insure clarity, problem solving items take more time and effort to write.  Fourth, there is a fine line between how much is enough information and how much is too much.  Problem solving items must be written succinctly, yet provide sufficient information to answer the questions.  This often involves a great deal of thought and input from more than one person.

Interpretation items are often easier to write in association with a visual.  For example, many medical and dental specialties require that the candidates be able to read x-rays of various types.  Thus, it is fairly easy to find a good x-ray and ask the candidate to interpret the x-ray and reach a diagnosis.  If the treatment plan is requested in addition to interpreting the x-ray, the item becomes problem solving.

The usual recommendation is that items be divided as follows: 33% recall; 33% interpretation; and 33% problem solving.  There is no data to support the validity of this distribution.  However, logic suggests that when items are distributed this way, candidates are measured on more than just the ability to recall facts.  They also must apply these facts in appropriate circumstances. This causes the items in the test to be viewed as closer to practice, but within the context of the multiple choice exam.  One interesting observation is that problem solving items are usually no more difficult statistically than recall items overall.

Measurement Research Associates, Inc.
505 North Lake Shore Dr., Suite 1304
Chicago, IL  60611
Phone: (312) 822-9648     Fax: (312) 822-9650



Rasch-Related Resources: Rasch Measurement YouTube Channel
Rasch Measurement Transactions & Rasch Measurement research papers - free An Introduction to the Rasch Model with Examples in R (eRm, etc.), Debelak, Strobl, Zeigenfuse Rasch Measurement Theory Analysis in R, Wind, Hua Applying the Rasch Model in Social Sciences Using R, Lamprianou Journal of Applied Measurement
Rasch Models: Foundations, Recent Developments, and Applications, Fischer & Molenaar Probabilistic Models for Some Intelligence and Attainment Tests, Georg Rasch Rasch Models for Measurement, David Andrich Constructing Measures, Mark Wilson Best Test Design - free, Wright & Stone
Rating Scale Analysis - free, Wright & Masters
Virtual Standard Setting: Setting Cut Scores, Charalambos Kollias Diseño de Mejores Pruebas - free, Spanish Best Test Design A Course in Rasch Measurement Theory, Andrich, Marais Rasch Models in Health, Christensen, Kreiner, Mesba Multivariate and Mixture Distribution Rasch Models, von Davier, Carstensen
Rasch Books and Publications: Winsteps and Facets
Applying the Rasch Model (Winsteps, Facets) 4th Ed., Bond, Yan, Heene Advances in Rasch Analyses in the Human Sciences (Winsteps, Facets) 1st Ed., Boone, Staver Advances in Applications of Rasch Measurement in Science Education, X. Liu & W. J. Boone Rasch Analysis in the Human Sciences (Winsteps) Boone, Staver, Yale Appliquer le modèle de Rasch: Défis et pistes de solution (Winsteps) E. Dionne, S. Béland
Introduction to Many-Facet Rasch Measurement (Facets), Thomas Eckes Rasch Models for Solving Measurement Problems (Facets), George Engelhard, Jr. & Jue Wang Statistical Analyses for Language Testers (Facets), Rita Green Invariant Measurement with Raters and Rating Scales: Rasch Models for Rater-Mediated Assessments (Facets), George Engelhard, Jr. & Stefanie Wind Aplicação do Modelo de Rasch (Português), de Bond, Trevor G., Fox, Christine M
Exploring Rating Scale Functioning for Survey Research (R, Facets), Stefanie Wind Rasch Measurement: Applications, Khine Winsteps Tutorials - free
Facets Tutorials - free
Many-Facet Rasch Measurement (Facets) - free, J.M. Linacre Fairness, Justice and Language Assessment (Winsteps, Facets), McNamara, Knoch, Fan

To be emailed about new material on www.rasch.org
please enter your email address here:

I want to Subscribe: & click below
I want to Unsubscribe: & click below

Please set your SPAM filter to accept emails from Rasch.org

www.rasch.org welcomes your comments:
Please email inquiries about Rasch books to books \at/ rasch.org

Your email address (if you want us to reply):

 

FORUMRasch Measurement Forum to discuss any Rasch-related topic

Coming Rasch-related Events
Oct. 6 - Nov. 3, 2023, Fri.-Fri. On-line workshop: Rasch Measurement - Core Topics (E. Smith, Facets), www.statistics.com
Oct. 12, 2023, Thursday 5 to 7 pm Colombian timeOn-line workshop: Deconstruyendo el concepto de validez y Discusiones sobre estimaciones de confiabilidad SICAPSI (J. Escobar, C.Pardo) www.colpsic.org.co
June 12 - 14, 2024, Wed.-Fri. 1st Scandinavian Applied Measurement Conference, Kristianstad University, Kristianstad, Sweden http://www.hkr.se/samc2024
Aug. 9 - Sept. 6, 2024, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com