MEASUREMENT RESEARCH ASSOCIATES
TEST INSIGHTS

January 2009
Greetings!

Oral examinations have many facets that come together in the scoring process.  We've identified that different oral exam formats (standardized protocols vs. practice-based cases) can function differently. Here's a brief summary of some of those findings.

Ross Brown
Manager of Test Development and Analysis

Differences in Candidate Performance Across Oral Exam Formats

Oral exams may include practice-based cases from the candidate's practice, or standardized protocols, or both. Often the same candidates are rated by the same examiners on the same clinical skills, for both cases and protocols.

Because the skills are the same for practice-based cases and protocols, it may seem the same candidate abilities, such as correctly diagnosing or treating a patient, are being assessed in both measurement contexts. However, when candidate performance on common skills are analyzed separately and compared across cases and protocols, systematic differences in candidate performance and clinical skill difficulty become apparent. These differences in performance and difficulty are observed after the many-facet Rasch model has accounted for differences in the specific exam elements that individual candidates encountered. The table shows examples from two different examinations.  Generally, the same skill seems to be more difficult in the context of the practice-based cases.  However, this does not always occur, as shown in Exam 2, Skill 3.

Little is known about the causes of these observed differences. The differences may be due to some aspect of the cases or protocols differentially affecting candidate performance or rater severity. Alternately, the differences may be due to differences in the skills in the different contexts. That is to say, the ability to diagnose a patient within a standardized, hypothetical protocol varies somewhat from the ability to diagnose a patient in the real-world context of an actual patient case. We continue to investigate the implications of using standardized protocols or candidate practice-based cases in the oral examination process.


Comparison of Skill Difficulties from Two Oral Exams Between  Practice-based Cases and Standardized Protocols

Number

Difficulty for Protocols

Difficulty for Cases

Exam 1

 

 

Skill 1

4.53

4.84

Skill 2

4.82

5.10

 

 

 

Exam 2

 

 

Skill 1

4.51

4.87

Skill 2

5.19

5.20

Skill 3

5.29

5.01

            Higher scaled scores indicate greater difficulty

Measurement Research Associates, Inc.
505 North Lake Shore Dr., Suite 1304
Chicago, IL  60611
Phone: (312) 822-9648     Fax: (312) 822-9650



Coming Rasch-related Events
June 23 - July 21, 2023, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
Aug. 11 - Sept. 8, 2023, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com