Oral exams may include practice-based cases from the candidate's practice, or standardized protocols, or both. Often the same candidates are rated by the same examiners on the same clinical skills, for both cases and protocols.

Because the skills are the same for practice-based cases and protocols, it may seem the same candidate abilities, such as correctly diagnosing or treating a patient, are being assessed in both measurement contexts. However, when candidate performance on common skills are analyzed separately and compared across cases and protocols, systematic differences in candidate performance and clinical skill difficulty become apparent. These differences in performance and difficulty are observed after the many-facet Rasch model has accounted for differences in the specific exam elements that individual candidates encountered. The table shows examples from two different examinations.  Generally, the same skill seems to be more difficult in the context of the practice-based cases.  However, this does not always occur, as shown in Exam 2, Skill 3.

Little is known about the causes of these observed differences. The differences may be due to some aspect of the cases or protocols differentially affecting candidate performance or rater severity. Alternately, the differences may be due to differences in the skills in the different contexts. That is to say, the ability to diagnose a patient within a standardized, hypothetical protocol varies somewhat from the ability to diagnose a patient in the real-world context of an actual patient case. We continue to investigate the implications of using standardized protocols or candidate practice-based cases in the oral examination process.

Comparison of Skill Difficulties from Two Oral Exams Between  Practice-based Cases and Standardized Protocols

            Higher scaled scores indicate greater difficulty