Oral exams may include practice-based cases from the
candidate's practice, or standardized protocols, or both. Often the same
candidates are rated by the same examiners on the same clinical skills, for both
cases and protocols.
Because the skills are the same for practice-based cases and protocols, it
may seem the same candidate abilities, such as correctly diagnosing or treating
a patient, are being assessed in both measurement contexts. However, when
candidate performance on common skills are analyzed separately and compared
across cases and protocols, systematic differences in candidate performance and
clinical skill difficulty become apparent. These differences in performance and
difficulty are observed after the many-facet Rasch model has accounted for
differences in the specific exam elements that individual candidates
encountered. The table shows examples from two different examinations. Generally, the same skill seems to be more
difficult in the context of the practice-based cases. However, this does not always occur, as shown
in Exam 2, Skill 3.
Little is known about the causes of these observed
differences. The differences may be due to some aspect of the cases or
protocols differentially affecting candidate performance or rater severity.
Alternately, the differences may be due to differences in the skills in the
different contexts. That is to say, the ability to diagnose a patient within a
standardized, hypothetical protocol varies somewhat from the ability to
diagnose a patient in the real-world context of an actual patient case. We continue
to investigate the implications of using standardized protocols or candidate
practice-based cases in the oral examination process.
Comparison of Skill Difficulties from Two Oral Exams Between Practice-based Cases and Standardized Protocols
Number
|
Difficulty for
Protocols
|
Difficulty for
Cases
|
Exam 1
|
|
|
Skill 1
|
4.53
|
4.84
|
Skill 2
|
4.82
|
5.10
|
|
|
|
Exam 2
|
|
|
Skill 1
|
4.51
|
4.87
|
Skill 2
|
5.19
|
5.20
|
Skill 3
|
5.29
|
5.01
|
Higher
scaled scores indicate greater difficulty
|