MEASUREMENT RESEARCH ASSOCIATES
TEST INSIGHTS

January 2009
Greetings!

Oral examinations have many facets that come together in the scoring process.  We've identified that different oral exam formats (standardized protocols vs. practice-based cases) can function differently. Here's a brief summary of some of those findings.

Ross Brown
Manager of Test Development and Analysis

Differences in Candidate Performance Across Oral Exam Formats

Oral exams may include practice-based cases from the candidate's practice, or standardized protocols, or both. Often the same candidates are rated by the same examiners on the same clinical skills, for both cases and protocols.

Because the skills are the same for practice-based cases and protocols, it may seem the same candidate abilities, such as correctly diagnosing or treating a patient, are being assessed in both measurement contexts. However, when candidate performance on common skills are analyzed separately and compared across cases and protocols, systematic differences in candidate performance and clinical skill difficulty become apparent. These differences in performance and difficulty are observed after the many-facet Rasch model has accounted for differences in the specific exam elements that individual candidates encountered. The table shows examples from two different examinations.  Generally, the same skill seems to be more difficult in the context of the practice-based cases.  However, this does not always occur, as shown in Exam 2, Skill 3.

Little is known about the causes of these observed differences. The differences may be due to some aspect of the cases or protocols differentially affecting candidate performance or rater severity. Alternately, the differences may be due to differences in the skills in the different contexts. That is to say, the ability to diagnose a patient within a standardized, hypothetical protocol varies somewhat from the ability to diagnose a patient in the real-world context of an actual patient case. We continue to investigate the implications of using standardized protocols or candidate practice-based cases in the oral examination process.


Comparison of Skill Difficulties from Two Oral Exams Between  Practice-based Cases and Standardized Protocols

Number

Difficulty for Protocols

Difficulty for Cases

Exam 1

 

 

Skill 1

4.53

4.84

Skill 2

4.82

5.10

 

 

 

Exam 2

 

 

Skill 1

4.51

4.87

Skill 2

5.19

5.20

Skill 3

5.29

5.01

            Higher scaled scores indicate greater difficulty

Measurement Research Associates, Inc.
505 North Lake Shore Dr., Suite 1304
Chicago, IL  60611
Phone: (312) 822-9648     Fax: (312) 822-9650



Coming Rasch-related Events
March 21, 2019, Thur. 13th annual meeting of the UK Rasch user group, Cambridge, UK, http://www.cambridgeassessment.org.uk/events/uk-rasch-user-group-2019
April 4 - 8, 2019, Thur.-Mon. NCME annual meeting, Toronto, Canada,https://ncme.connectedcommunity.org/meetings/annual
April 5 - 9, 2019, Fri.-Tue. AERA annual meeting, Toronto, Canada,www.aera.net/Events-Meetings/Annual-Meeting
April 12, 2019, Fri. On-line course: Understanding Rasch Measurement Theory - Master's Level (G. Masters), https://www.acer.org/au/professional-learning/postgraduate/rasch
May 24 - June 21, 2019, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
May 22 - 30, 2019, Wed.-Thu. Measuring and scale construction (with the Rasch Model), University of Manchester, England, https://www.cmist.manchester.ac.uk/study/short/intermediate/measurement-with-the-rasch-model/
June 4 - 7, 2019, Tue.-Fri.In-Person Italian Rasch Analysis Workshop based on RUMM (entirely in Italian). For enquiries and registration email to workshop.rasch@gmail.com.
June 17-19, 2019, Mon.-Wed. In-person workshop, Melbourne, Australia: Applying the Rasch Model in the Human Sciences: Introduction to Rasch measurement (Trevor Bond, Winsteps), Announcement
June 20-21, 2019, Thurs.-Fri. In-person workshop, Melbourne, Australia: Applying the Rasch Model in the Human Sciences: Advanced Rasch measurement with Facets (Trevor Bond, Facets), Announcement
June 28 - July 26, 2019, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
July 2-5, 2019, Tue.-Fri. 2019 International Measurement Confederation (IMEKO) Joint Symposium, St. Petersburg, Russia,https://imeko19-spb.org
July 11-12 & 15-19, 2019, Thu.-Fri. A Course in Rasch Measurement Theory (D.Andrich), University of Western Australia, Perth, Australia, flyer - http://www.education.uwa.edu.au/ppl/courses
Aug 5 - 10, 2019, Mon.-Sat. 6th International Summer School "Applied Psychometrics in Psychology and Education", Institute of Education at HSE University Moscow, Russia.https://ioe.hse.ru/en/announcements/248134963.html
Aug. 9 - Sept. 6, 2019, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com
August 25-30, 2019, Sun.-Fri. Pacific Rim Objective Measurement Society (PROMS) 2019, Surabaya, Indonesia https://proms.promsociety.org/2019/
Oct. 11 - Nov. 8, 2019, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
Nov. 3 - Nov. 4, 2019, Sun.-Mon. International Outcome Measurement Conference, Chicago, IL,http://jampress.org/iomc2019.htm
Jan. 24 - Feb. 21, 2020, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
May 22 - June 19, 2020, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 26 - July 24, 2020, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
Aug. 7 - Sept. 4, 2020, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com
Oct. 9 - Nov. 6, 2020, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 25 - July 23, 2021, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com