MEASUREMENT RESEARCH ASSOCIATES
TEST INSIGHTS
April 2010
Greetings,
 
The computer is a fascinating and all powerful machine that can be programmed to do just about anything. For certification, the question is how can the computer be programmed to make the examination more valid and reliable.


Mary E. Lunz, Ph.D.

Executive Director

Alternative Item Formats For Computer Based Testing
For certification, we commonly think of multiple choice items because they are generally accepted, easy to administer, familiar from the development perspective, easy to score, and convenient for testing larger samples of candidates.  This article explores some alternatives to standard four or five part multiple choice items that can be delivered with computer based testing.  All item formats have assets and liabilities which are noted. All of these formats are available at Pearson VUE.

Video clips can be used to show organ function or patient interaction. They have action and sound. It is easy to incorporate video clips into a standard multiple choice exam. They must meet a standard video format (MPEG-1). Sound, if any is heard through ear phones.

Medical animation requires the development of the graphic images which are associated with the questions or actions/manipulations being tested. The graphics can be programmed so that the candidate controls the actions and behavior of the characters. The development of the graphics can be expensive and requires a great deal of work by the subject matter experts. The candidate may have to learn how to operate the computer programming to make the graphic images perform the tasks being tested.

Short answer responses allow the candidate to offer their thoughts and ideas related to a specific topic or question. The candidate explains the rationale for his/her judgments, selected procedures, or problem solving strategies. The difficulty with the short answer responses, is that they must be graded by a content expert unless a long, complex, and expensive scoring algorithm is constructed and validated. No such scoring algorithm has been developed to date.

Extended multiple choice items allow the inclusion of many possible responses (e.g. 10 - 20) based on the theory that candidates require more knowledge, skill and judgment to sort through all of the possible alternatives. Often candidates are allowed to choose all appropriate responses for this type of item. In reality, we have often found that there are four or five responses that are frequently selected and the others are not selected by any candidates.

Hot spot technology allows candidates to respond by clicking on points in an image to answer the question. The mouse cursor becomes a crosshair shape when over a hot-spot image and a marker is placed at the center of the crosshair when the candidate clicks the mouse button. A simplistic example is asking the candidate to click on the capital of Texas with many cities in Texas shown on a map. It is parallel to clicking on the section of an x-ray that identifies the patient's problem. 

Zoom technology uses the pearsonvue:scalefactors to allow a candidate to zoom into and out of content within an image . For example, an image is displayed and the candidate scrutinizes the image, zooming (or scaling) the image between a predefined set of zoom levels. This would be appropriate for microscopic slides. However, there is the danger that the image may become pixilated when zoomed to a very high scaling factor.

Finally, almost any type of visual can be used in the exams. More creative use of visuals could enhance the knowledge and skill tested in the exam.  

What can be done with computer programming is limited only by our imaginations and budgets.  The use of alternate item formats raises potential scoring, expert commitment, and programming cost issues. The question is "what is gained vs. what is lost?"  While the enhanced computer administration formats increase validity, what might be lost with regard to reliability or accuracy of candidate measurement?  Is the financial investment worth the long term convenience of less travel and "away" time for both candidates and examiners.  For example, if examiners score short answer responses at home instead of administering an oral examination at a specified place and time, will there be an impact on candidate outcomes when examiner colleagues are no longer available for professional and social interaction.
Measurement Research Associates, Inc.
505 North Lake Shore Dr., Suite 1304
Chicago, IL  60611
Phone: (312) 822-9648     Fax: (312) 822-9650


Coming Rasch-related Events
March 21, 2019, Thur. 13th annual meeting of the UK Rasch user group, Cambridge, UK, http://www.cambridgeassessment.org.uk/events/uk-rasch-user-group-2019
April 4 - 8, 2019, Thur.-Mon. NCME annual meeting, Toronto, Canada,https://ncme.connectedcommunity.org/meetings/annual
April 5 - 9, 2019, Fri.-Tue. AERA annual meeting, Toronto, Canada,www.aera.net/Events-Meetings/Annual-Meeting
April 12, 2019, Fri. On-line course: Understanding Rasch Measurement Theory - Master's Level (G. Masters), https://www.acer.org/au/professional-learning/postgraduate/rasch
May 24 - June 21, 2019, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
May 22 - 30, 2019, Wed.-Thu. Measuring and scale construction (with the Rasch Model), University of Manchester, England, https://www.cmist.manchester.ac.uk/study/short/intermediate/measurement-with-the-rasch-model/
June 17-19, 2019, Mon.-Wed. In-person workshop, Melbourne, Australia: Applying the Rasch Model in the Human Sciences: Introduction to Rasch measurement (Trevor Bond, Winsteps), Announcement
June 20-21, 2019, Thurs.-Fri. In-person workshop, Melbourne, Australia: Applying the Rasch Model in the Human Sciences: Advanced Rasch measurement with Facets (Trevor Bond, Facets), Announcement
June 28 - July 26, 2019, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
July 2-5, 2019, Tue.-Fri. 2019 International Measurement Confederation (IMEKO) Joint Symposium, St. Petersburg, Russia,https://imeko19-spb.org
July 11-12 & 15-19, 2019, Thu.-Fri. A Course in Rasch Measurement Theory (D.Andrich), University of Western Australia, Perth, Australia, flyer - http://www.education.uwa.edu.au/ppl/courses
Aug 5 - 10, 2019, Mon.-Sat. 6th International Summer School "Applied Psychometrics in Psychology and Education", Institute of Education at HSE University Moscow, Russia.https://ioe.hse.ru/en/announcements/248134963.html
Aug. 9 - Sept. 6, 2019, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com
August 25-30, 2019, Sun.-Fri. Pacific Rim Objective Measurement Society (PROMS) 2019, Surabaya, Indonesia https://proms.promsociety.org/2019/
Oct. 11 - Nov. 8, 2019, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
Jan. 24 - Feb. 21, 2020, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
May 22 - June 19, 2020, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 26 - July 24, 2020, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
Aug. 7 - Sept. 4, 2020, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com
Oct. 9 - Nov. 6, 2020, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 25 - July 23, 2021, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com