MEASUREMENT RESEARCH ASSOCIATES
TEST INSIGHTS
April 2010
Greetings,
 
The computer is a fascinating and all powerful machine that can be programmed to do just about anything. For certification, the question is how can the computer be programmed to make the examination more valid and reliable.


Mary E. Lunz, Ph.D.

Executive Director

Alternative Item Formats For Computer Based Testing
For certification, we commonly think of multiple choice items because they are generally accepted, easy to administer, familiar from the development perspective, easy to score, and convenient for testing larger samples of candidates.  This article explores some alternatives to standard four or five part multiple choice items that can be delivered with computer based testing.  All item formats have assets and liabilities which are noted. All of these formats are available at Pearson VUE.

Video clips can be used to show organ function or patient interaction. They have action and sound. It is easy to incorporate video clips into a standard multiple choice exam. They must meet a standard video format (MPEG-1). Sound, if any is heard through ear phones.

Medical animation requires the development of the graphic images which are associated with the questions or actions/manipulations being tested. The graphics can be programmed so that the candidate controls the actions and behavior of the characters. The development of the graphics can be expensive and requires a great deal of work by the subject matter experts. The candidate may have to learn how to operate the computer programming to make the graphic images perform the tasks being tested.

Short answer responses allow the candidate to offer their thoughts and ideas related to a specific topic or question. The candidate explains the rationale for his/her judgments, selected procedures, or problem solving strategies. The difficulty with the short answer responses, is that they must be graded by a content expert unless a long, complex, and expensive scoring algorithm is constructed and validated. No such scoring algorithm has been developed to date.

Extended multiple choice items allow the inclusion of many possible responses (e.g. 10 - 20) based on the theory that candidates require more knowledge, skill and judgment to sort through all of the possible alternatives. Often candidates are allowed to choose all appropriate responses for this type of item. In reality, we have often found that there are four or five responses that are frequently selected and the others are not selected by any candidates.

Hot spot technology allows candidates to respond by clicking on points in an image to answer the question. The mouse cursor becomes a crosshair shape when over a hot-spot image and a marker is placed at the center of the crosshair when the candidate clicks the mouse button. A simplistic example is asking the candidate to click on the capital of Texas with many cities in Texas shown on a map. It is parallel to clicking on the section of an x-ray that identifies the patient's problem. 

Zoom technology uses the pearsonvue:scalefactors to allow a candidate to zoom into and out of content within an image . For example, an image is displayed and the candidate scrutinizes the image, zooming (or scaling) the image between a predefined set of zoom levels. This would be appropriate for microscopic slides. However, there is the danger that the image may become pixilated when zoomed to a very high scaling factor.

Finally, almost any type of visual can be used in the exams. More creative use of visuals could enhance the knowledge and skill tested in the exam.  

What can be done with computer programming is limited only by our imaginations and budgets.  The use of alternate item formats raises potential scoring, expert commitment, and programming cost issues. The question is "what is gained vs. what is lost?"  While the enhanced computer administration formats increase validity, what might be lost with regard to reliability or accuracy of candidate measurement?  Is the financial investment worth the long term convenience of less travel and "away" time for both candidates and examiners.  For example, if examiners score short answer responses at home instead of administering an oral examination at a specified place and time, will there be an impact on candidate outcomes when examiner colleagues are no longer available for professional and social interaction.
Measurement Research Associates, Inc.
505 North Lake Shore Dr., Suite 1304
Chicago, IL  60611
Phone: (312) 822-9648     Fax: (312) 822-9650

Please help with Standard Dataset 4: Andrich Rating Scale Model



Rasch Publications
Rasch Measurement Transactions (free, online) Rasch Measurement research papers (free, online) Probabilistic Models for Some Intelligence and Attainment Tests, Georg Rasch Applying the Rasch Model 3rd. Ed., Bond & Fox Best Test Design, Wright & Stone
Rating Scale Analysis, Wright & Masters Introduction to Rasch Measurement, E. Smith & R. Smith Introduction to Many-Facet Rasch Measurement, Thomas Eckes Invariant Measurement: Using Rasch Models in the Social, Behavioral, and Health Sciences, George Engelhard, Jr. Statistical Analyses for Language Testers, Rita Green
Rasch Models: Foundations, Recent Developments, and Applications, Fischer & Molenaar Journal of Applied Measurement Rasch models for measurement, David Andrich Constructing Measures, Mark Wilson Rasch Analysis in the Human Sciences, Boone, Stave, Yale
in Spanish: Análisis de Rasch para todos, Agustín Tristán Mediciones, Posicionamientos y Diagnósticos Competitivos, Juan Ramón Oreja Rodríguez

To be emailed about new material on www.rasch.org
please enter your email address here:

I want to Subscribe: & click below
I want to Unsubscribe: & click below

Please set your SPAM filter to accept emails from Rasch.org

www.rasch.org welcomes your comments:
Please email inquiries about Rasch books to books \at/ rasch.org

Your email address (if you want us to reply):

 

FORUMRasch Measurement Forum to discuss any Rasch-related topic

Coming Rasch-related Events
April 26-30, 2017, Wed.-Sun. NCME, San Antonio, TX, www.ncme.org - April 29: Ben Wright book
April 27 - May 1, 2017, Thur.-Mon. AERA, San Antonio, TX, www.aera.net
April 29, 2017, Sat., 16:35 to 18:05. NCME Presidents Invitational Symposium: a new book commemorating Ben Wright's life and career, 16:35 to 18:05, San Antonio, TX, www.ncme.org
May 26 - June 23, 2017, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 30 - July 29, 2017, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
July 31 - Aug. 3, 2017, Mon.-Thurs. Joint IMEKO TC1-TC7-TC13 Symposium 2017: Measurement Science challenges in Natural and Social Sciences, Rio de Janeiro, Brazil, imeko-tc7-rio.org.br
Aug. 7-9, 2017, Mon-Wed. In-person workshop and research coloquium: Effect size of family and school indexes in writing competence using TERCE data (C. Pardo, A. Atorressi, Winsteps), Bariloche Argentina. Carlos Pardo, Universidad Catòlica de Colombia
Aug. 7-9, 2017, Mon-Wed. PROMS 2017: Pacific Rim Objective Measurement Symposium, Sabah, Borneo, Malaysia, proms.promsociety.org/2017/
Aug. 10, 2017, Thurs. In-person Winsteps Training Workshop (M. Linacre, Winsteps), Sydney, Australia. www.winsteps.com/sydneyws.htm
Aug. 11 - Sept. 8, 2017, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com
Aug. 18-21, 2017, Fri.-Mon. IACAT 2017: International Association for Computerized Adaptive Testing, Niigata, Japan, iacat.org
Sept. 15-16, 2017, Fri.-Sat. IOMC 2017: International Outcome Measurement Conference, Chicago, jampress.org/iomc2017.htm
Oct. 13 - Nov. 10, 2017, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
Jan. 5 - Feb. 2, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
Jan. 10-16, 2018, Wed.-Tues. In-person workshop: Advanced Course in Rasch Measurement Theory and the application of RUMM2030, Perth, Australia (D. Andrich), Announcement
Jan. 17-19, 2018, Wed.-Fri. Rasch Conference: Seventh International Conference on Probabilistic Models for Measurement, Matilda Bay Club, Perth, Australia, Website
May 25 - June 22, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 29 - July 27, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
Aug. 10 - Sept. 7, 2018, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com
Oct. 12 - Nov. 9, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
The HTML to add "Coming Rasch-related Events" to your webpage is:
<script type="text/javascript" src="http://www.rasch.org/events.txt"></script>