June 2008
I have worked with multiple-choice items and the committees who write them for many years. While computer-based testing enhances exam security, many people involved in certification exam programs still worry that candidates share information about what is on the exam. 
Ross Brown
Consistency of Common Item p-Values across Multiple Choice Exam Administrations

An often-voiced concern is that candidates memorize item content, and then share the information with colleagues. Candidates so forewarned of what is on a test may have an unfair advantage, and might be able to pass the test without having the required level of knowledge and skill. To determine if candidates are able to gain an unfair advantage in this way, a systematic study was conducted.
The study examined item p-values across exam administrations. The p-value is the percent of candidates who answer an item correctly. If the p-values of items that appear on more than one examination increase by less than 10% from one exam to the next, there is a low probability that individual candidates are benefiting from items being leaked.
The p-values of items used on consecutive exams were compared. The table below shows the results. For all five examinations, the majority of the common items did not show p-value increases of more than 10%. Furthermore, even though some of the tests included a large number of carryover items, the items that had higher p-values (easier) were a very small percentage of the total number of items on the second test.
Thus, the use of common items on certification examinations does not seem to increase the percent of candidates who answer the item correctly. This makes one wonder about the efficiency of the networks of individuals who supposedly memorize items during a test to share with their colleagues. Perhaps their memories are not terribly accurate, their colleagues are not helped by the information, or misinformation is actually transmitted. 


Number of Common Items

Number of items on test

Number of items with p-value increases of 10% or more on second test

Percent of Common items that got easier as a percentage of all items on the second test

Exam 1  



23 items


Exam 2



6 items


Exam 3



3 items


Exam 4



3 items


Exam 5



7 items


Measurement Research Associates, Inc.
505 North Lake Shore Dr., Suite 1304
Chicago, IL  60611
Phone: (312) 822-9648     Fax: (312) 822-9650

Please help with Standard Dataset 4: Andrich Rating Scale Model

Rasch Publications
Rasch Measurement Transactions (free, online) Rasch Measurement research papers (free, online) Probabilistic Models for Some Intelligence and Attainment Tests, Georg Rasch Applying the Rasch Model 3rd. Ed., Bond & Fox Best Test Design, Wright & Stone
Rating Scale Analysis, Wright & Masters Introduction to Rasch Measurement, E. Smith & R. Smith Introduction to Many-Facet Rasch Measurement, Thomas Eckes Invariant Measurement: Using Rasch Models in the Social, Behavioral, and Health Sciences, George Engelhard, Jr. Statistical Analyses for Language Testers, Rita Green
Rasch Models: Foundations, Recent Developments, and Applications, Fischer & Molenaar Journal of Applied Measurement Rasch models for measurement, David Andrich Constructing Measures, Mark Wilson Rasch Analysis in the Human Sciences, Boone, Stave, Yale
in Spanish: Análisis de Rasch para todos, Agustín Tristán Mediciones, Posicionamientos y Diagnósticos Competitivos, Juan Ramón Oreja Rodríguez

To be emailed about new material on
please enter your email address here:

I want to Subscribe: & click below
I want to Unsubscribe: & click below

Please set your SPAM filter to accept emails from welcomes your comments:
Please email inquiries about Rasch books to books \at/

Your email address (if you want us to reply):


FORUMRasch Measurement Forum to discuss any Rasch-related topic

Coming Rasch-related Events
Sept. 15-16, 2017, Fri.-Sat. IOMC 2017: International Outcome Measurement Conference, Chicago,
Oct. 13 - Nov. 10, 2017, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps),
Oct. 25-27, 2017, Wed.-Fri. In-person workshop: Applying the Rasch Model hands-on introductory workshop, Melbourne, Australia (T. Bond, B&FSteps), Announcement
Jan. 5 - Feb. 2, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps),
Jan. 10-16, 2018, Wed.-Tues. In-person workshop: Advanced Course in Rasch Measurement Theory and the application of RUMM2030, Perth, Australia (D. Andrich), Announcement
Jan. 17-19, 2018, Wed.-Fri. Rasch Conference: Seventh International Conference on Probabilistic Models for Measurement, Matilda Bay Club, Perth, Australia, Website
April 13-17, 2018, Fri.-Tues. AERA, New York, NY,
May 25 - June 22, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps),
June 29 - July 27, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps),
Aug. 10 - Sept. 7, 2018, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets),
Oct. 12 - Nov. 9, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps),
The HTML to add "Coming Rasch-related Events" to your webpage is:
<script type="text/javascript" src=""></script>