MEASUREMENT RESEARCH ASSOCIATES
TEST INSIGHTS
June 2008
Greetings! 
 
I have worked with multiple-choice items and the committees who write them for many years. While computer-based testing enhances exam security, many people involved in certification exam programs still worry that candidates share information about what is on the exam. 
 
Ross Brown
Consistency of Common Item p-Values across Multiple Choice Exam Administrations

An often-voiced concern is that candidates memorize item content, and then share the information with colleagues. Candidates so forewarned of what is on a test may have an unfair advantage, and might be able to pass the test without having the required level of knowledge and skill. To determine if candidates are able to gain an unfair advantage in this way, a systematic study was conducted.
 
The study examined item p-values across exam administrations. The p-value is the percent of candidates who answer an item correctly. If the p-values of items that appear on more than one examination increase by less than 10% from one exam to the next, there is a low probability that individual candidates are benefiting from items being leaked.
 
The p-values of items used on consecutive exams were compared. The table below shows the results. For all five examinations, the majority of the common items did not show p-value increases of more than 10%. Furthermore, even though some of the tests included a large number of carryover items, the items that had higher p-values (easier) were a very small percentage of the total number of items on the second test.
 
Thus, the use of common items on certification examinations does not seem to increase the percent of candidates who answer the item correctly. This makes one wonder about the efficiency of the networks of individuals who supposedly memorize items during a test to share with their colleagues. Perhaps their memories are not terribly accurate, their colleagues are not helped by the information, or misinformation is actually transmitted. 


Exam

Number of Common Items

Number of items on test

Number of items with p-value increases of 10% or more on second test

Percent of Common items that got easier as a percentage of all items on the second test

Exam 1  

250

313

23 items

7%

Exam 2

53

144

6 items

4%

Exam 3

290

310

3 items

1%

Exam 4

52

150

3 items

2%

Exam 5

196

252

7 items

3%

 
Measurement Research Associates, Inc.
505 North Lake Shore Dr., Suite 1304
Chicago, IL  60611
Phone: (312) 822-9648     Fax: (312) 822-9650
 


Coming Rasch-related Events
March 21, 2019, Thur. 13th annual meeting of the UK Rasch user group, Cambridge, UK, http://www.cambridgeassessment.org.uk/events/uk-rasch-user-group-2019
April 4 - 8, 2019, Thur.-Mon. NCME annual meeting, Toronto, Canada,https://ncme.connectedcommunity.org/meetings/annual
April 5 - 9, 2019, Fri.-Tue. AERA annual meeting, Toronto, Canada,www.aera.net/Events-Meetings/Annual-Meeting
April 12, 2019, Fri. On-line course: Understanding Rasch Measurement Theory - Master's Level (G. Masters), https://www.acer.org/au/professional-learning/postgraduate/rasch
May 24 - June 21, 2019, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
May 22 - 30, 2019, Wed.-Thu. Measuring and scale construction (with the Rasch Model), University of Manchester, England, https://www.cmist.manchester.ac.uk/study/short/intermediate/measurement-with-the-rasch-model/
June 17-19, 2019, Mon.-Wed. In-person workshop, Melbourne, Australia: Applying the Rasch Model in the Human Sciences: Introduction to Rasch measurement (Trevor Bond, Winsteps), Announcement
June 20-21, 2019, Thurs.-Fri. In-person workshop, Melbourne, Australia: Applying the Rasch Model in the Human Sciences: Advanced Rasch measurement with Facets (Trevor Bond, Facets), Announcement
June 28 - July 26, 2019, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
July 2-5, 2019, Tue.-Fri. 2019 International Measurement Confederation (IMEKO) Joint Symposium, St. Petersburg, Russia,https://imeko19-spb.org
July 11-12 & 15-19, 2019, Thu.-Fri. A Course in Rasch Measurement Theory (D.Andrich), University of Western Australia, Perth, Australia, flyer - http://www.education.uwa.edu.au/ppl/courses
Aug 5 - 10, 2019, Mon.-Sat. 6th International Summer School "Applied Psychometrics in Psychology and Education", Institute of Education at HSE University Moscow, Russia.https://ioe.hse.ru/en/announcements/248134963.html
Aug. 9 - Sept. 6, 2019, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com
August 25-30, 2019, Sun.-Fri. Pacific Rim Objective Measurement Society (PROMS) 2019, Surabaya, Indonesia https://proms.promsociety.org/2019/
Oct. 11 - Nov. 8, 2019, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
Jan. 24 - Feb. 21, 2020, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
May 22 - June 19, 2020, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 26 - July 24, 2020, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
Aug. 7 - Sept. 4, 2020, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com
Oct. 9 - Nov. 6, 2020, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 25 - July 23, 2021, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com