MEASUREMENT RESEARCH ASSOCIATES
TEST INSIGHTS
February 2009
Greetings,
 
There is frequently a question about the number of items needed to cover the content and insure candidate separation reliability.  Our simple study revealed that the quality of items is as or more important than the absolute number of items on the test to achieve satisfactory reliability.
 
Mary E. Lunz, Ph.D.
Director

 
Test Length and Test Reliability for Multiple Choice Examinations

Some believe that longer multiple choice tests tend to be more reliable because more items automatically reduce the error of measurement.  Indeed, a sufficient number of items must be included to cover the content areas tested; however, there are other factors that contribute to how efficiently a test measures and separates candidate ability. 

The indication of reliability is candidate separation reliability, calculated as [SD2 - SE2/ SD2] using Rasch logit measures.  This reliability index is appropriate because the goal of any certification examination is to distinguish between those candidates who are worthy of passing and those who are not.  The better the test items distinguish among candidate abilities, the less measurement error there is in the examination and the higher the candidate separation reliability, regardless of the number of items.  For example, deleting poorly performing items makes the test shorter, but also increases the accuracy of measurement because the items producing the most error are eliminated from the examination. 

In this simple study, we looked at test length compared to test reliability. Six different exams were included. The table shows the tests in reliability order along with the number of items in each test.  The number of items and reliability do not appear to be related. A test of 211 items had a reliability of .93 while another test of 233 items had a reliability of .78.  It appears that the quality of the items is as important, or even more important, than the absolute number of items on the test. Of course, a reasonable number of items must be included to insure accurate measurement and content coverage.

   Table of Test Length and Reliability

Exam

N of items

Reliability

Exam 1

211

.93

Exam 2

133

.91

Exam 3

193

.90

Exam 4

129

.86

Exam 5

236

.85

Exam 6

233

.78


Measurement Research Associates, Inc.
505 North Lake Shore Dr., Suite 1304
Chicago, IL  60611
Phone: (312) 822-9648     Fax: (312) 822-9650



Coming Rasch-related Events
Sept. 27-29, 2017, Wed.-Fri. In-person workshop: Introductory Rasch Analysis using RUMM2030, Leeds, UK (M. Horton), Announcement
Oct. 13 - Nov. 10, 2017, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
Oct. 25-27, 2017, Wed.-Fri. In-person workshop: Applying the Rasch Model hands-on introductory workshop, Melbourne, Australia (T. Bond, B&FSteps), Announcement
Dec. 6-8, 2017, Wed.-Fri. In-person workshop: Introductory Rasch Analysis using RUMM2030, Leeds, UK (M. Horton), Announcement
Jan. 5 - Feb. 2, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
Jan. 10-16, 2018, Wed.-Tues. In-person workshop: Advanced Course in Rasch Measurement Theory and the application of RUMM2030, Perth, Australia (D. Andrich), Announcement
Jan. 17-19, 2018, Wed.-Fri. Rasch Conference: Seventh International Conference on Probabilistic Models for Measurement, Matilda Bay Club, Perth, Australia, Website
April 13-17, 2018, Fri.-Tues. AERA, New York, NY, www.aera.net
May 25 - June 22, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 29 - July 27, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
Aug. 10 - Sept. 7, 2018, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com
Oct. 12 - Nov. 9, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com