Computer-Adaptive Testing: CAT Fit Analysis

Fit is inferred from the consistency between examinee responses and item difficulties. Improbable responses signal inconsistent activity. Computer assisted testing (CAT) usually tailors item difficulties to match examinee ability, but when examinees are administered items selected to give them a 50% chance of success, both right and wrong answers become equally probable, precluding a basis for the usual fit analysis.

There are, however, two statistics which, in combination, can help detect misfit in CAT. The first is the observed percent correct on the test. If CAT targeting is set for 50% success, then instead of reviewing a person's response pattern item by item, we can compare the obtained percent correct with the intended 50%. A significant difference implies a fit problem.

The second statistic is the mean of the differences, (B-D), across all items administered to a person, between the person measure, B, estimated at each step, and the item difficulty, D, of that step. This mean (B-D) difference indicates how well the item bank has provided on-target items. When targeting has been set at 50%, its expected value is zero. When, however, the items provided have been too easy, the difference will be greater than zero, and vice-versa.

The mean (B-D) difference can be converted to an "expected" percent correct, e.g. when the mean difference is +.4 logits, the expected percent correct is 60%. If the expected percent correct deviates from 50%, the item bank is not providing correctly targeted items, and the examinee's expected success rate is no longer 50% but the expected percent correct. Failure of the item bank to provide correctly targeted items usually occurs for very high or very low ability examinees.

Person misfit is implied when the expected percent correct is near 50%, but the examinees observed success rate deviates from 50%. The item pool has provided correctly targeted items but the behavior of the examinee has been inconsistent.

In summary, comparison of observed percent correct with expected percent correct provides a useful check on item bank depth and permits the identification of some types of examinee inconsistency.



Computer-Adaptive Testing: CAT Fit Analysis, B Bergstrom … Rasch Measurement Transactions, 1990, 4:3 p. 112




Rasch Publications
Rasch Measurement Transactions (free, online) Rasch Measurement research papers (free, online) Probabilistic Models for Some Intelligence and Attainment Tests, Georg Rasch Applying the Rasch Model 2nd. Ed., Bond & Fox Best Test Design, Wright & Stone
Rating Scale Analysis, Wright & Masters Introduction to Rasch Measurement, E. Smith & R. Smith Introduction to Many-Facet Rasch Measurement, Thomas Eckes Invariant Measurement: Using Rasch Models in the Social, Behavioral, and Health Sciences, George Engelhard, Jr. Statistical Analyses for Language Testers, Rita Green
Rasch Models: Foundations, Recent Developments, and Applications, Fischer & Molenaar Journal of Applied Measurement Rasch models for measurement, David Andrich Constructing Measures, Mark Wilson Rasch Analysis in the Human Sciences, Boone, Stave, Yale

To be emailed about new material on www.rasch.org
please enter your email address here:

I want to Subscribe: & click below
I want to Unsubscribe: & click below

Please set your SPAM filter to accept emails from Rasch.org

www.rasch.org welcomes your comments:

Your email address (if you want us to reply):

 

ForumRasch Measurement Forum to discuss any Rasch-related topic

Go to Top of Page
Go to index of all Rasch Measurement Transactions
AERA members: Join the Rasch Measurement SIG and receive the printed version of RMT
Some back issues of RMT are available as bound volumes
Subscribe to Journal of Applied Measurement

Go to Institute for Objective Measurement Home Page. The Rasch Measurement SIG (AERA) thanks the Institute for Objective Measurement for inviting the publication of Rasch Measurement Transactions on the Institute's website, www.rasch.org.

Coming Rasch-related Events
Jan. 2-30, 2015, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
Feb 23 - June 19, 2015, Mon.-Fri. On-line graduate-level course: Advanced Rasch Measurement Theory (D. Andrich, I. Marais, RUMM), www.education.uwa.edu.au/ppl/courses
Jan. 12-14, 2015, Mon.-Wed. 6th Rasch Conference: Sixth International Conference on Probabilistic Models for Measurement in Education, Psychology, Social Science and Health, Cape Town, South Africa www.rasch.co.za/conference.php
March 11-13, 2015, Wed.-Fri. In-person workshop: Introductory Rasch (A. Tennant, RUMM), Leeds, UK, www.leeds.ac.uk/medicine/rehabmed/psychometric
March 11-13, 2015, Wed.-Fri. In-person workshop: Einführung in die Raschanalyse mit RUMM2030 (Böcker, Salzberger, RUMM), Aachen, Germany, www.psychometrikon.de
March 20, 2015, Fri. UK Rasch User Group Meeting, London, United Kingdom, www.rasch.org.uk
March 26-27, 2015, Thur.-Fri. In-person workshop: Introduction to Rasch Measurement with Winsteps (W. Boone), Cincinnati, raschmeasurementanalysis.com
April 16-20, 2015, Thurs.-Mon. AERA Annual Meeting, Chicago IL www.aera.net
April 21-22, 2015, Tues.-Wed. IOMC 2015: International Outcomes Measurement Conference, Chicago IL www.jampress.org
May 13-15, 2015, Wed.-Fri. In-person workshop: Introductory Rasch (A. Tennant, RUMM), Leeds, UK, www.leeds.ac.uk/medicine/rehabmed/psychometric
May 18-20, 2015, Mon.-Wed. In-person workshop: Intermediate Rasch (A. Tennant, RUMM), Leeds, UK, www.leeds.ac.uk/medicine/rehabmed/psychometric
May 29 - June 26, 2015, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
July 3-31, 2015, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
July 27 - Nov. 20, 2015, Mon.-Fri. On-line course: Introduction to Rasch Measurement Theory (D. Andrich, I. Marais, RUMM), www.education.uwa.edu.au/ppl/courses
Aug. 14 - Sept. 11, 2015, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com
Sept. 9-11, 2015, Wed.-Fri. In-person workshop: Introductory Rasch (A. Tennant, RUMM), Leeds, UK, www.leeds.ac.uk/medicine/rehabmed/psychometric
Sept. 14-16, 2015, Mon.-Wed. In-person workshop: Intermediate Rasch (A. Tennant, RUMM), Leeds, UK, www.leeds.ac.uk/medicine/rehabmed/psychometric
Sept. 17-18, 2015, Thur.-Fri. In-person workshop: Advanced Rasch (A. Tennant, RUMM), Leeds, UK, www.leeds.ac.uk/medicine/rehabmed/psychometric
Oct. 16 - Nov. 13, 2015, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
Sept. 4 - Oct. 16, 2015, Fri.-Fri. On-line workshop: Rasch Applications, Part 1: How to Construct a Rasch Scale (W. Fisher), www.statistics.com
Oct. 23 - Nov. 20, 2015, Fri.-Fri. On-line workshop: Rasch Applications, Part 2: Clinical Assessment, Survey Research, and Educational Measurement (W. Fisher), www.statistics.com
Dec. 2-4, 2015, Wed.-Fri. In-person workshop: Introductory Rasch (A. Tennant, RUMM), Leeds, UK, www.leeds.ac.uk/medicine/rehabmed/psychometric
Aug. 12 - Sept. 9, 2016, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com
The HTML to add "Coming Rasch-related Events" to your webpage is:
<script type="text/javascript" src="http://www.rasch.org/events.txt"></script>

The URL of this page is www.rasch.org/rmt/rmt43b.htm

Website: www.rasch.org/rmt/contents.htm