Reasonable mean-square fit values.

"Less control over the data → more off-dimensional behavior → more departures in the data from the Rasch model"

Though the ideal for measurement construction is that data fit the Rasch model, all empirical data departs from the model to some extent. But how much unmodelled noise is tolerable? Conventional statisticians base their decisions on significance tests, but these are heavily influenced by sample size.

"Very large samples form a special source of problems. This is because no model can ever be supposed to be perfectly fitted by data, so with a sufficiently large sample any model would have to be discarded. In connection with the problem Martin-Löf (1974) stated: 'This indicates that for large sets of data it is too destructive to let an ordinary significance test decide whether or not to accept a proposed statistical model [or data], because, with few exceptions, we know that we shall have to reject it even without looking at the data simply because the number of observations is so large. In such cases, we need instead a quantitative measure of the size of the discrepancy between the statistical model and the observed set of data.'" (Gustafson 1980).

Convenient quantitative measures of fit discrepancy are mean-square residual summary statistics, such as OUTFIT and INFIT. These statistics have expectation 1.0, and range from 0 to infinity. Mean-squares greater than 1.0 indicate underfit to the Rasch model, i.e., the data are less predictable than the model expects. Mean-squares less than 1.0 indicate overfit to the Rasch model, i.e., the data are more predictable than the model expects. A mean-square of 1.2 indicates that there is 20% more randomness (i.e., noise) in the data than modelled. A mean-square of 0.7 indicates a 30% deficiency in Rasch-model-predicted randomness (i.e., the data are too Guttman-like), which implies 100*(1-0.7)/0.7 = 43% more ambiguity in the inferred measure than modelled (e.g., the item difficulty estimated from low-ability persons differs noticeably from the item difficulty estimated from high-ability persons).


Reasonable Item Mean-square Ranges
for INFIT and OUTFIT
Type of Test Range
MCQ (High stakes)
MCQ (Run of the mill)
Rating scale (survey)
Clinical observation
Judged (agreement encouraged)
0.8 - 1.2
0.7 - 1.3
0.6 - 1.4
0.5 - 1.7
0.4 - 1.2


When is a mean-square too large or too small? There are no hard-and-fast rules. Particular features of a testing situation, e.g., mixing item types or off-target testing, can produce idiosyncratic mean-square distributions. Nevertheless, here, as a rule of thumb, are some reasonable ranges for item mean-square fit statistics. In Classical Test Theory, 3-PL IRT, and conventional statistics, low mean-squares are considered good. In Rasch theory, they indicate some redundancy in the responses, but they do no harm.

Ben Wright & John Michael Linacre

J.-E. Gustafson (1980) Testing and obtaining fit of data to the Rasch model. British Journal of mathematical and Statistical Psychology, 33, p.220.

P. Martin-Löf (1974) The notion of redundancy and its use as a quantitative measure of the discrepancy between a statistical hypothesis and observational data. Scandinavian Journal of Statistics, 1, 3.

Note by Linacre: Informal simulations studies and experience analyzing hundreds of datasets indicate that:

Interpretation of parameter-level mean-square fit statistics:
>2.0Distorts or degrades the measurement system
1.5 - 2.0Unproductive for construction of measurement, but not degrading
0.5 - 1.5Productive for measurement
<0.5Less productive for measurement, but not degrading.
May produce misleadingly good reliabilities and separations

The Mean-Square statistic is also called the Relative Chi-Square and the Normed Chi-Square. According to www.psych-it.com.au/Psychlopedia/article.asp?id=277, in Structural Equation Modeling, the criterion for acceptance varies across researchers, ranging from less than 2 (Ullman, 2001) to less than 5 (Schumacker & Lomax, 2004).


[Later:] Overfit (mean-squares less than 1.0).

See also "Size vs. Significance: Mean-Square and Standardized Chi-Square Fit Statistics" www.rasch.org/rmt/rmt171n.htm


Item Fit and Person Fit

From a statistical Rasch perspective, persons and items are exactly the same. They are merely parameters of the Rasch model. So the fit criteria would be exactly the same. But, from a substantive perspective, persons and items differ. We expect the items to be better-behaved than the persons. We also expect item difficulties to continue into the future, but we expect person abilities to change. Also, we expect items to be encountered by many, many persons, but persons to encounter relatively few items. Consequently, we are usually stricter in our application of fit rules to items than to persons. A few maverick persons in a dataset don't worry us - they will have negligible impact on anything else. But a few maverick items raise questions about test administration, data entry accuracy, the definition of the latent variable, etc. We will immediately focus our attention on them because they may be symptomatic of a more pervasive problem, such as the wrong key for a multiple-choice test, or reversed-coded items on a survey.


Reasonable mean-square fit values. Wright BD, Linacre JM … Rasch Measurement Transactions, 1994, 8:3 p.370


Please help with Standard Dataset 4: Andrich Rating Scale Model



Rasch Publications
Rasch Measurement Transactions (free, online) Rasch Measurement research papers (free, online) Probabilistic Models for Some Intelligence and Attainment Tests, Georg Rasch Applying the Rasch Model 3rd. Ed., Bond & Fox Best Test Design, Wright & Stone
Rating Scale Analysis, Wright & Masters Introduction to Rasch Measurement, E. Smith & R. Smith Introduction to Many-Facet Rasch Measurement, Thomas Eckes Invariant Measurement: Using Rasch Models in the Social, Behavioral, and Health Sciences, George Engelhard, Jr. Statistical Analyses for Language Testers, Rita Green
Rasch Models: Foundations, Recent Developments, and Applications, Fischer & Molenaar Journal of Applied Measurement Rasch models for measurement, David Andrich Constructing Measures, Mark Wilson Rasch Analysis in the Human Sciences, Boone, Stave, Yale
in Spanish: Análisis de Rasch para todos, Agustín Tristán Mediciones, Posicionamientos y Diagnósticos Competitivos, Juan Ramón Oreja Rodríguez

To be emailed about new material on www.rasch.org
please enter your email address here:

I want to Subscribe: & click below
I want to Unsubscribe: & click below

Please set your SPAM filter to accept emails from Rasch.org

www.rasch.org welcomes your comments:

Your email address (if you want us to reply):

 

ForumRasch Measurement Forum to discuss any Rasch-related topic

Go to Top of Page
Go to index of all Rasch Measurement Transactions
AERA members: Join the Rasch Measurement SIG and receive the printed version of RMT
Some back issues of RMT are available as bound volumes
Subscribe to Journal of Applied Measurement

Go to Institute for Objective Measurement Home Page. The Rasch Measurement SIG (AERA) thanks the Institute for Objective Measurement for inviting the publication of Rasch Measurement Transactions on the Institute's website, www.rasch.org.

Coming Rasch-related Events
Dec. 7-9, 2016, Wed.-Fri. In-person workshop: Introductory Rasch (M. Horton, RUMM), Leeds, UK, www.leeds.ac.uk/medicine/rehabmed/psychometric
Dec. 16, 2016, Fri. Psychometrician's Day (Elena Kardanova), Higher School of Economics, Moscow, Russia
Jan. 6 - Feb. 3, 2017, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
March 31, 2017, Fri. Conference: 11th UK Rasch Day, Warwick, UK, www.rasch.org.uk
April 2-3, 2017, Sun.-Mon. Conference: Validity Evidence for Measurement in Mathematics Education (V-M2Ed), San Antonio, TX, Information
May 26 - June 23, 2017, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 30 - July 29, 2017, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
July 31 - Aug. 3, 2017, Mon.-Thurs. Joint IMEKO TC1-TC7-TC13 Symposium 2017: Measurement Science challenges in Natural and Social Sciences, Rio de Janeiro, Brazil, imeko-tc7-rio.org.br
Aug. 11 - Sept. 8, 2017, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com
Aug. 18-21, 2017, Fri.-Mon. IACAT 2017: International Association for Computerized Adaptive Testing, Niigata, Japan, iacat.org
Oct. 13 - Nov. 10, 2017, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
Jan. 5 - Feb. 2, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
Jan. 10-16, 2018, Wed.-Tues. In-person workshop: Advanced Course in Rasch Measurement Theory and the application of RUMM2030, Perth, Australia (D. Andrich), Announcement
Jan. 17-19, 2018, Wed.-Fri. Rasch Conference: Matilda Bay Club, Perth, Australia, Website
May 25 - June 22, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 29 - July 27, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
Aug. 10 - Sept. 7, 2018, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com
Oct. 12 - Nov. 9, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
The HTML to add "Coming Rasch-related Events" to your webpage is:
<script type="text/javascript" src="http://www.rasch.org/events.txt"></script>

The URL of this page is www.rasch.org/rmt/rmt83b.htm

Website: www.rasch.org/rmt/contents.htm