"Less control over the data → more off-dimensional behavior → more departures in the data from the Rasch model"
Though the ideal for measurement construction is that data fit the Rasch model, all empirical data departs from the model to some extent. But how much unmodelled noise is tolerable? Conventional statisticians base their decisions on significance tests, but these are heavily influenced by sample size.
"Very large samples form a special source of problems. This is because no model can ever be supposed to be perfectly fitted by data, so with a sufficiently large sample any model would have to be discarded. In connection with the problem Martin-Löf (1974) stated: 'This indicates that for large sets of data it is too destructive to let an ordinary significance test decide whether or not to accept a proposed statistical model [or data], because, with few exceptions, we know that we shall have to reject it even without looking at the data simply because the number of observations is so large. In such cases, we need instead a quantitative measure of the size of the discrepancy between the statistical model and the observed set of data.'" (Gustafson 1980).
Convenient quantitative measures of fit discrepancy are mean-square residual summary statistics, such as OUTFIT and INFIT. These statistics have expectation 1.0, and range from 0 to infinity. Mean-squares greater than 1.0 indicate underfit to the Rasch model, i.e., the data are less predictable than the model expects. Mean-squares less than 1.0 indicate overfit to the Rasch model, i.e., the data are more predictable than the model expects. A mean-square of 1.2 indicates that there is 20% more randomness (i.e., noise) in the data than modelled. A mean-square of 0.7 indicates a 30% deficiency in Rasch-model-predicted randomness (i.e., the data are too Guttman-like), which implies 100*(1-0.7)/0.7 = 43% more ambiguity in the inferred measure than modelled (e.g., the item difficulty estimated from low-ability persons differs noticeably from the item difficulty estimated from high-ability persons).
Reasonable Item Mean-square Ranges for INFIT and OUTFIT | |
---|---|
Type of Test | Range |
MCQ (High stakes) MCQ (Run of the mill) Rating scale (survey) Clinical observation Judged (agreement encouraged) |
0.8 - 1.2 0.7 - 1.3 0.6 - 1.4 0.5 - 1.7 0.4 - 1.2 |
When is a mean-square too large or too small? There are no hard-and-fast rules. Particular features of a testing situation, e.g., mixing item types or off-target testing, can produce idiosyncratic mean-square distributions. Nevertheless, here, as a rule of thumb, are some reasonable ranges for item mean-square fit statistics. In Classical Test Theory, 3-PL IRT, and conventional statistics, low mean-squares are considered good. In Rasch theory, they indicate some redundancy in the responses, but they do no harm.
Ben Wright & John Michael Linacre
J.-E. Gustafson (1980) Testing and obtaining fit of data to the Rasch model. British Journal of mathematical and Statistical Psychology, 33, p.220.
P. Martin-Löf (1974) The notion of redundancy and its use as a quantitative measure of the discrepancy between a statistical hypothesis and observational data. Scandinavian Journal of Statistics, 1, 3.
Note by Linacre: Informal simulations studies and experience analyzing hundreds of datasets indicate that:
Interpretation of parameter-level mean-square fit statistics: | |
---|---|
>2.0 | Distorts or degrades the measurement system |
1.5 - 2.0 | Unproductive for construction of measurement, but not degrading |
0.5 - 1.5 | Productive for measurement |
<0.5 | Less productive for measurement, but not degrading. May produce misleadingly good reliabilities and separations |
The Mean-Square statistic is also called the Relative Chi-Square and the Normed Chi-Square. According to www.psych-it.com.au/Psychlopedia/article.asp?id=277, in Structural Equation Modeling, the criterion for acceptance varies across researchers, ranging from less than 2 (Ullman, 2001) to less than 5 (Schumacker & Lomax, 2004).
[Later:] Overfit (mean-squares less than 1.0).
See also "Size vs. Significance: Mean-Square and Standardized Chi-Square Fit Statistics" www.rasch.org/rmt/rmt171n.htm
Item Fit and Person Fit
From a statistical Rasch perspective, persons and items are exactly the same. They are merely parameters of the Rasch model. So the fit criteria would be exactly the same. But, from a substantive perspective, persons and items differ. We expect the items to be better-behaved than the persons. We also expect item difficulties to continue into the future, but we expect person abilities to change. Also, we expect items to be encountered by many, many persons, but persons to encounter relatively few items. Consequently, we are usually stricter in our application of fit rules to items than to persons. A few maverick persons in a dataset don't worry us - they will have negligible impact on anything else. But a few maverick items raise questions about test administration, data entry accuracy, the definition of the latent variable, etc. We will immediately focus our attention on them because they may be symptomatic of a more pervasive problem, such as the wrong key for a multiple-choice test, or reversed-coded items on a survey.
Reasonable mean-square fit values. Wright BD, Linacre JM … Rasch Measurement Transactions, 1994, 8:3 p.370
Rasch Publications | ||||
---|---|---|---|---|
Rasch Measurement Transactions (free, online) | Rasch Measurement research papers (free, online) | Probabilistic Models for Some Intelligence and Attainment Tests, Georg Rasch | Applying the Rasch Model 3rd. Ed., Bond & Fox | Best Test Design, Wright & Stone |
Rating Scale Analysis, Wright & Masters | Introduction to Rasch Measurement, E. Smith & R. Smith | Introduction to Many-Facet Rasch Measurement, Thomas Eckes | Invariant Measurement: Using Rasch Models in the Social, Behavioral, and Health Sciences, George Engelhard, Jr. | Statistical Analyses for Language Testers, Rita Green |
Rasch Models: Foundations, Recent Developments, and Applications, Fischer & Molenaar | Journal of Applied Measurement | Rasch models for measurement, David Andrich | Constructing Measures, Mark Wilson | Rasch Analysis in the Human Sciences, Boone, Stave, Yale |
in Spanish: | Análisis de Rasch para todos, Agustín Tristán | Mediciones, Posicionamientos y Diagnósticos Competitivos, Juan Ramón Oreja Rodríguez |
Forum | Rasch Measurement Forum to discuss any Rasch-related topic |
Go to Top of Page
Go to index of all Rasch Measurement Transactions
AERA members: Join the Rasch Measurement SIG and receive the printed version of RMT
Some back issues of RMT are available as bound volumes
Subscribe to Journal of Applied Measurement
Go to Institute for Objective Measurement Home Page. The Rasch Measurement SIG (AERA) thanks the Institute for Objective Measurement for inviting the publication of Rasch Measurement Transactions on the Institute's website, www.rasch.org.
Coming Rasch-related Events | |
---|---|
Jan. 25 - March 8, 2023, Wed..-Wed. | On-line course: Introductory Rasch Analysis (M. Horton, RUMM2030), medicinehealth.leeds.ac.uk |
June 23 - July 21, 2023, Fri.-Fri. | On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com |
Aug. 11 - Sept. 8, 2023, Fri.-Fri. | On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com |
The URL of this page is www.rasch.org/rmt/rmt83b.htm
Website: www.rasch.org/rmt/contents.htm