Reasonable mean-square fit values.

"Less control → more off-dimensional behavior → more departures in the data from the Rasch model"

Though the ideal for measurement construction is that data fit the Rasch model, all empirical data departs from the model to some extent. But how much unmodelled noise is tolerable? Conventional statisticians base their decisions on significance tests, but these are heavily influenced by sample size.

"Very large samples form a special source of problems. This is because no model can ever be supposed to be perfectly fitted by data, so with a sufficiently large sample any model would have to be discarded. In connection with the problem Martin-Löf (1974) stated: 'This indicates that for large sets of data it is too destructive to let an ordinary significance test decide whether or not to accept a proposed statistical model [or data], because, with few exceptions, we know that we shall have to reject it even without looking at the data simply because the number of observations is so large. In such cases, we need instead a quantitative measure of the size of the discrepancy between the statistical model and the observed set of data.'" (Gustafson 1980).

Convenient quantitative measures of fit discrepancy are mean-square residual summary statistics, such as OUTFIT and INFIT. These statistics have expectation 1.0, and range from 0 to infinity. Mean-squares greater than 1.0 indicate underfit to the Rasch model, i.e., the data are less predictable than the model expects. Mean-squares less than 1.0 indicate overfit to the Rasch model, i.e., the data are more predictable than the model expects. A mean-square of 1.2 indicates that there is 20% more randomness (i.e., noise) in the data than modelled. A mean-square of 0.7 indicates a 30% deficiency in Rasch-model-predicted randomness (i.e., the data are too Guttman-like), which implies 100*(1-0.7)/0.7 = 43% more ambiguity in the inferred measure than modelled (e.g., the item difficulty estimated from low-ability persons differs noticeably from the item difficulty estimated from high-ability persons).

When is a mean-square too large or too small? There are no hard-and-fast rules. Particular features of a testing situation, e.g., mixing item types or off-target testing, can produce idiosyncratic mean-square distributions. Nevertheless, here, as a rule of thumb, are some reasonable ranges for item mean-square fit statistics. Please tell us the ranges you have found useful in your own work.

Ben Wright & Mike Linacre

J.-E. Gustafson (1980) Testing and obtaining fit of data to the Rasch model. British Journal of mathematical and Statistical Psychology, 33, p.220.

P. Martin-Löf (1974) The notion of redundancy and its use as a quantitative measure of the discrepancy between a statistical hypothesis and observational data. Scandinavian Journal of Statistics, 1, 3.

Reasonable Item Mean-square Ranges for INFIT and OUTFIT
Type of Test	Range
MCQ (High stakes) MCQ (Run of the mill) Rating scale (survey) Clinical observation Judged (agreement encouraged)	0.8 - 1.2 0.7 - 1.3 0.6 - 1.4 0.5 - 1.7 0.4 - 1.2

Note by Linacre: Informal simulations studies and experience analyzing hundreds of datasets indicate that:

[Later:] Overfit (mean-squares less than 1.0).

1. Mean-squares are forced to average near 1.0. So, if there are very large mean-squares (for instance, due to random lucky guessing), this forces there also to be low mean-squares.
   Advice: always investigate and remove items with high mean-squares before looking at items with low mean-squares.
2. Mean-squares less than 1.0 are too predictable. They do not contradict what we know, but they do not tell us much that is new about what we want to know. So overfitting items are inefficient.
   Advice:
   i) if we are developing a new test: replace overfitting items with more efficient items.
   ii) if we are analyzing data from an existing test: take no action, even an item with a very low mean-square tells us a little something that is new and useful. We don't want to waste any useful information.

Reasonable mean-square fit values. Wright BD, Linacre JM … Rasch Measurement Transactions, 1994, 8:3 p.370

The URL of this page is www.rasch.org/rmt/rmt83b.htm

Website: www.rasch.org/rmt/contents.htm