Polytomous Mean-Square Fit Statistics

Smith R.M. (1996) Polytomous Mean-Square Fit Statistics. Rasch Measurement Transactions 10:3 p. 516-517.

Fit statistics in Rasch analysis serve a different purpose from those in regression analysis. In descriptive statistical methodology, fit statistics are used to discover a model that fits the data well enough that the data could be considered to have been generated by the model. In Rasch analysis, the model is already chosen. The purpose of the fit statistics is to aid in measurement quality control, to identify those parts of the data which meet Rasch model specifications and those parts which don't. Parts that don't are not automatically rejected, but are examined to identify in what way, and why, they fall short, and whether, on balance, they contribute to or corrupt measurement. Then the decision is made to accept, reject or modify the data. Modification includes simple actions such as correcting obvious data entry errors and respondent mistakes, and more sophisticated actions such as collapsing rating scale categories.

We have 30 years of experience investigating mean-square (MnSq) statistics for dichotomous data (RMT 8:2 360 www.rasch.org/rmt/rmt82a.htm). The interpretation of fit statistics for polytomous items, however, is a recent development.

One subtlety of rating scale fit analysis is the detection of idiosyncratic category usage, particularly respondents' over-use of central or extreme categories. The Table illustrates response strings and their diagnostic fit statistics. The responses are reported from left-to-right in descending order of expected values. Representative values have been chosen for item and step calibrations with a 4 category (0-3) rating scale. The response strings that best fit the Rasch model (Section I) descend in value stochastically. They exhibit MnSq's near 1.0 and positive point-measure correlations (which are similar to point-biserial correlations, but correlate responses with Rasch measures rather than raw scores).

In Section II, The Guttman pattern matches the expectations as closely as possible. As a result, it has low MnSq statistics. MnSq's less than 1.0 indicate better than expected fit to the model. These responses agree with, but add little additional information to, other responses. This pattern also has high point-measure correlation.

In Section III, other over-fitting (muted) response strings include those in which the full range of the categories is not employed. Their low MnSq statistics and high correlations seem to mark them as matching the model especially well, but low MnSq statistics indicate a lack of statistical information, here resulting from a category range restriction. Raters trying not to contradict other raters may emphasize central categories or exhibit less discrimination (Section II) and so be reported with low MnSq's, i.e., as less statistically informative.

Section IV depicts response strings that contain useful measurement information, but also challenge the construct hierarchy. To the naked eye, response strings in Section IV look much like those in Section I. MnSq's above 1.0 indicate the presence of unmodelled variance (noise) along with the useful information in the responses. At some level, the noise in the responses overwhelms the information (music) and the response string, as it stands, is no longer assisting in measurement construction. Looking at Sections V and VI, it appears that MnSq's of 1.5 or more, and correlations near or below 0, are indicative of disruptive response strings for these data.

Section V presents non-informative response sets, used by respondents to avoid engaging the rating scale. The symptom here is the lack of relationship between the responses and the construct. Measurement construction would be aided by omitting or pruning such response strings. Once the measurement framework has been constructed and established with anchor measures and calibrations, all response strings may be reported so that the misfit statistics can be used to inform the use of the measures. Paradoxically, perfect agreement, in which all raters rate an examinee with the same category is not an ideal of Rasch measurement. Unanimity of choice of response category contributes no information about the relative standing of the categories, but implies that the category is so wide that large differences in perceived levels of performance are still classified under that one category.

Section VI demonstrates response strings that occur when prompts are misunderstood or miscoded. Large fit statistics and negative correlations flag response strings that are coded "backwards". Except with unusual patterns of missing data or rank ordering, response patterns with negative correlations should be recoded or omitted from the measurement system.

Richard M. Smith
Rehabilitation Foundation Inc.
P.O. Box 675, Wheaton IL 60189-9931

This is the BIGSTEPS control file for the data above:

&inst
TITLE='COMPUTING STATISTICS'
NI=20
ITEM1=1           ; include response strings in person name
name1=1
namlen=30
CODES=0123
ptbis=no          ; compute point-measure correlation
INUMB=YES         ; no item labels
TFILE=*
6        ; Table 6 - persons in fit order
18       ; table 18 - persons in entry order
*
IAFILE=*          ; item anchor values - uniform
1 -1.9
2 -1.7
3 -1.5
4 -1.3
5 -1.1
6 -0.9
7 -0.7
8 -0.5
9 -0.3
10 -0.1
11 0.1
12 0.3
13 0.5
14 0.7
15 0.9
16 1.1
17 1.3
18 1.5
19 1.7
20 1.9
*
SAFILE=*          ; step anchor values
0 0
1 -1
2 0
3 1
*
&end
33333132210000001011 modelled
31332332321220000000 modelled
33333331122300000000 modelled
33333331110010200001 modelled
33222222221111111100 most expected        
33333222221111100000 most likely          
33333333221100000000 high discrimination  
32222222221111111110 low discrimination   
32323232121212101010 tight progression    
33333333332222222222 high (low) categories
22222222221111111111 2 central categories 
33333322222222211111 only 3 categories    
32222222201111111130 noisy outliers       
33233332212333000000 erratic transitions  
33333333330000000000 extreme categories   
33133330232300101000 noisy progression    
22222222222222222222 one category         
12121212121212121212 central flip-flop    
03202002101113311002 random responses     
01230123012301230123 rotate categories    
03030303030303030303 extreme flip-flop    
11111122233222111111 folded pattern       
22222222223333333333 high reversal        
11111111112222222222 central reversal     
00111111112222222233 Guttman reversal     
00000000003333333333 extreme reversal     

Polytomous mean-square fit statistics. Smith R.M. … Rasch Measurement Transactions, 1996, 10:3 p. 516-517.

The URL of this page is www.rasch.org/rmt/rmt103a.htm

Website: www.rasch.org/rmt/contents.htm