Table 1. RMSEA Results for Set 1 (10 polytomous items) | |||
Sample Size | No Misfit | 10% Misfit | 20% Misfit |
200 | 0.000 | 0.000 | 0.033 |
500 | 0.004 | 0.024 | 0.035 |
2000 | 0.011 | 0.024 | 0.030 |
5000 | 0.014 | 0.024 | 0.031 |
10000 | 0.014 | 0.024 | 0.031 |
Table 2. RMSEA Results for Set 2 (20 polytomous items) | |||
Sample Size | No Misfit | 10% Misfit | 20% Misfit |
200 | 0.000 | 0.053 | 0.043 |
500 | 0.000 | 0.024 | 0.040 |
2000 | 0.004 | 0.031 | 0.038 |
5000 | 0.006 | 0.030 | 0.038 |
10000 | 0.009 | 0.031 | 0.038 |
Table 3. RMSEA Results for Set 3 (20 dichotomous items) | |||
Sample Size | No Misfit | 10% Misfit | 20% Misfit |
200 | 0.000 | 0.061 | 0.073 |
500 | 0.016 | 0.019 | 0.035 |
2000 | 0.013 | 0.026 | 0.040 |
5000 | 0.011 | 0.027 | 0.040 |
10000 | 0.012 | 0.027 | 0.041 |
Georg Rasch mentioned chi-square statistics as a way of evaluating fit of data to the model (Rasch, 1980, p. 25). Ben Wright's Infit and Outfit mean-square statistics are the chi-square divided by their degrees of freedom. However, large sample sizes have always posed problems for significance tests based on chi-square statistics. The issue is that, the larger the sample, the greater the power, and so ever smaller differences are reported as indicating statistically significant misfit between the data and the model. Thus very large sample sizes can detect miniscule differences, and with such samples there is almost no need to undertake a chi-square test as we know that it will be significant (P. Martin-Löf (1974). Indeed, Georg Rasch himself remarked: "On the whole we should not overlook that since a model is never true, but only more or less adequate, deficiencies are bound to show, given sufficient data" (Rasch, 1980, p. 92).
Smith et al. (1998) show that the critical interval values for a Type I error (rejection of a true hypothesis) associated with these statistics varies with sample size. Experience indicates that, while the value of mean-square tends to increase only slowly with sample size, the critical interval associated with a 5% significance level shrinks considerably as sample size increases. Thus a sample of 50 would have a 5% range for Infit of 0.72-1.28, whereas a sample of 500 would have a 5% range of 0.91-1.09. A sample size of 5000 would have a 5% range of 0.97-1.03 (RMT 17:1 p. 918).
In general, large sample sizes will cause most chi-square-based statistics to almost always report a statistically significant difference between the observed data and model expectations, suggesting misfit, regardless of the true situation.
One potential mechanism for accommodating large sample sizes may be to use the Root Mean Square Error of Approximation (RMSEA, Steiger and Lind, 1980) as a supplementary fit. The RMSEA is widely used in Structural Equation Modeling to provide a mechanism for adjusting for sample size where chi-square statistics are used.
Consequently, we set out to test the potential of the RMSEA to supplement the chi-square fit tests reported for Rasch analyses performed by RUMM2030. This investigation focuses on the "summary fit chi-square" (the item trait interaction statistic). The utility of the RMSEA to supplement the interpretation of the chi square fit in larger samples was assessed, along with determination of the level of RMSEA that is consistent with fit to the Rasch model.
Methods
A number of simulations were undertaken with the RUMMss simulation package (Marais I, Andrich D, 2007). Two polytomous item sets of 10 and 20 items with five response categories were simulated with different degrees of fit to the Rasch model. In addition, a set of dichotomous (30) items were also simulated. Perfect fit (100% of the items with simulated discriminations of 1.0), minor deviations (90% with 1.0, 10% with 3.0) and more serious deviations from model expectations (80% with 1.0, 20% with 3.0) were simulated. Each set of simulations was repeated for 200, 500, 2000, 5000, and 10,000 cases. All other parameters were held constant.
The RMSEA was calculated for each simulation, based upon the summary chi-square interaction statistic reported by RUMM2030. The RMSEA formulae can be shown to be equal to:
RMSEA = √ max( [((χ²/df) - 1)/(N - 1)] , 0)
where χ² is the RUMM2030 chi-square value, df is its degrees of freedom and N is the sample size. Notice that the RMSEA has an expected value of zero when the data fit the model. Overfit of the data to the model, χ²/df < 1, is ignored. For a given χ², RMSEA decreases as sample size, N, increases.
Results
In Tables 1-3, the average RMSEA for each simulated condition is reported. Within each column of each Table, the RMSEA is largely invariant as the sample size increases, as we had hoped.
Across each row of each Table, for sample sizes of 500 or more, the RMSEA is sensitive to increasing misfit. Thus it may be appropriate to use this supplementary fit statistic in the presence of sample sizes of 500 or more cases, to inform if sample size is inflating the chi-square statistic, and hence its significance.
Conclusion
The results of this study suggest that investigations of fit to the Rasch model using RUMM2030 and specifically the item-trait interaction chi-square fit statistic, in the presence of large sample sizes, can be supplemented through applying the RMSEA statistic. RMSEA values of < 0.02 with sample sizes of 500+, and certainly 1000+, may indicate that the data do not underfit the model, and that the chi-square was inflated by sample size.
Alan Tennant, Department of Rehabilitation Medicine, Faculty of Medicine and Health, The University of Leeds, UK
Julie F. Pallant, Rural Health Academic Centre, University of Melbourne, Australia.
References
Marais I, Andrich D (2007)\: RUMMss. Rasch Unidimensional Measurement Models Simulation Studies Software. The University of Western Australia, Perth.
Martin-Löf P. (1974). The notion of redundancy and its use as a quantitative measure of the discrepancy between a statistical hypothesis and observational data. Scandinavian Journal of Statistics, 1:3.
Rasch, G. (1980). Probabilistic models for some intelligence and attainment tests. Chicago: University of Chicago Press.
Smith, R. M, Schumacker RE, Bush MJ. (1998). Using item mean squares to evaluate fit to the Rasch model. Journal of Outcome Measurement, 2: 66-78.
Steiger, J. H. and Lind, J. (1980) Statistically-based tests for the number of common factors. Paper presented at the Annual Spring Meeting of the Psychometric Society, Iowa City.
The Root Mean Square Error of Approximation (RMSEA) as a supplementary statistic to determine fit to the Rasch model with large sample sizes. Alan Tennant & Julie F. Pallant ... Rasch Measurement Transactions, 2012, 25:4, 1348-9
Rasch Publications | ||||
---|---|---|---|---|
Rasch Measurement Transactions (free, online) | Rasch Measurement research papers (free, online) | Probabilistic Models for Some Intelligence and Attainment Tests, Georg Rasch | Applying the Rasch Model 3rd. Ed., Bond & Fox | Best Test Design, Wright & Stone |
Rating Scale Analysis, Wright & Masters | Introduction to Rasch Measurement, E. Smith & R. Smith | Introduction to Many-Facet Rasch Measurement, Thomas Eckes | Invariant Measurement: Using Rasch Models in the Social, Behavioral, and Health Sciences, George Engelhard, Jr. | Statistical Analyses for Language Testers, Rita Green |
Rasch Models: Foundations, Recent Developments, and Applications, Fischer & Molenaar | Journal of Applied Measurement | Rasch models for measurement, David Andrich | Constructing Measures, Mark Wilson | Rasch Analysis in the Human Sciences, Boone, Stave, Yale |
in Spanish: | Análisis de Rasch para todos, Agustín Tristán | Mediciones, Posicionamientos y Diagnósticos Competitivos, Juan Ramón Oreja Rodríguez |
Forum | Rasch Measurement Forum to discuss any Rasch-related topic |
Go to Top of Page
Go to index of all Rasch Measurement Transactions
AERA members: Join the Rasch Measurement SIG and receive the printed version of RMT
Some back issues of RMT are available as bound volumes
Subscribe to Journal of Applied Measurement
Go to Institute for Objective Measurement Home Page. The Rasch Measurement SIG (AERA) thanks the Institute for Objective Measurement for inviting the publication of Rasch Measurement Transactions on the Institute's website, www.rasch.org.
Coming Rasch-related Events | |
---|---|
Aug. 11 - Sept. 8, 2023, Fri.-Fri. | On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com |
Aug. 29 - 30, 2023, Tue.-Wed. | Pacific Rim Objective Measurement Society (PROMS), World Sports University, Macau, SAR, China https://thewsu.org/en/proms-2023 |
Oct. 6 - Nov. 3, 2023, Fri.-Fri. | On-line workshop: Rasch Measurement - Core Topics (E. Smith, Facets), www.statistics.com |
June 12 - 14, 2024, Wed.-Fri. | 1st Scandinavian Applied Measurement Conference, Kristianstad University, Kristianstad, Sweden http://www.hkr.se/samc2024 |
The URL of this page is www.rasch.org/rmt/rmt254d.htm
Website: www.rasch.org/rmt/contents.htm