Tuneable Goodness-of-Fit Statistics

One of the persistent problems in psychometrics is the determination of the goodness-of-fit between observed and expected values. The problem is particularly tricky with discrete multivariate data that form the basis for measurement in the social, behavioral, and health sciences.

Early work in statistics led to Pearson's chi-square statistic (Pearson, 1900). The chi-square statistic has been quite robust and useful in a variety of applications. Several researchers have proposed adaptations and improvements of chi-square statistics that have ranged from adjustments in the degrees of freedom (Fisher, 1924) to the development of the closely related loge likelihood ratio statistic (Wilks, 1935). Unfortunately, the assumptions of the Pearson chi-square statistic are not always met, and therefore the χ2 sampling distribution is not necessarily a useful guide for judgments regarding model-data fit.

The purpose of this note is to describe a family of tuneable goodness-of-fit statistics based on the Power Divergence (PD) Statistics (Cressie & Read, 1988). Tuneable goodness-of-fit statistics offer a useful approach for examining both person and item fit that has not been explored with Rasch measurement models.

The basic equation for tuneable statistics, τ2, is

where Oi is the observed frequency in a cell i, Ei is the expected frequency for cell i based on the model, and k is the number of cells. Tuneable goodness-of-fit statistics can be obtained by inserting the appropriate λ value. The λ values can range from -∞ to +∞.

In order to illustrate the use of tuneable statistics, data from Stouffer and Toby (1951) are presented in Table 1. These data are used to illustrate the obtained estimates of τ2 for several λ values. See Engelhard (2008) for additional details regarding the Stouffer-Toby scale, as well as the calculation of conditional probabilities and expected frequencies based on Rasch analyses.

Table 1. Stouffer-Toby (1951) Data

Note. Rasch item difficulties are -1.89, -.20, -.10, and 2.20 logits for items A to D respectively.

Conditional probabilities and expected frequencies are based on the Rasch model.

Table 2. Values of the Tuneable Goodness-of-fit statistics

Table 2 presents the values for various goodness-of-fit statistics with λ values reported at various points between -3.00 and 3.00. Some of these λ values correspond to other goodness-of-fit statistics, and these are labeled in the Table 2. The 95th percentile of the chi-squared distribution with 17 degrees of freedom is χ2 (17, p=.05) = 27.59. Based on this value, we conclude that the goodness-of-fit is quite good between the observed and expected frequencies based on the Rasch model. Only one of the estimated values suggests rejecting the null hypothesis (λ value = 2).

This note describes a potentially useful set of tuneable goodness-of-fit statistics. It is important to recognize that explorations of goodness-of-fit should not involve a simple decision (e.g., reject the null hypothesis), but also require judgments and "cognitive evaluations of propositions" (Rozeboom, 1960, p. 427).

Additional research is needed on the utility of these tuneable statistics for making judgments regarding overall goodness-of-fit, item and person fit, and various approaches for defining and conducting residual analyses within the framework of Rasch measurement. This research should include research on the sampling distributions for various tuneable statistics applied to different aspects of goodness-of-fit, research on appropriate degrees of freedom, and research on the versions of the τ2 statistic that yield the most relevant substantive interpretations within the context of Rasch measurement theory and the construct being measured.

George Engelhard, Jr.

Emory University

Engelhard, G. (2008). Historical perspectives on invariant measurement: Guttman, Rasch, and Mokken. Measurement: Interdisciplinary Research and Perspectives (6), 1-35.

Fisher, R.A. (1924). The conditions under which χ2 measures the discrepancy between observations and hypothesis. Journal of the Royal Statistical Society, 87, 442-450.

Freeman, D.H., & Tukey, J.W. (1950). Transformations related to the angular and the square root. Annals of Mathematical Statistics, 21, 607-611.

Kullback, S. (1959). Information theory and statistics. New York: John Wiley.

Neyman, J. (1949). Contribution to the theory of the χ2 test. Proceedings of the First Berkeley Symposium on Mathematical statistics and Probability, 239-273.

Pearson, K. (1900). On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling. Philosophy Magazine, 50, 157-172.

Read, T.R.C., & Cressie, N.A.C. (1988). Goodness-of-fit for discrete multivariate data. New York: Springer-Verlag.

Rozeboom, W.W. (1960). The fallacy of the null-hypothesis significance test. Psychological Bulletin, 57, 416-428.

Stouffer, S.A. & Toby, J. (1951). Role conflict and personality. The American Journal of Sociology, 56, 395-406.

Wilks, S.S. (1935). The likelihood test of independence in contingency tables. Annals of Mathematical Statistics, 6, 190-196.

Tuneable Goodness-of-Fit Statistics. Engelhard, G. Jr. … Rasch Measurement Transactions, 2008, 22:1 p. 1158-9

Rasch Publications
Rasch Measurement Transactions (free, online) Rasch Measurement research papers (free, online) Probabilistic Models for Some Intelligence and Attainment Tests, Georg Rasch Applying the Rasch Model 3rd. Ed., Bond & Fox Best Test Design, Wright & Stone
Rating Scale Analysis, Wright & Masters Introduction to Rasch Measurement, E. Smith & R. Smith Introduction to Many-Facet Rasch Measurement, Thomas Eckes Invariant Measurement: Using Rasch Models in the Social, Behavioral, and Health Sciences, George Engelhard, Jr. Statistical Analyses for Language Testers, Rita Green
Rasch Models: Foundations, Recent Developments, and Applications, Fischer & Molenaar Journal of Applied Measurement Rasch models for measurement, David Andrich Constructing Measures, Mark Wilson Rasch Analysis in the Human Sciences, Boone, Stave, Yale
in Spanish: Análisis de Rasch para todos, Agustín Tristán Mediciones, Posicionamientos y Diagnósticos Competitivos, Juan Ramón Oreja Rodríguez

To be emailed about new material on www.rasch.org
please enter your email address here:

I want to Subscribe: & click below
I want to Unsubscribe: & click below

Please set your SPAM filter to accept emails from Rasch.org

www.rasch.org welcomes your comments:

Your email address (if you want us to reply):


ForumRasch Measurement Forum to discuss any Rasch-related topic

Go to Top of Page
Go to index of all Rasch Measurement Transactions
AERA members: Join the Rasch Measurement SIG and receive the printed version of RMT
Some back issues of RMT are available as bound volumes
Subscribe to Journal of Applied Measurement

Go to Institute for Objective Measurement Home Page. The Rasch Measurement SIG (AERA) thanks the Institute for Objective Measurement for inviting the publication of Rasch Measurement Transactions on the Institute's website, www.rasch.org.

Coming Rasch-related Events
June 23 - July 21, 2023, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
Aug. 11 - Sept. 8, 2023, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com


The URL of this page is www.rasch.org/rmt/rmt221l.htm

Website: www.rasch.org/rmt/contents.htm