Effective data analysis begins with the realization that neither the model nor the data are true. Both are inventions and fictions. The purpose of a model is (1) to represent important requirements in a decisive way (the more decisive the better) and (2) to be useful for the construction of knowledge. Throughout physics, chemistry, business and sports, a central requirement for a measurement model is that, as far as the model is concerned, comparisons of objects are independent of which agents are used, and comparisons of agents are independent of which objects are used. The model which operates in this manner to construct quantities from ordered qualities is the Rasch model.
The Rasch model is a conceptual ideal. No data can ever fit this (or any other) model. But the degree to which the data do cooperate with the model, and the local departures of the data from the ideal of the model, guide the management of the data. If at all possible, neither the data nor the model is accepted or rejected totally. Since the model represents our requirements, to reject it is to reject our line of inquiry. Since the data set contains what the world has to tell us, to reject it is to state that the world, or, at least that part of it, has no information relating to our concern. Very rarely, a variable is so poorly conceptualized, or a data set so confused, that no meaningful analysis can be performed. Then it is the data set, not the model, that is rejected.
The measurement model provides local indicators, fit statistics, of the degree to which the data is cooperating with the model's requirements. These local indicators identify pieces of data consistent with the main pattern, and also those least consistent. Those data consistent with the main pattern, usually the vast majority for any data set collected with any degree of thought and care, become the basis for measures and the construction of knowledge. The inconsistent data become a source of further discovery. They need to be studied in more detail, and, when the inconsistency is due to irrelevant activity, those pieces of data need to be put to one side or manipulated to be in accord with the main pattern.
When diagnosis of inconsistent data indicates that an examinee has a spoiled performance, (perhaps there is clear evidence of guessing, carelessness or such like,) the examinee's measure is also spoiled. To draw inferences from this measure about the ability of the examinee is to risk being misinformed. Sometimes the measure can be rescued by putting to one side the spoiled part of the data, and using only the part that cooperates with the model. Sometimes there is nothing to rescue. The data provides no firm base from which to infer the examinee's ability. Of course, if the purpose of the analysis is merely to award a prize for the most "correctly" marked squares on an answer sheet, even coffee stains, smudges and doodles count! But that is not science.
When an item has not functioned in the same way as most other items, it is not acting as an agent assisting in the measurement of the same variable. It is acting like a broken ruler. It is contributing not to measurement, but to noise, until either it is removed from the analysis, or, preferably mended. A common form of repair is to correct an errant scoring key.
What is the precise role of fit statistics in identifying spoiled performances or broken items? Is accurate knowledge of the distributions of the fit statistics for unspoiled performances, the "null" distributions, necessary for data analysis to proceed fairly, rationally, systematically, routinely and automatically? Automatically?? Empirical data are too complex in their origin for any automatic decision maker to be entirely successful. Fit statistics provide guidance, but not decisions. Decisions must be made within a larger context and for richer reasons than the null distribution of a fit statistic can provide. A substantive explanation of data identified as irregular is more important than the absolute magnitude of significance level of some fit statistic, chosen purely for its ability to aid the hunt for inconsistency.
Even were the null distribution available, it too is an ideal from which any real set of data inevitably falls short. Therefore, it is sufficient for thoughtful data analysis to know that the fit statistics can be relied upon (1) to order the persons from most underfitting to most overfitting, (2) to center this ordering near some easily remembered numerical value, such as 0 or 1, as a guide to where the transition is from under to overfitting and (3) to provide some frame of reference, so that a fit statistic of value, say 2, continues to represent about the same degree of spoiling for similar data collected under similar conditions. Whatever the values of the fit statistics, extreme cases require examination to discover why they are spoiled, or to confirm their freedom from spoilage.
Ben Wright
Data analysis and fit. Wright BD. Rasch Measurement Transactions 1994 7:4 p.324
Data analysis and fit. Wright BD. Rasch Measurement Transactions, 1994, 7:4 p.324
Forum | Rasch Measurement Forum to discuss any Rasch-related topic |
Go to Top of Page
Go to index of all Rasch Measurement Transactions
AERA members: Join the Rasch Measurement SIG and receive the printed version of RMT
Some back issues of RMT are available as bound volumes
Subscribe to Journal of Applied Measurement
Go to Institute for Objective Measurement Home Page. The Rasch Measurement SIG (AERA) thanks the Institute for Objective Measurement for inviting the publication of Rasch Measurement Transactions on the Institute's website, www.rasch.org.
Coming Rasch-related Events | |
---|---|
Oct. 4 - Nov. 8, 2024, Fri.-Fri. | On-line workshop: Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com |
Jan. 17 - Feb. 21, 2025, Fri.-Fri. | On-line workshop: Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com |
May 16 - June 20, 2025, Fri.-Fri. | On-line workshop: Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com |
June 20 - July 18, 2025, Fri.-Fri. | On-line workshop: Rasch Measurement - Further Topics (E. Smith, Facets), www.statistics.com |
Oct. 3 - Nov. 7, 2025, Fri.-Fri. | On-line workshop: Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com |
The URL of this page is www.rasch.org/rmt/rmt74f.htm
Website: www.rasch.org/rmt/contents.htm