Go to Top of Page
"If items do not fit the Rasch model, the recommendation is to drop them. But Samuel Messick (1995, Validity of psychological assessment: validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9); 741-749) argues that `it is not sufficient merely to select tasks that are relevant to the construct domain. In addition, the assessment should assemble tasks that are representative of the domain... The intent is to insure that all important parts of the construct domain are covered...' (p. 745)."
"Assuming that the original pool of items is selected to fully
represent the content of interest, how does one insure that content
validity is maintained throughout the process of test development
when dropping items simply because they do not fit the Rasch
model?"
Michael Bohlig
"If I can be allowed a slight over-generalization, no one has ever seriously suggested that any aspect of a domain be omitted from an instrument simply because the item(s) representing it produce inconsistent observations. Items and aspects of a domain are not the same thing. There are many reasons why items don't work (data entry errors; consistent differences in response propensities introduced by respondents' personalities, gender, age, or culture; rater or instrument administration errors; ambiguous item phrasings; etc.). Furthermore, whether or not any item belongs to one particular construct or to another is not a question that can be answered on a purely theoretical basis, without reference to data. Often misfitting items belong to a separate construct that needs to be measured separately, with a different instrument."
"In an earlier work, Messick (1975, The standard problem: meaning
and values in measurement and evaluation, ibid. 30; 955-966) says
not only that `all measurement should be construct-referenced' (p.
957), but that `any concept of validity of measurement must include
reference to empirical consistency' (p. 960). He should then hold
that construct validity depends on empirical consistency, exactly
the matter of interest in the application of the Rasch model."
William P. Fisher, Jr., wfishe@lsumc.edu
"Do test constructors drop items simply because they do not fit the Rasch model, and without considering the extent to which the remaining items represent the domain as originally conceptualized and intended? I've worked with the Rasch model for more that 20 years, and I don't know anybody who does this. There can't be many people who are in the luxurious position of having so many items that they can discard them on statistical grounds alone."
"The late Bruce Choppin (RMT 8:4 p. 394) told me that he never deleted an item on statistical grounds unless he could see a problem with it on substantive grounds, and this is the practice that I've tried to follow. In my experience, serious item misfit can almost always be understood and usually indicates an unanticipated problem. Mostly it's a problem with the quality of the item. Users of the Rasch model are sometimes accused of letting the model determine and define domains for them. When I used Rasch to construct the 14 TORCH tests of reading comprehension in the early '80s, I was told that I allowed Rasch to distort my definition of reading comprehension. But the truth is that the items we identified and deleted from those tests using Rasch were simply bad items that we could have identified in any number of other ways."
"But I sometimes find that the problem is with my conceptualization of the construct. Item misfit can be an indication that performances in the domain, as I originally conceptualized it, cannot be summarized in a single number. Under these circumstances, it's rarely an option to simply discard misfitting items. As Messick says, usually all important parts of a domain must be assessed. But this is not to say that performances in all parts of a domain must be summarized in a single number."
"In my experience, item misfit is sometimes an indication that
performances in an area originally conceptualized as one domain
must be reported on more than one dimension. In Australia, 'oral
language proficiency' is sometimes represented as one domain, but
our analyses show that speaking and listening skills function as
two different dimensions of language proficiency."
Geoff Masters, ACER
"In summary: let us presume that the items for a potential test
were all included for some very good (substantive, theoretical,
construct-related) reasons. Then, less than pleasing fit
statistics say `Think again', not `Throw it out'."
Trevor Bond, Trevor.Bond@jcu.edu.au
Bohlig M., Fisher W.P. Jr., Masters, G.N., Bond, T. (1998) Content Validity and Misfitting Items. Rasch Measurement Transactions 12:1 p. 607.
Content Validity and Misfitting Items. Bohlig M., Fisher W.P. Jr., Masters, G.N., Bond, T. Rasch Measurement Transactions, 1998, 12:1 p. 607.
Rasch Publications | ||||
---|---|---|---|---|
Rasch Measurement Transactions (free, online) | Rasch Measurement research papers (free, online) | Probabilistic Models for Some Intelligence and Attainment Tests, Georg Rasch | Applying the Rasch Model 3rd. Ed., Bond & Fox | Best Test Design, Wright & Stone |
Rating Scale Analysis, Wright & Masters | Introduction to Rasch Measurement, E. Smith & R. Smith | Introduction to Many-Facet Rasch Measurement, Thomas Eckes | Invariant Measurement: Using Rasch Models in the Social, Behavioral, and Health Sciences, George Engelhard, Jr. | Statistical Analyses for Language Testers, Rita Green |
Rasch Models: Foundations, Recent Developments, and Applications, Fischer & Molenaar | Journal of Applied Measurement | Rasch models for measurement, David Andrich | Constructing Measures, Mark Wilson | Rasch Analysis in the Human Sciences, Boone, Stave, Yale |
in Spanish: | Análisis de Rasch para todos, Agustín Tristán | Mediciones, Posicionamientos y Diagnósticos Competitivos, Juan Ramón Oreja Rodríguez |
Forum | Rasch Measurement Forum to discuss any Rasch-related topic |
Go to Top of Page
Go to index of all Rasch Measurement Transactions
AERA members: Join the Rasch Measurement SIG and receive the printed version of RMT
Some back issues of RMT are available as bound volumes
Subscribe to Journal of Applied Measurement
Go to Institute for Objective Measurement Home Page. The Rasch Measurement SIG (AERA) thanks the Institute for Objective Measurement for inviting the publication of Rasch Measurement Transactions on the Institute's website, www.rasch.org.
Coming Rasch-related Events | |
---|---|
Oct. 6 - Nov. 3, 2023, Fri.-Fri. | On-line workshop: Rasch Measurement - Core Topics (E. Smith, Facets), www.statistics.com |
Oct. 12, 2023, Thursday 5 to 7 pm Colombian time | On-line workshop: Deconstruyendo el concepto de validez y Discusiones sobre estimaciones de confiabilidad SICAPSI (J. Escobar, C.Pardo) www.colpsic.org.co |
June 12 - 14, 2024, Wed.-Fri. | 1st Scandinavian Applied Measurement Conference, Kristianstad University, Kristianstad, Sweden http://www.hkr.se/samc2024 |
Aug. 9 - Sept. 6, 2024, Fri.-Fri. | On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com |
The URL of this page is www.rasch.org/rmt/rmt121f.htm
Website: www.rasch.org/rmt/contents.htm