Go to Top of Page

Content Validity, Construct Validity and Misfitting Items

"If items do not fit the Rasch model, the recommendation is to drop them. But Samuel Messick (1995, Validity of psychological assessment: validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9); 741-749) argues that `it is not sufficient merely to select tasks that are relevant to the construct domain. In addition, the assessment should assemble tasks that are representative of the domain... The intent is to insure that all important parts of the construct domain are covered...' (p. 745)."

"Assuming that the original pool of items is selected to fully represent the content of interest, how does one insure that content validity is maintained throughout the process of test development when dropping items simply because they do not fit the Rasch model?"
Michael Bohlig

"If I can be allowed a slight over-generalization, no one has ever seriously suggested that any aspect of a domain be omitted from an instrument simply because the item(s) representing it produce inconsistent observations. Items and aspects of a domain are not the same thing. There are many reasons why items don't work (data entry errors; consistent differences in response propensities introduced by respondents' personalities, gender, age, or culture; rater or instrument administration errors; ambiguous item phrasings; etc.). Furthermore, whether or not any item belongs to one particular construct or to another is not a question that can be answered on a purely theoretical basis, without reference to data. Often misfitting items belong to a separate construct that needs to be measured separately, with a different instrument."

"In an earlier work, Messick (1975, The standard problem: meaning and values in measurement and evaluation, ibid. 30; 955-966) says not only that `all measurement should be construct-referenced' (p. 957), but that `any concept of validity of measurement must include reference to empirical consistency' (p. 960). He should then hold that construct validity depends on empirical consistency, exactly the matter of interest in the application of the Rasch model."
William P. Fisher, Jr., wfishe@lsumc.edu

"Do test constructors drop items simply because they do not fit the Rasch model, and without considering the extent to which the remaining items represent the domain as originally conceptualized and intended? I've worked with the Rasch model for more that 20 years, and I don't know anybody who does this. There can't be many people who are in the luxurious position of having so many items that they can discard them on statistical grounds alone."

"The late Bruce Choppin (RMT 8:4 p. 394) told me that he never deleted an item on statistical grounds unless he could see a problem with it on substantive grounds, and this is the practice that I've tried to follow. In my experience, serious item misfit can almost always be understood and usually indicates an unanticipated problem. Mostly it's a problem with the quality of the item. Users of the Rasch model are sometimes accused of letting the model determine and define domains for them. When I used Rasch to construct the 14 TORCH tests of reading comprehension in the early '80s, I was told that I allowed Rasch to distort my definition of reading comprehension. But the truth is that the items we identified and deleted from those tests using Rasch were simply bad items that we could have identified in any number of other ways."

"But I sometimes find that the problem is with my conceptualization of the construct. Item misfit can be an indication that performances in the domain, as I originally conceptualized it, cannot be summarized in a single number. Under these circumstances, it's rarely an option to simply discard misfitting items. As Messick says, usually all important parts of a domain must be assessed. But this is not to say that performances in all parts of a domain must be summarized in a single number."

"In my experience, item misfit is sometimes an indication that performances in an area originally conceptualized as one domain must be reported on more than one dimension. In Australia, 'oral language proficiency' is sometimes represented as one domain, but our analyses show that speaking and listening skills function as two different dimensions of language proficiency."
Geoff Masters, ACER

"In summary: let us presume that the items for a potential test were all included for some very good (substantive, theoretical, construct-related) reasons. Then, less than pleasing fit statistics say `Think again', not `Throw it out'."
Trevor Bond, Trevor.Bond@jcu.edu.au

Bohlig M., Fisher W.P. Jr., Masters, G.N., Bond, T. (1998) Content Validity and Misfitting Items. Rasch Measurement Transactions 12:1 p. 607.

Content Validity and Misfitting Items. Bohlig M., Fisher W.P. Jr., Masters, G.N., Bond, T. … Rasch Measurement Transactions, 1998, 12:1 p. 607.

Rasch Publications
Rasch Measurement Transactions (free, online)	Rasch Measurement research papers (free, online)	Probabilistic Models for Some Intelligence and Attainment Tests, Georg Rasch	Applying the Rasch Model 3rd. Ed., Bond & Fox	Best Test Design, Wright & Stone
Rating Scale Analysis, Wright & Masters	Introduction to Rasch Measurement, E. Smith & R. Smith	Introduction to Many-Facet Rasch Measurement, Thomas Eckes	Invariant Measurement: Using Rasch Models in the Social, Behavioral, and Health Sciences, George Engelhard, Jr.	Statistical Analyses for Language Testers, Rita Green
Rasch Models: Foundations, Recent Developments, and Applications, Fischer & Molenaar	Journal of Applied Measurement	Rasch models for measurement, David Andrich	Constructing Measures, Mark Wilson	Rasch Analysis in the Human Sciences, Boone, Stave, Yale
in Spanish:	Análisis de Rasch para todos, Agustín Tristán		Mediciones, Posicionamientos y Diagnósticos Competitivos, Juan Ramón Oreja Rodríguez

Forum

Rasch Measurement Forum to discuss any Rasch-related topic

Go to Top of Page
Go to index of all Rasch Measurement Transactions
AERA members: Join the Rasch Measurement SIG and receive the printed version of RMT
Some back issues of RMT are available as bound volumes
Subscribe to Journal of Applied Measurement

Go to Institute for Objective Measurement Home Page. The Rasch Measurement SIG (AERA) thanks the Institute for Objective Measurement for inviting the publication of Rasch Measurement Transactions on the Institute's website, www.rasch.org.

The URL of this page is www.rasch.org/rmt/rmt121f.htm

Website: www.rasch.org/rmt/contents.htm