Go to Top of Page

Content Validity, Construct Validity and Misfitting Items

"If items do not fit the Rasch model, the recommendation is to drop them. But Samuel Messick (1995, Validity of psychological assessment: validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9); 741-749) argues that `it is not sufficient merely to select tasks that are relevant to the construct domain. In addition, the assessment should assemble tasks that are representative of the domain... The intent is to insure that all important parts of the construct domain are covered...' (p. 745)."

"Assuming that the original pool of items is selected to fully represent the content of interest, how does one insure that content validity is maintained throughout the process of test development when dropping items simply because they do not fit the Rasch model?"
Michael Bohlig

"If I can be allowed a slight over-generalization, no one has ever seriously suggested that any aspect of a domain be omitted from an instrument simply because the item(s) representing it produce inconsistent observations. Items and aspects of a domain are not the same thing. There are many reasons why items don't work (data entry errors; consistent differences in response propensities introduced by respondents' personalities, gender, age, or culture; rater or instrument administration errors; ambiguous item phrasings; etc.). Furthermore, whether or not any item belongs to one particular construct or to another is not a question that can be answered on a purely theoretical basis, without reference to data. Often misfitting items belong to a separate construct that needs to be measured separately, with a different instrument."

"In an earlier work, Messick (1975, The standard problem: meaning and values in measurement and evaluation, ibid. 30; 955-966) says not only that `all measurement should be construct-referenced' (p. 957), but that `any concept of validity of measurement must include reference to empirical consistency' (p. 960). He should then hold that construct validity depends on empirical consistency, exactly the matter of interest in the application of the Rasch model."
William P. Fisher, Jr., wfishe@lsumc.edu

"Do test constructors drop items simply because they do not fit the Rasch model, and without considering the extent to which the remaining items represent the domain as originally conceptualized and intended? I've worked with the Rasch model for more that 20 years, and I don't know anybody who does this. There can't be many people who are in the luxurious position of having so many items that they can discard them on statistical grounds alone."

"The late Bruce Choppin (RMT 8:4 p. 394) told me that he never deleted an item on statistical grounds unless he could see a problem with it on substantive grounds, and this is the practice that I've tried to follow. In my experience, serious item misfit can almost always be understood and usually indicates an unanticipated problem. Mostly it's a problem with the quality of the item. Users of the Rasch model are sometimes accused of letting the model determine and define domains for them. When I used Rasch to construct the 14 TORCH tests of reading comprehension in the early '80s, I was told that I allowed Rasch to distort my definition of reading comprehension. But the truth is that the items we identified and deleted from those tests using Rasch were simply bad items that we could have identified in any number of other ways."

"But I sometimes find that the problem is with my conceptualization of the construct. Item misfit can be an indication that performances in the domain, as I originally conceptualized it, cannot be summarized in a single number. Under these circumstances, it's rarely an option to simply discard misfitting items. As Messick says, usually all important parts of a domain must be assessed. But this is not to say that performances in all parts of a domain must be summarized in a single number."

"In my experience, item misfit is sometimes an indication that performances in an area originally conceptualized as one domain must be reported on more than one dimension. In Australia, 'oral language proficiency' is sometimes represented as one domain, but our analyses show that speaking and listening skills function as two different dimensions of language proficiency."
Geoff Masters, ACER

"In summary: let us presume that the items for a potential test were all included for some very good (substantive, theoretical, construct-related) reasons. Then, less than pleasing fit statistics say `Think again', not `Throw it out'."
Trevor Bond, Trevor.Bond@jcu.edu.au

Bohlig M., Fisher W.P. Jr., Masters, G.N., Bond, T. (1998) Content Validity and Misfitting Items. Rasch Measurement Transactions 12:1 p. 607.

Content Validity and Misfitting Items. Bohlig M., Fisher W.P. Jr., Masters, G.N., Bond, T. … Rasch Measurement Transactions, 1998, 12:1 p. 607.




Rasch Publications
Rasch Measurement Transactions (free, online) Rasch Measurement research papers (free, online) Probabilistic Models for Some Intelligence and Attainment Tests, Georg Rasch Applying the Rasch Model 3rd. Ed., Bond & Fox Best Test Design, Wright & Stone
Rating Scale Analysis, Wright & Masters Introduction to Rasch Measurement, E. Smith & R. Smith Introduction to Many-Facet Rasch Measurement, Thomas Eckes Invariant Measurement: Using Rasch Models in the Social, Behavioral, and Health Sciences, George Engelhard, Jr. Statistical Analyses for Language Testers, Rita Green
Rasch Models: Foundations, Recent Developments, and Applications, Fischer & Molenaar Journal of Applied Measurement Rasch models for measurement, David Andrich Constructing Measures, Mark Wilson Rasch Analysis in the Human Sciences, Boone, Stave, Yale
in Spanish: Análisis de Rasch para todos, Agustín Tristán Mediciones, Posicionamientos y Diagnósticos Competitivos, Juan Ramón Oreja Rodríguez

To be emailed about new material on www.rasch.org
please enter your email address here:

I want to Subscribe: & click below
I want to Unsubscribe: & click below

Please set your SPAM filter to accept emails from Rasch.org

www.rasch.org welcomes your comments:

Your email address (if you want us to reply):

 

ForumRasch Measurement Forum to discuss any Rasch-related topic

Go to Top of Page
Go to index of all Rasch Measurement Transactions
AERA members: Join the Rasch Measurement SIG and receive the printed version of RMT
Some back issues of RMT are available as bound volumes
Subscribe to Journal of Applied Measurement

Go to Institute for Objective Measurement Home Page. The Rasch Measurement SIG (AERA) thanks the Institute for Objective Measurement for inviting the publication of Rasch Measurement Transactions on the Institute's website, www.rasch.org.

Coming Rasch-related Events
May 17 - June 21, 2024, Fri.-Fri. On-line workshop: Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 12 - 14, 2024, Wed.-Fri. 1st Scandinavian Applied Measurement Conference, Kristianstad University, Kristianstad, Sweden http://www.hkr.se/samc2024
June 21 - July 19, 2024, Fri.-Fri. On-line workshop: Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
Aug. 5 - Aug. 6, 2024, Fri.-Fri. 2024 Inaugural Conference of the Society for the Study of Measurement (Berkeley, CA), Call for Proposals
Aug. 9 - Sept. 6, 2024, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com
Oct. 4 - Nov. 8, 2024, Fri.-Fri. On-line workshop: Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
Jan. 17 - Feb. 21, 2025, Fri.-Fri. On-line workshop: Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
May 16 - June 20, 2025, Fri.-Fri. On-line workshop: Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 20 - July 18, 2025, Fri.-Fri. On-line workshop: Rasch Measurement - Further Topics (E. Smith, Facets), www.statistics.com
Oct. 3 - Nov. 7, 2025, Fri.-Fri. On-line workshop: Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com

 

The URL of this page is www.rasch.org/rmt/rmt121f.htm

Website: www.rasch.org/rmt/contents.htm