Measurement is the process of converting observations (e.g., counts) into measures (quantities) via a construct theory. The Rasch Model states a requirement for the way observations and construct theory combine in a probability model to make measures. There is no other combination of observation and theory that produces sufficiency invariance and objectivity in the resultant measures.
As scientists, we cherish parsimony (do much with little) and as statisticians, we cherish uniqueness. This is why we speak of data that fit the Rasch Model - not of fitting some model, Rasch or otherwise, to data. The Rasch Model combines the three components in the definition of measurement (observation, theory and measure) into a simple, elegant and, in more important respects, unique representation.
It is unfortunately too easy to apply the Rasch Model without a construct theory. Rasch programs enable practitioners to estimate item difficulties and person measures on a logit or transformed logit scale. Item maps can be built and variables named, all without any a priori of our intention. In my own case, this practice reminds me of how quickly I could name a factor structure coming out of an "exploratory" factor analysis. Never once did I specify, prior to data collection, what I expected to find in the data. To compound the heresy, I wrote a suite of Educational and Psychological Measurement articles on factor replicability and factor invariance.
If we are to have human science(s), we must know enough about the constructs we intend to measure to be able to specify item calibrations prior to data collection. These item calibrations come from a construct theory. Theory and item engineering improve as we bring observed item difficulties and theory-based item calibrations into closer and closer coincidence. The just announced "temporal thermometers" are a recent example of this kind of improvement in the measurement of human temperature.
"Exploratory" Rasch analysis is no substitute, because, without a construct theory and associated specification equation, the data does not "bite". Just like in exploratory factor analysis, exploratory Rasch analysis produces too many successful studies. We need to increase our failure rate and the shortest route is to make our intentions explicit and then see if the data confirm to the theoretical expectations.
What price do we pay for extensive reliance on exploratory Rasch analysis? We make it almost impossible to demonstrate how a common theory and specification equation can unify the diverse instruments purporting to measure a single construct.
Two hundred and fifty instruments (tests) for measuring "reading comprehension" each with proprietary
non-exchangeable scales can be unified only if there is
(1) a compelling demonstration that most of these instruments measure a single unidimensional construct called
"reading comprehension",
(2) a common supplemental metric is proposed that is based on a construct theory and supported by a specification
equation capable of explaining high proportions of item difficulty variance within and across the candidate tests,
(3) there is some compelling advantage beyond parsimony to encourage the "community" to adopt the standard (the
possibility of linking test scores to books is such an application for reading comprehension), and
(4) a business model is proposed that makes it advantageous for the power elite (e.g., publishers, academicians, key
users, policy makers) to support unification.
Adoption of the metric system in the US in the 1970s failed criteria 3 and 4 and, ergo, no metric system in our everyday lives despite a huge expenditure.
There is a simple thought experiment that can inform us regarding how well we understand the construct under study. If presented with an instrument purportedly measuring the construct, can we use our knowledge about the construct-associated (construct theory) specification/calibration equation(s) together with item engineering rules to produce a clone or copy of the instrument - such that the score-to-measure table for the clone is identical to that of the original instrument?
Moving now from theory to experiment, if the clone produces measures that are statistically equivalent to those produced by the original, then we have demonstrated that we understand what we are measuring. We can reproduce the phenomena experimentally and we are in control of the major contaminants.
The first time we attempted the above protocol with "reading comprehension", we failed. After some four years and 30 attempts, we are getting pretty good at building reading comprehension tests based on theory alone that produce raw score-to-measure correspondences that conform to design requirements. With this kind of control, it is practically feasible to contemplate unifying the measurement of reading.
Finally, William P. Fisher, Jr. states: "I think that the need for these metrological networks will eventually be seen simply because of the way independent experiments addressing the same variable are converging on common constructs!" I want this to happen because human science is impossible without it. But, the unification process that precedes consensus on a unit of measurement for a construct will be built on improved construct theories and resultant improved instrumentation, not more exploratory Rasch analysis.
Exploratory studies don't converge on anything. As always, the great unifier is substantive theory. See www.lexile.com for a report card on the unification of the measurement of reading comprehension in English and Spanish.
Jack Stenner
Trevor Bond & Christine Fox (2001, 2001, pp.xxi-xxii) comment:
"The Rasch approach to investigating measures illustrates an appreciation of the dialectical nature of the
measurement process: Theory informs practice, and practice informs theory. ... From our viewpoint, every item you
write puts either the theory, your understanding of it, or your ability to translate it into an empirical indicator on the
line. Tossing out items might be fine if you have trawled up a huge bunch of possible items and are looking for the
probables to keep in your test. However, if you have developed items that encapsulate for you the very essence of
the theory or construct to which you are attached, then all the items are probables. In fact, you might consider all of
them as somehow indispensable! Thus, whereas your theory tells you how to write the item, the item's performance
should tell you about your theory, your understanding of it, or your item-writing skills."
The Necessity of Construct Theory. Stenner J. … Rasch Measurement Transactions, 2001, 15:1 p.804-5
Forum | Rasch Measurement Forum to discuss any Rasch-related topic |
Go to Top of Page
Go to index of all Rasch Measurement Transactions
AERA members: Join the Rasch Measurement SIG and receive the printed version of RMT
Some back issues of RMT are available as bound volumes
Subscribe to Journal of Applied Measurement
Go to Institute for Objective Measurement Home Page. The Rasch Measurement SIG (AERA) thanks the Institute for Objective Measurement for inviting the publication of Rasch Measurement Transactions on the Institute's website, www.rasch.org.
Coming Rasch-related Events | |
---|---|
Oct. 4 - Nov. 8, 2024, Fri.-Fri. | On-line workshop: Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com |
Jan. 17 - Feb. 21, 2025, Fri.-Fri. | On-line workshop: Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com |
May 16 - June 20, 2025, Fri.-Fri. | On-line workshop: Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com |
June 20 - July 18, 2025, Fri.-Fri. | On-line workshop: Rasch Measurement - Further Topics (E. Smith, Facets), www.statistics.com |
Oct. 3 - Nov. 7, 2025, Fri.-Fri. | On-line workshop: Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com |
The URL of this page is www.rasch.org/rmt/rmt151q.htm
Website: www.rasch.org/rmt/contents.htm