How can we know when a single achievement test result is valid for a particular student? We can't. The universally-applicable uncertainty principle applies in several ways: we cannot determine the effect on the measured by the measuring instrumentation from a single test result. However, in some camps, there is a belief that it is possible to tell that a test produced an invalid measure of a student's performance by perusing the pattern of responses to test items. Is this an application of science or an attractive mythology? This is a serious question that has important implications for kids, teachers, and testing programs.
Is calculating "student fit" an example of detecting a difference? A discrepancy between what and what? Difference between students' item-by-item performance and the item-by-item relationships specified by Rasch mathematics? Student personalized test performance being different from behavior we expected to happen on a purely probabilistic basis? Difference between what a student did on a test and what that student should have done under more favorable sampling conditions?
It is always troubling when something done with numbers is equated to human behavior or misbehavior, if the connection between them is an explanation. Explanations never prove anything; they just make people say "Oh! I see." Since William Fisher has done such an artistic in-service on the contextual nature of seemingly objective observations of reality, such as response patterns, it might be appropriate to back off and consider the sources of the response patterns in tests and the way they are observed by psychometricians. For ground rules let's concern ourselves with the "what" and "how" questions of science and leave the "why" questions to philosophy.
Consider the mythology surrounding the traditional item discrimination index calculation and its use, without regard to the reality of curriculum pacing. What does this kind of "discrimination" really mean? Information about the wide or narrow range of the curriculum over which an item is supposed to reveal student knowledge is embodied in instructional programs. This range is only superficially, rather than causally, represented by high and low groups in some specified population.
Another myth is that some items should count more than others because they are mathematically related to high and low groups. These examples of overextended, tangential reasoning seem plausible enough to be swallowed by some statisticians.
In contrast, a different kind of background information comes from the practice, some years ago, of providing teachers with test information that incorporated information about other related characteristics of their students. The number of items students got correct and the number they attempted on a speeded verbal test were included in the teacher's report. To get a derived score, the number correct was divided by the number tried and this was then multiplied once more by the number correct. All scores were reported in standard notation based on the district population. Although it was not clear whether this "score" reflected study skills, a turtle-or-hare approach to problem solving, or was just equated for time constraints, it was clear that the teachers recognized and liked the way this information reflected their own assessment of the students approach to schooling. Some students worked carefully and got all of the few items they tried correct. Some raced ahead to increase the total number correct and take the penalty of more errors. Both behaviors reflected students' ways of working - their achievement mode. This derived "score" was more useful in reflecting ability to do school work on a day-to-day basis than any other we could derive from the data.
We know that test validity is specific to a student. We assume that validity differs across students taking a test. And, although we can't know their covert behaviors, we assume each student is unique in the combination of learning style, background experiences, cortical cycling speed, learned social responses, interest in different aspects of the curriculum, success or failure with schooling, and other personality characteristics, as well as in transitory attitudes toward school, testing, teachers, and others. Can't we, then, simply say that, whatever responses a student makes, they reflect ability to answer questions in this situation? Scores taken previously and in the future can be the data to use in deciding to delete a score as aberrant.
Consider also that validity is actually made up of two separate components; instrument validity is one and the student component, what we sometimes call reliability, is the other. Like heredity and environment, neither can exist without the other, but we can discuss their contributions individually. The personal part of test taking can only be guessed. Our perception of its manifestation is biased by many experiences and our learned response of searching for a statistic when all else fails. Why not leave reliability to whatever unknown factors might be exerting influence?
We know that learning for an individual should not be expected to progress in a linear manner or at a constant pace. We know that reinforcement of different learnings in the school situation varies and its effects on different individuals vary. And we know that teaching some lessons make earlier learnings evaporate. For example, students who formerly knew how to correctly add a number to zero will often incorrectly answer later that a number and zero equal zero after they have been taught that a number multiplied by zero equals zero.
Low raw scores are a different matter. They raise a different question - poor test construction. First, get rid of ambiguous items. Second, use a different test. When students get less than a third of the items correct on a test, it is due to faulty test assignment procedures and can be corrected by retesting with a lower level test. Inevitable idiosyncrasies in teaching and learning must never be penalized in a misguided effort to report higher test reliabilities.
Portland Public Schools
Student personality and test objectivity. Ingebo G. Rasch Measurement Transactions, 1990, 3:4 p.86
Please help with Standard Dataset 4: Andrich Rating Scale Model
|Rasch Measurement Transactions (free, online)||Rasch Measurement research papers (free, online)||Probabilistic Models for Some Intelligence and Attainment Tests, Georg Rasch||Applying the Rasch Model 3rd. Ed., Bond & Fox||Best Test Design, Wright & Stone|
|Rating Scale Analysis, Wright & Masters||Introduction to Rasch Measurement, E. Smith & R. Smith||Introduction to Many-Facet Rasch Measurement, Thomas Eckes||Invariant Measurement: Using Rasch Models in the Social, Behavioral, and Health Sciences, George Engelhard, Jr.||Statistical Analyses for Language Testers, Rita Green|
|Rasch Models: Foundations, Recent Developments, and Applications, Fischer & Molenaar||Journal of Applied Measurement||Rasch models for measurement, David Andrich||Constructing Measures, Mark Wilson||Rasch Analysis in the Human Sciences, Boone, Stave, Yale|
|in Spanish:||Análisis de Rasch para todos, Agustín Tristán||Mediciones, Posicionamientos y Diagnósticos Competitivos, Juan Ramón Oreja Rodríguez|
|Forum||Rasch Measurement Forum to discuss any Rasch-related topic|
Go to Top of Page
Go to index of all Rasch Measurement Transactions
AERA members: Join the Rasch Measurement SIG and receive the printed version of RMT
Some back issues of RMT are available as bound volumes
Subscribe to Journal of Applied Measurement
Go to Institute for Objective Measurement Home Page. The Rasch Measurement SIG (AERA) thanks the Institute for Objective Measurement for inviting the publication of Rasch Measurement Transactions on the Institute's website, www.rasch.org.
|Coming Rasch-related Events|
|Sept. 15-16, 2017, Fri.-Sat.||IOMC 2017: International Outcome Measurement Conference, Chicago, jampress.org/iomc2017.htm|
|Oct. 13 - Nov. 10, 2017, Fri.-Fri.||On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com|
|Oct. 25-27, 2017, Wed.-Fri.||In-person workshop: Applying the Rasch Model hands-on introductory workshop, Melbourne, Australia (T. Bond, B&FSteps), Announcement|
|Jan. 5 - Feb. 2, 2018, Fri.-Fri.||On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com|
|Jan. 10-16, 2018, Wed.-Tues.||In-person workshop: Advanced Course in Rasch Measurement Theory and the application of RUMM2030, Perth, Australia (D. Andrich), Announcement|
|Jan. 17-19, 2018, Wed.-Fri.||Rasch Conference: Seventh International Conference on Probabilistic Models for Measurement, Matilda Bay Club, Perth, Australia, Website|
|April 13-17, 2018, Fri.-Tues.||AERA, New York, NY, www.aera.net|
|May 25 - June 22, 2018, Fri.-Fri.||On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com|
|June 29 - July 27, 2018, Fri.-Fri.||On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com|
|Aug. 10 - Sept. 7, 2018, Fri.-Fri.||On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com|
|Oct. 12 - Nov. 9, 2018, Fri.-Fri.||On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com|
|The HTML to add "Coming Rasch-related Events" to your webpage is:|
The URL of this page is www.rasch.org/rmt/rmt34f.htm