Equitable Test Equating

Unfortunately, we cannot construct one licensing test to administer to all candidates forever. We must construct a new test every year. But how can we compare candidate performances on different tests or across years? How can we maintain pass-fail standards? We need a practical and defensible method of test equating to make fair comparisons of scores from one test to another. But most equating methods fail to solve the basic equating problems.

1) The Test Length Problem. When Al gets 9 right on test A, while Bob gets 15 right on test B, which score means more? If the tests have the same number of items, Bob's 15 looks better than Al's 9. But if Al's test has only 10 items, while Bob's has 20, then Al's 90% looks better than Bob's 75%. To solve this problem, we could insist that all tests have the same number of items or always use percentages.

2) The Test Difficulty Problem. What if test B is harder than test A? What if the easiest item on test B (where Bob got 75%) is harder than the hardest item on test A (where Al got 90%)? Then even one right answer on Bob's hard test could mean more than 95% on Al's easy test. Now Bob's 75% looks better than Al's 90%.

Does matching item success rates solve this problem? When an item is administered to a sample of candidates, some proportion succeed on the item. This proportion, the item "P-value", describes how easy the item was this time for this sample of candidates. When two items obtain the same P-value from the same sample at the same time, they have appeared equally difficult. We could construct equated tests from pairs of items that have the same P-value obtained from the same sample at the same time. Such perfect matching, however, is impractical and restrictive.

3) The Item Difficulty Distribution Problem. If we cannot match pairs of item P-values, why not just match average P-values of tests? Tests A and B are "equated" to have the same average P-value. Al gets 4 right on test A and Bob gets 4 right on test B. Have Al and Bob done equally well? If Al's test A has 8 items with P-values of .3, and 4 with P-values of .9, but Bob's test B has 12 items with P-values of .5, then both tests have an average P-value of .5. The only reasonable way Al scores 4 on test A, however, is by succeeding on the four easy items with P-values of .9. Bob's 4 successes must come from harder items with P-values of .5. Bob outperforms Al! Equating average item P-values does not equate tests.

4) The Sample Ability Problem. If we do not use the same sample to equate items, then the inevitably different sample ability distributions yield incomparable P-values. Differences in sample distributions, however slight, incapacitate all equating methods that rely on the fiction of identical sample distributions.

5) The Linear Scale Problem. To maintain standards from year to year, the scores from each year's examination must be equated in terms of the ability they represent. To measure growth or change from year to year, the results of each test must also be expressed on a shared scale that keeps the size of its unit the same from beginning to end. No raw score equating methods can provide the linear metric necessary for quantitative comparisons.

6) The Missing Data Problem. Most equating methods require complete data: every candidate must take every test item. This fiction, always compromised in paper-and-pencil testing, is completely insupportable in adaptive testing.

7) The Standard Error Problem. For realistic decision-making, not to mention statistical analysis of the results of testing, we need an estimate of the precision of each item difficulty calibration and of each candidate measure. Reliability coefficients, averaged over all candidates and all items, are deficient because they provide only one average value for the standard error of any score. The actual precision of any score varies widely around this average value and so this value is always incorrect for any particular score.

8) The Quality Control Problem. Are the items in the tests cooperating to measure the same variable in the same way? Are candidates performing in the same way not only across tests, but also across items within tests? The quantitative validity of each measure is the extent to which the pattern of right and wrong answers fits the pattern of item difficulties. Fit statistics reveal items that are not working as intended and candidates whose performance is anomalous. Anomalies can be investigated and prevented from spoiling the equating process. If no check for anomalies is made, however, we have no idea whether our scores are meaningful accumulations or meaningless collections.

Rasch's Equating Invention

In 1953 Danish mathematician Georg Rasch was asked to equate some reading tests. He devised a method for estimating item difficulties entirely free of the effects of the abilities of the candidates who happen to respond to the items. His method also estimates candidate measures entirely free of the effects of the difficulties of whichever items elicit their responses.

Rasch saw that, while he could not determine exactly how a candidate would respond to an item, it should be possible to estimate the candidate's probability of success on that item. He also saw that, for useful results, this probability of a right answer must not be governed by anything except the candidate's ability and the item's difficulty. He deduced this formula:

Rasch's formula is the only practical way to solve the eight equating problems. Data from different tests taken by different candidates can be combined and analyzed together, so long as there is some network of commonalities (candidates and/or items) linking the tests. This combined analysis provides a calibration, standard error and fit statistic for every item, and a measure, standard error and fit statistic for every candidate involved in any of the testings. These item calibrations and candidate measures are completely equated because they are all expressed at once on one common linear scale. Once a bank of items has been calibrated, inclusion of items from the bank into each new test automatically equates that test to the common metric of the bank, and so to all other tests derived from that bank.

An Example of Successful Equating

The University of Chicago Center for School Improvement equated 36 tests covering math from 1st to 8th grades. Each test was administered to a small but relevant fraction of the total sample of 1st to 8th grade pupils. Since the sequence of successively more difficult tests overlapped in the items they contained, Rasch analysis could equate the 36 tests by placing the difficulties of the 3,000 different items and the abilities of 12,000 different pupils on one common linear scale. The results of this one-step (concurrent) equating of the 36 math tests is now applied to 2 million test records collected over 8 years in order to make a quantitative study of individual growth in math from 1st to 8th grade.

Benjamin D. Wright

Equitable test equating. Wright BD. … 1993, 7:2 p.298-9


Equitable test equating. Wright BD. … Rasch Measurement Transactions, 1993, 1993, 7:2 p.298-9

Please help with Standard Dataset 4: Andrich Rating Scale Model



Rasch Publications
Rasch Measurement Transactions (free, online) Rasch Measurement research papers (free, online) Probabilistic Models for Some Intelligence and Attainment Tests, Georg Rasch Applying the Rasch Model 3rd. Ed., Bond & Fox Best Test Design, Wright & Stone
Rating Scale Analysis, Wright & Masters Introduction to Rasch Measurement, E. Smith & R. Smith Introduction to Many-Facet Rasch Measurement, Thomas Eckes Invariant Measurement: Using Rasch Models in the Social, Behavioral, and Health Sciences, George Engelhard, Jr. Statistical Analyses for Language Testers, Rita Green
Rasch Models: Foundations, Recent Developments, and Applications, Fischer & Molenaar Journal of Applied Measurement Rasch models for measurement, David Andrich Constructing Measures, Mark Wilson Rasch Analysis in the Human Sciences, Boone, Stave, Yale
in Spanish: Análisis de Rasch para todos, Agustín Tristán Mediciones, Posicionamientos y Diagnósticos Competitivos, Juan Ramón Oreja Rodríguez

To be emailed about new material on www.rasch.org
please enter your email address here:

I want to Subscribe: & click below
I want to Unsubscribe: & click below

Please set your SPAM filter to accept emails from Rasch.org

www.rasch.org welcomes your comments:

Your email address (if you want us to reply):

 

ForumRasch Measurement Forum to discuss any Rasch-related topic

Go to Top of Page
Go to index of all Rasch Measurement Transactions
AERA members: Join the Rasch Measurement SIG and receive the printed version of RMT
Some back issues of RMT are available as bound volumes
Subscribe to Journal of Applied Measurement

Go to Institute for Objective Measurement Home Page. The Rasch Measurement SIG (AERA) thanks the Institute for Objective Measurement for inviting the publication of Rasch Measurement Transactions on the Institute's website, www.rasch.org.

Coming Rasch-related Events
May 26 - June 23, 2017, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 30 - July 29, 2017, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
July 31 - Aug. 3, 2017, Mon.-Thurs. Joint IMEKO TC1-TC7-TC13 Symposium 2017: Measurement Science challenges in Natural and Social Sciences, Rio de Janeiro, Brazil, imeko-tc7-rio.org.br
Aug. 7-9, 2017, Mon-Wed. In-person workshop and research coloquium: Effect size of family and school indexes in writing competence using TERCE data (C. Pardo, A. Atorressi, Winsteps), Bariloche Argentina. Carlos Pardo, Universidad Catòlica de Colombia
Aug. 7-9, 2017, Mon-Wed. PROMS 2017: Pacific Rim Objective Measurement Symposium, Sabah, Borneo, Malaysia, proms.promsociety.org/2017/
Aug. 10, 2017, Thurs. In-person Winsteps Training Workshop (M. Linacre, Winsteps), Sydney, Australia. www.winsteps.com/sydneyws.htm
Aug. 11 - Sept. 8, 2017, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com
Aug. 18-21, 2017, Fri.-Mon. IACAT 2017: International Association for Computerized Adaptive Testing, Niigata, Japan, iacat.org
Sept. 15-16, 2017, Fri.-Sat. IOMC 2017: International Outcome Measurement Conference, Chicago, jampress.org/iomc2017.htm
Oct. 13 - Nov. 10, 2017, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
Jan. 5 - Feb. 2, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
Jan. 10-16, 2018, Wed.-Tues. In-person workshop: Advanced Course in Rasch Measurement Theory and the application of RUMM2030, Perth, Australia (D. Andrich), Announcement
Jan. 17-19, 2018, Wed.-Fri. Rasch Conference: Seventh International Conference on Probabilistic Models for Measurement, Matilda Bay Club, Perth, Australia, Website
April 13-17, 2018, Fri.-Tues. AERA, New York, NY, www.aera.net
May 25 - June 22, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 29 - July 27, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
Aug. 10 - Sept. 7, 2018, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com
Oct. 12 - Nov. 9, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
The HTML to add "Coming Rasch-related Events" to your webpage is:
<script type="text/javascript" src="http://www.rasch.org/events.txt"></script>

 

The URL of this page is www.rasch.org/rmt/rmt72q.htm

Website: www.rasch.org/rmt/contents.htm