Does Test-Item Interaction Invalidate one-step (concurrent) Concurrent Equating and Item Banking?

The following letter was sent from George Ingebo to Ben Wright

Dear Ben:
I had a chance to read the statements by Matthew Schulz in the SIG newsletter (RMT 1:2, p. 10-11) describing what he called one-step (concurrent) item banking, also known as concurrent equating, and I'm a little worried. I need your thinking, as a scientist, to help me with an important operating principle to rely on in proceeding with research on Rasch technology. Your opinion has been valuable on two other similar situations in the past.

It seems to me, although I'm no physicist, that the situation we face in achievement test data is similar to that described by Heisenberg's Uncertainty Principle with each single data point existing as such only after it is observed.

Before we get to the process for linking groups, let's go over what happens when we define a scale using one group. We can understand that as a student perceives a test item the information that he recalls because it is associated with the item is a part of the apperceptive mass of the student and is processed in a way learned by the student for an item of that kind. When "readiness", both in associative information and information processing, exists to some extent the observation establishes a bit of data useful for defining a scale and for measuring a student. As long as we stay in the area of basic skills, it seems that the latitude in both knowledge and process is wide for practical item calibrating purposes. But it can be abused.

Calibrating items for use in an item bank is quite different from scaling an established test for a defined population, such as the SATs and other published tests for particular grades, because the purposes are different. The item calibration irrespective of any population becomes the focus in item banking. In pinpointing calibration values each instance of non-readiness for an item on the part of a student becomes noise in the information stream. Unfortunately, the computer does not recognize counterproductive data, it always forces a set of numbers regardless of data quality.

Now to my problem, Ben. Just as calibrations generated in a group are influenced by responses from students for whom some of the questions are inappropriate, it has been established repeatedly that links between groups are seriously influenced by instances where an item operates quite differently in one test than it does in the other. Like working with students who exhibit individual differences in so many ways that we cannot consider replication at the individual level, we also find that many differences exist between groups in instructional backgrounds, in chance discussion the previous day, in test administration atmosphere or logistics and etc. For such reasons acceptable items can work well in two different tests as determined by the item characteristic curves and yet be counterproductive in linking because they provide information differently in the two tests -- item interaction is only one plausible explanation.

Again, linking for a bank is different from linking two established tests to establish a scale for some special purpose even when the tests are appropriate for the students taking them. Bank calibrations need to be all purpose values, precisely and uniformly linked in a relationship that holds with other similar calibrations and with other item calibrations several logits away. As you once told us, one item is enough to establish an accurate link if it is the right item. Detecting "right" items, then, becomes the critical factor in the linking equation. Those items far afield in the linking pattern could be the "right" ones, but are most likely adding more noise to the information stream.

Now it seems to me that Matthew Schulz's one-step (concurrent) banking program MFORMS, for all its elegance and efficiency, accepts data uncritically. If calibrating and linking were situations calling for central tendencies of all instances of reality there would be no objection, but in the effort to develop a scale against which these instances of reality are ultimately to be judged it seems that detecting and deleting instances of misapplication of data in both calibrating and linking processes must be avoided to achieve valid linking.

It is likely that I have overlooked something important in this line of thinking, Ben, and if so you will be able to point out where. Otherwise, since we both dislike data free argument, you would be the one to help set up an experiment that would determine quantitatively what, if any, difference results from using MFORMS or the technology that discards questionable data. Two experiments could be made with existing data; running MFORMS with raw data from a testing session in Portland and by using the predetermined calibrations, without setting grade differences, for another MFORMS run. However, since the main attribute of the Rasch model is that it provides symmetry to support item response theory, the crucial experiment has to test the measurement logit - calibration logit relationship, their interchangeability in the technology creating them. This experimenting can only be done in the basic skills, I think, but the design used in Portland's Area III in 1979 to test the flexibility of the item bank and the expectations of the Levels Tests seem both necessary and sufficient to establish the capability of a methodology to produce theoretically specified results.

The sour note in the symphony is logistical. Can you lead the charge to get an MFORMS constructed bank and levels tests developed from it introduced into a large district's testing program? The SIG group might be the place to plant the idea and you are the one to encourage us.

As Always,
George Ingebo

Does test-item interaction invalidate one-step (concurrent) item banking? Ingebo G. … Rasch Measurement Transactions 2:2 p.17-18


Does test-item interaction invalidate one-step (concurrent) item banking? Ingebo G. … Rasch Measurement Transactions, 1988, 2:2 p.17-18

Please help with Standard Dataset 4: Andrich Rating Scale Model



Rasch Publications
Rasch Measurement Transactions (free, online) Rasch Measurement research papers (free, online) Probabilistic Models for Some Intelligence and Attainment Tests, Georg Rasch Applying the Rasch Model 3rd. Ed., Bond & Fox Best Test Design, Wright & Stone
Rating Scale Analysis, Wright & Masters Introduction to Rasch Measurement, E. Smith & R. Smith Introduction to Many-Facet Rasch Measurement, Thomas Eckes Invariant Measurement: Using Rasch Models in the Social, Behavioral, and Health Sciences, George Engelhard, Jr. Statistical Analyses for Language Testers, Rita Green
Rasch Models: Foundations, Recent Developments, and Applications, Fischer & Molenaar Journal of Applied Measurement Rasch models for measurement, David Andrich Constructing Measures, Mark Wilson Rasch Analysis in the Human Sciences, Boone, Stave, Yale
in Spanish: Análisis de Rasch para todos, Agustín Tristán Mediciones, Posicionamientos y Diagnósticos Competitivos, Juan Ramón Oreja Rodríguez

To be emailed about new material on www.rasch.org
please enter your email address here:

I want to Subscribe: & click below
I want to Unsubscribe: & click below

Please set your SPAM filter to accept emails from Rasch.org

www.rasch.org welcomes your comments:

Your email address (if you want us to reply):

 

ForumRasch Measurement Forum to discuss any Rasch-related topic

Go to Top of Page
Go to index of all Rasch Measurement Transactions
AERA members: Join the Rasch Measurement SIG and receive the printed version of RMT
Some back issues of RMT are available as bound volumes
Subscribe to Journal of Applied Measurement

Go to Institute for Objective Measurement Home Page. The Rasch Measurement SIG (AERA) thanks the Institute for Objective Measurement for inviting the publication of Rasch Measurement Transactions on the Institute's website, www.rasch.org.

Coming Rasch-related Events
March 31, 2017, Fri. Conference: 11th UK Rasch Day, Warwick, UK, www.rasch.org.uk
April 2-3, 2017, Sun.-Mon. Conference: Validity Evidence for Measurement in Mathematics Education (V-M2Ed), San Antonio, TX, Information
April 26-30, 2017, Wed.-Sun. NCME, San Antonio, TX, www.ncme.org - April 29: Ben Wright book
April 27 - May 1, 2017, Thur.-Mon. AERA, San Antonio, TX, www.aera.net
May 26 - June 23, 2017, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 30 - July 29, 2017, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
July 31 - Aug. 3, 2017, Mon.-Thurs. Joint IMEKO TC1-TC7-TC13 Symposium 2017: Measurement Science challenges in Natural and Social Sciences, Rio de Janeiro, Brazil, imeko-tc7-rio.org.br
Aug. 7-9, 2017, Mon-Wed. In-person workshop and research coloquium: Effect size of family and school indexes in writing competence using TERCE data (C. Pardo, A. Atorressi, Winsteps), Bariloche Argentina. Carlos Pardo, Universidad Catòlica de Colombia
Aug. 7-9, 2017, Mon-Wed. PROMS 2017: Pacific Rim Objective Measurement Symposium, Sabah, Borneo, Malaysia, proms.promsociety.org/2017/
Aug. 10, 2017, Thurs. In-person Winsteps Training Workshop (M. Linacre, Winsteps), Sydney, Australia. www.winsteps.com/sydneyws.htm
Aug. 11 - Sept. 8, 2017, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com
Aug. 18-21, 2017, Fri.-Mon. IACAT 2017: International Association for Computerized Adaptive Testing, Niigata, Japan, iacat.org
Sept. 15-16, 2017, Fri.-Sat. IOMC 2017: International Outcome Measurement Conference, Chicago, jampress.org/iomc2017.htm
Oct. 13 - Nov. 10, 2017, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
Jan. 5 - Feb. 2, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
Jan. 10-16, 2018, Wed.-Tues. In-person workshop: Advanced Course in Rasch Measurement Theory and the application of RUMM2030, Perth, Australia (D. Andrich), Announcement
Jan. 17-19, 2018, Wed.-Fri. Rasch Conference: Seventh International Conference on Probabilistic Models for Measurement, Matilda Bay Club, Perth, Australia, Website
May 25 - June 22, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 29 - July 27, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
Aug. 10 - Sept. 7, 2018, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com
Oct. 12 - Nov. 9, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
The HTML to add "Coming Rasch-related Events" to your webpage is:
<script type="text/javascript" src="http://www.rasch.org/events.txt"></script>

 

The URL of this page is www.rasch.org/rmt/rmt22a.htm

Website: www.rasch.org/rmt/contents.htm