The purpose of this essay is to revisit Chapter 3 of Probabilistic Models (Rasch, 1960) and to consider to what extent the arguments from this chapter might be applied to individual items in a computer based math test. The intention is to look for inspiration from Rasch, but not to follow his methodology exactly.
Hippisley (1999) showed that total completion times from a computer based math test conformed to the (Rasch 1960) reading rate model. However, the assumption of item homogeneity and uniform student speed, rather limits the usefulness of that analysis. In reality, test items are not homogenous, and students do not course through a test at a uniform speed.
Theory
Rasch considered the "speededness" of students reading a text in two ways. For those students who did not complete the text within a prescribed time limit, he treated the number of words read within the time as the stochastic variable. For the students who did complete the test (within the time limit), he treated the time actually taken as the stochastic variable.
For the former group, if λ is mean or expected reading rate, the probability of reading a words in time t is given as:
P{a|t} = ((λt)^{a}/a!)e^{-λt} | (1) |
The probability that no words are read in time t is a special case, where N = 0. This reduces down to:
P{0 | t} = e-λt | (2) |
This expression may be applied to a student tackling a single item in a math test. And while it derives from Expression (1) for reading rates, and incorporates λ, (which could be interpreted as an expected item completion rate), the application will be from measuring the time during which the item is not completed.
Rasch (1960) broke down λ into two factors. He argued that the ratio of the reading rates between two pupils A and B is interesting if it applies to a number of reading texts. So if in a series of texts, 1, 2, ..., i, pupil A reads twice as fast as B:
λ_{A1} = 2λ_{B1} |
and
λ_{Ai} = 2λ_{Bi} |
Dividing:
λ_{Ai}/λ_{A1} = λ_{Bi}/λ_{B1} |
Generalising:
λ_{li}/λ_{11} = λ_{Ni}/λ_{N1} | (3) |
So the ratio of the mean or expected reading rates of two texts is independent of the pupils. That ratio tells is something about the texts (relative ease of reading or some other applicable descriptor) and it might be given the parameter ε:
λ_{vi}/λ_{v1} = ε |
Rearranging:
λ_{vi} = λ_{v1}ε | (4) |
Rasch (1960) described the term λv1, the reading speed of pupil v in a base text, as the person parameter ζ_{v}. He also defined the reciprocal of ε as the difficulty δ of a text.
When applying this argument to an individual item in a computer based math test, it should be born in mind that the focus is on a single event. It is impossible to predict exactly when the event will occur, and it is equally impossible to estimate the value of λ from knowing when the event occurred. To overcome this conundrum, a method might be borrowed from natural science.
When physicists consider a sample of radioactive material comprising many atoms, they apply the Law of Large Numbers (Khoshnevisan, 2007), which essentially states that if you run an experiment N times, where N is a very large number, if p is the probability of an event, the number of times the event actually occurs will approximate to Np. From Expression (2) above, the probability of a word not being read, or a math item not being completed, or a radioactive atom not decaying, in time t, is e^{-λt}. Studying a sample originally comprising N_{0} atoms, is like running an experiment N_{0} times. After time t, the approximate number N of atoms, which have not decayed, will be given by:
N(t) = N_{0}e^{-λt} | (5) |
The time t_{h} taken for N to become exactly half N_{0} is known as the half-life of the material.
N_{0}/2 = N_{0}e^{-λth} | ||
2 = e^{λth} | ||
ln(2) = λt_{h} | ||
λ = ln(2)/t_{h} | (6) |
So if you had a room full of clones all addressing the same item at the same time, you could estimate λ from the time it takes half of them to complete the item. Clones are not easy to come by, but there is another formula from physics which deals with composite radioactive material (L'Annunziata, 2012), and which could be applied to a heterogeneous set of pupils.
In the case of two elements, if the decay rate of Element 1 with N_{1} atoms is λ_{1} and that of Element 2 with N_{2} atoms is λ_{2}, the combined decay rate λ_{c}, or number of atoms decaying per unit of time is:
-dN/dt = N_{1}λ_{1} + N_{2}λ_{2} |
In psychometrics, there is usually assumed to be just one pupil of each type, so for two pupils, the combined item completion rate becomes:
λ_{c} = λ_{1} + λ_{2} |
Reverting to the Rasch notation of Expression (5), if these pupils are addressing item i:
λ_{ci} = ζ_{1}ε_{i} + ζ_{2}ε_{i} | ||
λ_{ci} = ε_{i}(ζ_{1} + ζ_{2}ε_{i}) |
If the same pupils address a second item j:
λ_{cj} = ε_{j}(ζ_{1} + ζ_{2}ε_{i}) |
Dividing:
λ_{ci}/λ_{cj} = ε_{i}(ζ_{1} + ζ_{2}ε_{i}) / ε_{j}(ζ_{1} + ζ_{2}ε_{i}) | ||
λ_{ci}/λ_{cj} = ε_{i} / ε_{j} | (7) |
So the ratio of the combined expected completion rates becomes the ratio of the easiness of the two items, and is independent of the person parameters of the two pupils. A similar argument applies to three or more pupils. Furthermore, the combined expected completion rate can be estimated for each item from the median completion time on each item using Expression (7).
Illustration
Figures 1 and 2 show for 85 West Australian primary school students, all of whom completed (correctly) the items "4+4", "3+5", and "12+8" in a computer-based math test, the completion times on item "3+5" against those on item "4+4", and the completion times on item "12+8" against those on item "4+4". This triple intersection set arose from a universal set of 14,480 student-item interactions. The settings were informal, with class teachers using the computer-based test as a regular class activity, as opposed to a formal exam.
for item 3+5 against 4+4 Figure 1. |
for item 12+8 against 4+4 Figure 2. |
Table 1 | ||||
---|---|---|---|---|
Item | Median Time (s) | λ | ε ratio | δ ratio |
4+4 | 1.58 | 0.44 | 1.00 | 1.00 |
3+5 | 2.50 | 0.28 | 0.63 | 1.58 |
12+8 | 5.00 | 0.14 | 0.50 | 2.00 |
Table 1 shows the median completion times in seconds and an estimate of λ_{c} for each item. The table also shows ratios of easiness ε and difficulty δ. From the table, if item "4+4" is treated as the base item, item "3+5" seems to be approximately two thirds as easy or 1.5 times as difficult, while "12+8" seems to be approximately half as easy or twice as difficult.
The purpose of this essay was to set out a method of estimating the Rasch item parameter from time on task. A method has been laid out, and an illustration has been given. The illustration looked at just three items, all of which had been addressed by the same pupils. Extending the method to cover all of the possible items, which might come up in a simple math test, will either require a very large sample of student-item interactions, or the development of a system, which does not require exactly the same pupils to address every item.
Jonathan Hippisley
Email: jhipp -/at\- softway.org
References
Hippisley J (1999) Looking at Data from an Interactive Arithmetic Test from the Perspective of a Probabilistic Model. Education Research & Perspectives, (25)2, 59-67.
Khoshnevisan D (2007) Probability Graduate Studies in Mathematics, 80. American Mathematical Society.
L'Annunziata M (2012) Handbook of Radioactivity Analysis. Academic Press.
Rasch G (1960) Probabilistic models for some intelligence and attainment tests University of Chicago Press, Chicago
A Method of Estimating the Item Parameter from Time on Task. Jonathan Hippisley … Rasch Measurement Transactions, 2012, 26:3 p. 1380-2
Rasch Publications | ||||
---|---|---|---|---|
Rasch Measurement Transactions (free, online) | Rasch Measurement research papers (free, online) | Probabilistic Models for Some Intelligence and Attainment Tests, Georg Rasch | Applying the Rasch Model 3rd. Ed., Bond & Fox | Best Test Design, Wright & Stone |
Rating Scale Analysis, Wright & Masters | Introduction to Rasch Measurement, E. Smith & R. Smith | Introduction to Many-Facet Rasch Measurement, Thomas Eckes | Invariant Measurement: Using Rasch Models in the Social, Behavioral, and Health Sciences, George Engelhard, Jr. | Statistical Analyses for Language Testers, Rita Green |
Rasch Models: Foundations, Recent Developments, and Applications, Fischer & Molenaar | Journal of Applied Measurement | Rasch models for measurement, David Andrich | Constructing Measures, Mark Wilson | Rasch Analysis in the Human Sciences, Boone, Stave, Yale |
in Spanish: | Análisis de Rasch para todos, Agustín Tristán | Mediciones, Posicionamientos y Diagnósticos Competitivos, Juan Ramón Oreja Rodríguez |
Forum | Rasch Measurement Forum to discuss any Rasch-related topic |
Go to Top of Page
Go to index of all Rasch Measurement Transactions
AERA members: Join the Rasch Measurement SIG and receive the printed version of RMT
Some back issues of RMT are available as bound volumes
Subscribe to Journal of Applied Measurement
Go to Institute for Objective Measurement Home Page. The Rasch Measurement SIG (AERA) thanks the Institute for Objective Measurement for inviting the publication of Rasch Measurement Transactions on the Institute's website, www.rasch.org.
Coming Rasch-related Events | |
---|---|
June 29 - July 27, 2018, Fri.-Fri. | On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com |
July 25 - July 27, 2018, Wed.-Fri. | Pacific-Rim Objective Measurement Symposium (PROMS), (Preconference workshops July 23-24, 2018) Fudan University, Shanghai, China "Applying Rasch Measurement in Language Assessment and across the Human Sciences", www.promsociety.org |
July 29 - August 4, 2018 | Vth International Summer School `Applied Psychometrics in Psychology and Education`, Institute of Education at the Higher School of Economics, St. Petersburg, Russia, https://ioe.hse.ru/en/announcements/215681182.html |
July 30 - Nov., 2018 | Online Introduction to Classical and Rasch Measurement Theories (D.Andrich), University of Western Australia, Perth, Australia, http://www.education.uwa.edu.au/ppl/courses |
Aug. 10 - Sept. 7, 2018, Fri.-Fri. | On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com |
August 25 - 28, 2018, Sat.-Tue. | Análisis de Rasch introductorio (en español). (Agustín Tristán), Instituto de Evaluación e Ingeniería Avanzada. San Luis Potosí, México. www.ieia.com.mx |
Sept. 3 - 6, 2018, Mon.-Thurs. | IMEKO World Congress, Belfast, Northern Ireland, www.imeko2018.org |
Oct. 12 - Nov. 9, 2018, Fri.-Fri. | On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com |
The URL of this page is www.rasch.org/rmt/rmt263d.htm,
Website: www.rasch.org/rmt/contents.htm,