A Method of Estimating the Item Parameter from Time on Task

The purpose of this essay is to revisit Chapter 3 of Probabilistic Models (Rasch, 1960) and to consider to what extent the arguments from this chapter might be applied to individual items in a computer based math test. The intention is to look for inspiration from Rasch, but not to follow his methodology exactly.

Hippisley (1999) showed that total completion times from a computer based math test conformed to the (Rasch 1960) reading rate model. However, the assumption of item homogeneity and uniform student speed, rather limits the usefulness of that analysis. In reality, test items are not homogenous, and students do not course through a test at a uniform speed.

Theory

Rasch considered the "speededness" of students reading a text in two ways. For those students who did not complete the text within a prescribed time limit, he treated the number of words read within the time as the stochastic variable. For the students who did complete the test (within the time limit), he treated the time actually taken as the stochastic variable.

For the former group, if λ is mean or expected reading rate, the probability of reading a words in time t is given as:

 P{a|t} = ((λt)a/a!)e-λt (1)

The probability that no words are read in time t is a special case, where N = 0. This reduces down to:

 P{0 | t} = e-λt (2)

This expression may be applied to a student tackling a single item in a math test. And while it derives from Expression (1) for reading rates, and incorporates λ, (which could be interpreted as an expected item completion rate), the application will be from measuring the time during which the item is not completed.

Rasch (1960) broke down λ into two factors. He argued that the ratio of the reading rates between two pupils A and B is interesting if it applies to a number of reading texts. So if in a series of texts, 1, 2, ..., i, pupil A reads twice as fast as B:

 λA1 = 2λB1

and

 λAi = 2λBi

Dividing:

 λAi/λA1 = λBi/λB1

Generalising:

 λli/λ11 = λNi/λN1 (3)

So the ratio of the mean or expected reading rates of two texts is independent of the pupils. That ratio tells is something about the texts (relative ease of reading or some other applicable descriptor) and it might be given the parameter ε:

 λvi/λv1 = ε

Rearranging:

 λvi = λv1ε (4)

Rasch (1960) described the term λv1, the reading speed of pupil v in a base text, as the person parameter ζv. He also defined the reciprocal of ε as the difficulty δ of a text.

When applying this argument to an individual item in a computer based math test, it should be born in mind that the focus is on a single event. It is impossible to predict exactly when the event will occur, and it is equally impossible to estimate the value of λ from knowing when the event occurred. To overcome this conundrum, a method might be borrowed from natural science.

When physicists consider a sample of radioactive material comprising many atoms, they apply the Law of Large Numbers (Khoshnevisan, 2007), which essentially states that if you run an experiment N times, where N is a very large number, if p is the probability of an event, the number of times the event actually occurs will approximate to Np. From Expression (2) above, the probability of a word not being read, or a math item not being completed, or a radioactive atom not decaying, in time t, is e-λt. Studying a sample originally comprising N0 atoms, is like running an experiment N0 times. After time t, the approximate number N of atoms, which have not decayed, will be given by:

 N(t) = N0e-λt (5)

The time th taken for N to become exactly half N0 is known as the half-life of the material.

 N0/2 = N0e-λth 2 = eλth ln(2) = λth λ = ln(2)/th (6)

So if you had a room full of clones all addressing the same item at the same time, you could estimate λ from the time it takes half of them to complete the item. Clones are not easy to come by, but there is another formula from physics which deals with composite radioactive material (L'Annunziata, 2012), and which could be applied to a heterogeneous set of pupils.

In the case of two elements, if the decay rate of Element 1 with N1 atoms is λ1 and that of Element 2 with N2 atoms is λ2, the combined decay rate λc, or number of atoms decaying per unit of time is:

 -dN/dt = N1λ1 + N2λ2

In psychometrics, there is usually assumed to be just one pupil of each type, so for two pupils, the combined item completion rate becomes:

 λc = λ1 + λ2

Reverting to the Rasch notation of Expression (5), if these pupils are addressing item i:

 λci = ζ1εi + ζ2εi λci = εi(ζ1 + ζ2εi)

If the same pupils address a second item j:

 λcj = εj(ζ1 + ζ2εi)

Dividing:

 λci/λcj = εi(ζ1 + ζ2εi) / εj(ζ1 + ζ2εi) λci/λcj = εi / εj (7)

So the ratio of the combined expected completion rates becomes the ratio of the easiness of the two items, and is independent of the person parameters of the two pupils. A similar argument applies to three or more pupils. Furthermore, the combined expected completion rate can be estimated for each item from the median completion time on each item using Expression (7).

Illustration

Figures 1 and 2 show for 85 West Australian primary school students, all of whom completed (correctly) the items "4+4", "3+5", and "12+8" in a computer-based math test, the completion times on item "3+5" against those on item "4+4", and the completion times on item "12+8" against those on item "4+4". This triple intersection set arose from a universal set of 14,480 student-item interactions. The settings were informal, with class teachers using the computer-based test as a regular class activity, as opposed to a formal exam.

 Completion times in secsfor item 3+5 against 4+4 Figure 1. Completion times in secsfor item 12+8 against 4+4 Figure 2.

Table 1
ItemMedian Time (s)λε ratioδ ratio
4+41.580.441.001.00
3+52.500.280.631.58
12+85.000.140.502.00

Table 1 shows the median completion times in seconds and an estimate of λc for each item. The table also shows ratios of easiness ε and difficulty δ. From the table, if item "4+4" is treated as the base item, item "3+5" seems to be approximately two thirds as easy or 1.5 times as difficult, while "12+8" seems to be approximately half as easy or twice as difficult.

The purpose of this essay was to set out a method of estimating the Rasch item parameter from time on task. A method has been laid out, and an illustration has been given. The illustration looked at just three items, all of which had been addressed by the same pupils. Extending the method to cover all of the possible items, which might come up in a simple math test, will either require a very large sample of student-item interactions, or the development of a system, which does not require exactly the same pupils to address every item.

Jonathan Hippisley
Email: jhipp -/at\- softway.org

References

Hippisley J (1999) Looking at Data from an Interactive Arithmetic Test from the Perspective of a Probabilistic Model. Education Research & Perspectives, (25)2, 59-67.

Khoshnevisan D (2007) Probability Graduate Studies in Mathematics, 80. American Mathematical Society.

Rasch G (1960) Probabilistic models for some intelligence and attainment tests University of Chicago Press, Chicago

A Method of Estimating the Item Parameter from Time on Task. Jonathan Hippisley … Rasch Measurement Transactions, 2012, 26:3 p. 1380-2

Rasch Publications
Rasch Measurement Transactions (free, online) Rasch Measurement research papers (free, online) Probabilistic Models for Some Intelligence and Attainment Tests, Georg Rasch Applying the Rasch Model 3rd. Ed., Bond & Fox Best Test Design, Wright & Stone
Rating Scale Analysis, Wright & Masters Introduction to Rasch Measurement, E. Smith & R. Smith Introduction to Many-Facet Rasch Measurement, Thomas Eckes Invariant Measurement: Using Rasch Models in the Social, Behavioral, and Health Sciences, George Engelhard, Jr. Statistical Analyses for Language Testers, Rita Green
Rasch Models: Foundations, Recent Developments, and Applications, Fischer & Molenaar Journal of Applied Measurement Rasch models for measurement, David Andrich Constructing Measures, Mark Wilson Rasch Analysis in the Human Sciences, Boone, Stave, Yale
in Spanish: Análisis de Rasch para todos, Agustín Tristán Mediciones, Posicionamientos y Diagnósticos Competitivos, Juan Ramón Oreja Rodríguez

 Forum Rasch Measurement Forum to discuss any Rasch-related topic

Go to Top of Page
Go to index of all Rasch Measurement Transactions
AERA members: Join the Rasch Measurement SIG and receive the printed version of RMT
Some back issues of RMT are available as bound volumes
Subscribe to Journal of Applied Measurement

Go to Institute for Objective Measurement Home Page. The Rasch Measurement SIG (AERA) thanks the Institute for Objective Measurement for inviting the publication of Rasch Measurement Transactions on the Institute's website, www.rasch.org.

Coming Rasch-related Events
March 21, 2019, Thur. 13th annual meeting of the UK Rasch user group, Cambridge, UK, http://www.cambridgeassessment.org.uk/events/uk-rasch-user-group-2019
April 4 - 8, 2019, Thur.-Mon. NCME annual meeting, Toronto, Canada,https://ncme.connectedcommunity.org/meetings/annual
April 5 - 9, 2019, Fri.-Tue. AERA annual meeting, Toronto, Canada,www.aera.net/Events-Meetings/Annual-Meeting
April 12, 2019, Fri. On-line course: Understanding Rasch Measurement Theory - Master's Level (G. Masters), https://www.acer.org/au/professional-learning/postgraduate/rasch
May 24 - June 21, 2019, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
May 22 - 30, 2019, Wed.-Thu. Measuring and scale construction (with the Rasch Model), University of Manchester, England, https://www.cmist.manchester.ac.uk/study/short/intermediate/measurement-with-the-rasch-model/
June 4 - 7, 2019, Tue.-Fri.In-Person Italian Rasch Analysis Workshop based on RUMM (Fabio La Porta and Serena Caselli; entirely in Italian). Prof David Andrich from Western Australia University will be hosted by the workshop. For enquiries and registration email to workshop.rasch@gmail.com
June 17-19, 2019, Mon.-Wed. In-person workshop, Melbourne, Australia: Applying the Rasch Model in the Human Sciences: Introduction to Rasch measurement (Trevor Bond, Winsteps), Announcement
June 20-21, 2019, Thurs.-Fri. In-person workshop, Melbourne, Australia: Applying the Rasch Model in the Human Sciences: Advanced Rasch measurement with Facets (Trevor Bond, Facets), Announcement
June 28 - July 26, 2019, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
July 2-5, 2019, Tue.-Fri. 2019 International Measurement Confederation (IMEKO) Joint Symposium, St. Petersburg, Russia,https://imeko19-spb.org
July 11-12 & 15-19, 2019, Thu.-Fri. A Course in Rasch Measurement Theory (D.Andrich), University of Western Australia, Perth, Australia, flyer - http://www.education.uwa.edu.au/ppl/courses
Aug 5 - 10, 2019, Mon.-Sat. 6th International Summer School "Applied Psychometrics in Psychology and Education", Institute of Education at HSE University Moscow, Russia.https://ioe.hse.ru/en/announcements/248134963.html
Aug. 9 - Sept. 6, 2019, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com
Aug. 14 - 16, 2019. Wed.-Fri. An Introduction to Rasch Measurement: Theory and Applications (workshop led by Richard M. Smith) https://www.hkr.se/pmhealth2019rs
August 25-30, 2019, Sun.-Fri. Pacific Rim Objective Measurement Society (PROMS) 2019, Surabaya, Indonesia https://proms.promsociety.org/2019/
Oct. 11 - Nov. 8, 2019, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
Nov. 3 - Nov. 4, 2019, Sun.-Mon. International Outcome Measurement Conference, Chicago, IL,http://jampress.org/iomc2019.htm
Jan. 24 - Feb. 21, 2020, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
May 22 - June 19, 2020, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 26 - July 24, 2020, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
Aug. 7 - Sept. 4, 2020, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com
Oct. 9 - Nov. 6, 2020, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 25 - July 23, 2021, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com