Comparison of measures is frequently required. Item calibrations are compared when tests are equated to investigate stability. Person measures are compared to investigate change. Here are some techniques of comparison.
A necessary step before final conclusions can be drawn is to establish and maintain the frame of reference. Suppose the pre-test is Form E and the post-test is Form H. Simply subtracting a mean ability measure at pre-test from a mean ability at post-test is misleading as a measure of change for many reasons. The most obvious is that pre-test measures are relative to Form E and post-test measures are relative to Form H. The two Forms must be equated into a single frame of reference, i.e., with a shared origin and scale. Only then can valid comparisons between pre- and post-test measures be made. This equating may be an iterative procedure in which the following techniques can help.
A. Numerical comparisons.
A.1 Direct comparisons:
If Bn is the measure of subject n, and Bm the measure of subject
m, the obvious comparison is Bn-Bm. If SEn and SEm are the
standard errors of Bn and Bm, then the SE of Bn-Bm =
sqrt((SEn^2+SEm^2). A two-tailed test of the null hypothesis
"Bn-Bm=0" is rejected at the .05 level when
|Bn-Bm| >= 1.96*sqrt((SEn^2+SEm^2)
>= 2.8 * SEn when SEn = SEm,
and at the .01 level when
|Bn-Bm| >= 2.58*sqrt((SEn^2+SEm^2)
>= 3.7 * SEn when SEn = SEm.
A.2 Normative comparisons:
It is often of interest to discover whether, as a group changes,
individuals are changing faster or slower than their group.
Suppose B1 is the measure of a subject at time 1, when the mean
of the N measures is G1, and B2 is the measure of the same
subject at time 2, when the group mean is G2. Then the gain of
the subject relative to the group is (B2-G2)-(B1-G1). The error
variances of the () terms have the form
SEBG1^2 = SEB1^2 + SDG1^2/(N-1).
The SE of the gain = sqrt((SEBG1^2+SEBG2^2).
A.3 Comparisons requiring rescaling:
Tests X and Y contain some equivalent items, but, due to
different candidate behavior, rating scale specifications or
analytical decisions, the logit units of the measurement scales
estimated for the two tests have different lengths.
Suppose there are C items common to both tests. Their mean and
variance in Test X are MX and VX, and in Test Y, MY and VY. Then
the rescaling factor to place measures on Test Y into the scale
of Test X is
SCALEYX = sqrt((VX/VY)
so that
B'Y = (BY - MY)*SCALEYX + MX
with
SE'Y = SEY*SCALEYX
Comparisons can now be made between BX and B'Y in the metric of
Test X.
A.4 Evaluation of group stability:
Suppose there are C common elements in tests X and Y. Whether to
attribute the inevitable differences in pairs of measures solely
to measurement error is a statistical question. A test of the
null hypothesis that the differences are attributable to
measurement error is the homogeneity chi-square:
B. Graphical comparisons.
B.1 Graphing without common scaling:
The numerical calculations can be summarized in a scatterplot of
measures and confidence bands. Each element has two measures, BX
and BY, and two standard errors, SEX and SEY.
Plotting confidence intervals is a two-stage process. For conventional 95% two-tailed confidence bands, the pairs of points are located at {UX, UY} for the upper confidence band and {LX, LY} for the lower band, where
When you plot the {UX, UY} points, they will form a curved cloud (Figure 1). The points are exact, but provide too much detail. To expedite conceptualization, draw a smooth curve through these points with a thick felt-tip pen or computer drawing tool. Then erase the points. Repeat this for the {LX,LY} points. This draws the 95% confidence bands. Now plot your {BX,BY} data points (see Figure 2 (in printed text)). The identity line goes through {MEANX+AVGEX, MEANY+AVGEY}.
B.2 Graphing with local scaling:
In Figure 2, the best fit line goes through
points {MEANX+AVGEX-SDX, MEANY+AVGEY-SDY} and {MEANX+AVGEX+SDX,
MEANY+AVGEY+SDY}. If this forms a noticeable angle with an
identity line, then plot the confidence bands with the X and Y
axes separately scaled. The locally scaled confidence bands are
plotted in the same way as in B.1, but using these values:
The best fit line now falls evenly between the bands.
Comparing measures: scatterplots. Luppescu S. … Rasch Measurement Transactions, 1995, 9:1 p.410
Rasch Publications | ||||
---|---|---|---|---|
Rasch Measurement Transactions (free, online) | Rasch Measurement research papers (free, online) | Probabilistic Models for Some Intelligence and Attainment Tests, Georg Rasch | Applying the Rasch Model 3rd. Ed., Bond & Fox | Best Test Design, Wright & Stone |
Rating Scale Analysis, Wright & Masters | Introduction to Rasch Measurement, E. Smith & R. Smith | Introduction to Many-Facet Rasch Measurement, Thomas Eckes | Invariant Measurement: Using Rasch Models in the Social, Behavioral, and Health Sciences, George Engelhard, Jr. | Statistical Analyses for Language Testers, Rita Green |
Rasch Models: Foundations, Recent Developments, and Applications, Fischer & Molenaar | Journal of Applied Measurement | Rasch models for measurement, David Andrich | Constructing Measures, Mark Wilson | Rasch Analysis in the Human Sciences, Boone, Stave, Yale |
in Spanish: | Análisis de Rasch para todos, Agustín Tristán | Mediciones, Posicionamientos y Diagnósticos Competitivos, Juan Ramón Oreja Rodríguez |
Forum | Rasch Measurement Forum to discuss any Rasch-related topic |
Go to Top of Page
Go to index of all Rasch Measurement Transactions
AERA members: Join the Rasch Measurement SIG and receive the printed version of RMT
Some back issues of RMT are available as bound volumes
Subscribe to Journal of Applied Measurement
Go to Institute for Objective Measurement Home Page. The Rasch Measurement SIG (AERA) thanks the Institute for Objective Measurement for inviting the publication of Rasch Measurement Transactions on the Institute's website, www.rasch.org.
Coming Rasch-related Events | |
---|---|
Oct. 9 - Nov. 6, 2020, Fri.-Fri. | On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com |
Jan. 22 -Feb. 19, 2021, Fri.-Fri. | On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com |
May 21 -June 18, 2021, Fri.-Fri. | On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com |
June 25 - July 23, 2021, Fri.-Fri. | On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com |
Aug. 13 - Sept. 10, 2021, Fri.-Fri. | On-line workshop: Many-Facet Rasch Measurement (E. Smith,Facets), www.statistics.com |
June 24 - July 22, 2022, Fri.-Fri. | On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com |
The URL of this page is www.rasch.org/rmt/rmt91c.htm
Website: www.rasch.org/rmt/contents.htm