What is The Attenuation Paradox?

"A paradoxical property of a test is a property such that the validity of the test is not a monotonic function of that property... Is it not intuitively valid, however, to demand that the most basic concept of psychometrics shall be a non-paradoxical property of tests? Reliability is paradoxical." J. Loevinger, 1954, pp. 500-501

The Attenuation Paradox was named by Loevinger (1954). It was recognized earlier by Gulliksen: "The criterion of maximizing test variance [reliability] cannot be pushed to extremes. Test variance is a maximum, if half of the population makes zero scores, and the other half makes perfect scores. Such a score distribution is not desirable for obvious reasons, yet current [true-score classical] test theory (CTT) provides no rationale for rejecting such a score distribution. Obviously the best test score distribution is one which accurately reflects the true ability distribution in the group, but there is perhaps little hope of obtaining such a distribution by the current procedure of assigning a score based upon the sheer number of correct answers. At present the only solution to such difficulties seems to lie in some type of absolute scaling theory (Thurstone, 1925), to replace the number correct score" (1945 pp. 90-91). Gulliksen,however, ignores Thurstone and perpetuates the paradoxical true-score tradition: "In order to maximize the reliability and variance of a test, the items should have high inter-correlations, all items should be of the same difficulty level, and the level should be as near to 50% as possible" (1945 p. 79).

Tucker (1946) provides an excellent analysis of the "inconsistencies between higher reliability and better measurement" (p.1). He observes that "if the reliability of the items were increased to unity, all correlations between the items would also become unity and a person passing one item would pass all items and another failing one item would fail all items. Thus the only possible scores are a perfect score or one of zero... Is a dichotomy of scores the best that can be expected from a test with items of equal difficulty?" (p. 2). Using scaling theory (in current terminology, a two-parameter item response theory model based on the normal ogive and random normal probabilities), Tucker shows how increasing test reliability must lead to decreasing test validity.

Laudan, Laudan and Donovan (1988) describe seven empirically testable hypotheses regarding how scientists react to data-dominated empirical anomalies and theory-dominated conceptual paradoxes. The reactions of psychometricians to the attenuation paradox of true-score theory provide instructive case studies of how scientists function. In the next columns, I will examine how measurement theorists of the 1940s, 1950s and today react to the Attenuation Paradox.

Gulliksen, H. (1945). The relation of item difficulty and inter-item correlation to test variance and reliability. Psychometrika, 10(2), 79-91.

Laudan, R., Laudan, L., & Donovan, A. (1988). Testing theories of scientific change. In A. Donovan, L. Laudan, & R. Laudan (Eds.), Scrutinizing science: Empirical studies of scientific change (pp. 3- 44). Dordrecht, The Netherlands: Kluwer Academic Publishers.

Loevinger, J. (1954). The attenuation paradox in test theory. Psychological Bulletin, 51, 493-504.

Thurstone, L.L. (1925). A method of scaling psychological and educational tests. Journal of Educational Psychology, 16(7), 433-451.

Tucker, L.R. (1946). Maximum validity of a test with equivalent items. Psychometrika, 11(1), 1-13.



What is The Attenuation Paradox?, G Engelhard Jr … Rasch Measurement Transactions, 1993, 6:4 p. 257




Rasch-Related Resources: Rasch Measurement YouTube Channel
Rasch Measurement Transactions & Rasch Measurement research papers - free An Introduction to the Rasch Model with Examples in R (eRm, etc.), Debelak, Strobl, Zeigenfuse Rasch Measurement Theory Analysis in R, Wind, Hua Applying the Rasch Model in Social Sciences Using R, Lamprianou El modelo métrico de Rasch: Fundamentación, implementación e interpretación de la medida en ciencias sociales (Spanish Edition), Manuel González-Montesinos M.
Rasch Models: Foundations, Recent Developments, and Applications, Fischer & Molenaar Probabilistic Models for Some Intelligence and Attainment Tests, Georg Rasch Rasch Models for Measurement, David Andrich Constructing Measures, Mark Wilson Best Test Design - free, Wright & Stone
Rating Scale Analysis - free, Wright & Masters
Virtual Standard Setting: Setting Cut Scores, Charalambos Kollias Diseño de Mejores Pruebas - free, Spanish Best Test Design A Course in Rasch Measurement Theory, Andrich, Marais Rasch Models in Health, Christensen, Kreiner, Mesba Multivariate and Mixture Distribution Rasch Models, von Davier, Carstensen
Rasch Books and Publications: Winsteps and Facets
Applying the Rasch Model (Winsteps, Facets) 4th Ed., Bond, Yan, Heene Advances in Rasch Analyses in the Human Sciences (Winsteps, Facets) 1st Ed., Boone, Staver Advances in Applications of Rasch Measurement in Science Education, X. Liu & W. J. Boone Rasch Analysis in the Human Sciences (Winsteps) Boone, Staver, Yale Appliquer le modèle de Rasch: Défis et pistes de solution (Winsteps) E. Dionne, S. Béland
Introduction to Many-Facet Rasch Measurement (Facets), Thomas Eckes Rasch Models for Solving Measurement Problems (Facets), George Engelhard, Jr. & Jue Wang Statistical Analyses for Language Testers (Facets), Rita Green Invariant Measurement with Raters and Rating Scales: Rasch Models for Rater-Mediated Assessments (Facets), George Engelhard, Jr. & Stefanie Wind Aplicação do Modelo de Rasch (Português), de Bond, Trevor G., Fox, Christine M
Exploring Rating Scale Functioning for Survey Research (R, Facets), Stefanie Wind Rasch Measurement: Applications, Khine Winsteps Tutorials - free
Facets Tutorials - free
Many-Facet Rasch Measurement (Facets) - free, J.M. Linacre Fairness, Justice and Language Assessment (Winsteps, Facets), McNamara, Knoch, Fan

To be emailed about new material on www.rasch.org
please enter your email address here:

I want to Subscribe: & click below
I want to Unsubscribe: & click below

Please set your SPAM filter to accept emails from Rasch.org

www.rasch.org welcomes your comments:

Your email address (if you want us to reply):

 

ForumRasch Measurement Forum to discuss any Rasch-related topic

Go to Top of Page
Go to index of all Rasch Measurement Transactions
AERA members: Join the Rasch Measurement SIG and receive the printed version of RMT
Some back issues of RMT are available as bound volumes
Subscribe to Journal of Applied Measurement

Go to Institute for Objective Measurement Home Page. The Rasch Measurement SIG (AERA) thanks the Institute for Objective Measurement for inviting the publication of Rasch Measurement Transactions on the Institute's website, www.rasch.org.

Coming Rasch-related Events
Oct. 4 - Nov. 8, 2024, Fri.-Fri. On-line workshop: Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
Jan. 17 - Feb. 21, 2025, Fri.-Fri. On-line workshop: Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
May 16 - June 20, 2025, Fri.-Fri. On-line workshop: Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 20 - July 18, 2025, Fri.-Fri. On-line workshop: Rasch Measurement - Further Topics (E. Smith, Facets), www.statistics.com
Oct. 3 - Nov. 7, 2025, Fri.-Fri. On-line workshop: Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com

 

The URL of this page is www.rasch.org/rmt/rmt64h.htm

Website: www.rasch.org/rmt/contents.htm