"The reliability of any set of measurements is logically defined as the proportion of their variance that is true variance... We think of the total variance of a set of measures as being made up of two sources of variance: true variance and error variance... The true measure is assumed to be the genuine value of whatever is being measured... The error components occur independently and at random" (Guilford 1965, p.439-40).
Observed Variance = "True" Variance + Error Variance
Reliability = "True" Variance / Observed Variance
In Rasch terms, "True" valiance is the "adjusted" variance (observed variance adjusted for measurement error). Error Variance is a mean-square error (derived from the model) inflated by misfit to the model encountered in the data.
Kubiszyn and Borich (1993, p.353) say "an acceptable standardized test should have reliability coefficients of about .95 for internal consistency". But "reliability depends upon the population measured as well as the measuring instrument... One should speak of the reliability of a certain instrument applied to a certain population under certain conditions" (Guilford p.439). This is because the "true" variance is a characteristic of the sample tested and the "error" variance is a characteristic of the measuring instrument.
Since reliability is restricted to the range 0 to 1, it is convenient to express it as a separation coefficient with range 0 to infinity:
G = Separation = sqrt(Rel./(1-Rel.)) =
"True" Standard Deviation / Error Standard Deviation
In Rasch literature, the "True" S.D. is often called the "Adjusted" (for measurement error) S.D.
Separation is the number of statistically different performance strata that the test can identify in the sample. This can be pictured by placing an error distribution in each stratum (see Examples). A separation of "2" implies that only two levels of performance can be consistently identified by the test for samples like the one tested. Kubiszyn & Borich's 0.95 corresponds to a separation of 4.5, i.e., 4 consistently identifiable strata. (See RMT 6:3 p. 238 for a Table of Reliability - Separation equivalences). Compare the Separation plot with with a different definition of "ability strata" at Number of Person or Item Strata (4G+1)/3 (RMT 16:3).
Example 1: Separation = 2.0, "True" S.D. = 2.0, Error S.D. = 1.0
Reliability = (2.0*2.0) / (2.0*2.0 + 1.0*.1.0) = 0.8
Example 2:Separation = 3.0, "True" S.D. = 3.0, Error S.D. = 1.0
Reliability = (3.0*3.0) / (3.0*3.0 + 1.0*.1.0) = 0.9
In the graph below, notice that the middle error strata is from -1.5 Error S.D. to +1.5 Error S.D.
Guilford J. P. (1965) Fundamental Statistics in Psychology and Education. 4th Edn. New York: McGraw-Hill.
Kubiszyn T., Borich G. (1993) Educational Testing and Measurement. New York: Harper Collins.
The levels in the observed distribution are 3*"Observed S.D."/G apart, centered on the sample mean.
Reliability and separation. Wright BD. Rasch Measurement Transactions, 1996, 9:4 p.472
Please help with Standard Dataset 4: Andrich Rating Scale Model
|Rasch Measurement Transactions (free, online)||Rasch Measurement research papers (free, online)||Probabilistic Models for Some Intelligence and Attainment Tests, Georg Rasch||Applying the Rasch Model 3rd. Ed., Bond & Fox||Best Test Design, Wright & Stone|
|Rating Scale Analysis, Wright & Masters||Introduction to Rasch Measurement, E. Smith & R. Smith||Introduction to Many-Facet Rasch Measurement, Thomas Eckes||Invariant Measurement: Using Rasch Models in the Social, Behavioral, and Health Sciences, George Engelhard, Jr.||Statistical Analyses for Language Testers, Rita Green|
|Rasch Models: Foundations, Recent Developments, and Applications, Fischer & Molenaar||Journal of Applied Measurement||Rasch models for measurement, David Andrich||Constructing Measures, Mark Wilson||Rasch Analysis in the Human Sciences, Boone, Stave, Yale|
|in Spanish:||Análisis de Rasch para todos, Agustín Tristán||Mediciones, Posicionamientos y Diagnósticos Competitivos, Juan Ramón Oreja Rodríguez|
|Forum||Rasch Measurement Forum to discuss any Rasch-related topic|
Go to Top of Page
Go to index of all Rasch Measurement Transactions
AERA members: Join the Rasch Measurement SIG and receive the printed version of RMT
Some back issues of RMT are available as bound volumes
Subscribe to Journal of Applied Measurement
Go to Institute for Objective Measurement Home Page. The Rasch Measurement SIG (AERA) thanks the Institute for Objective Measurement for inviting the publication of Rasch Measurement Transactions on the Institute's website, www.rasch.org.
|Coming Rasch-related Events|
|Jan. 25-26, 2017, Wed.-Thurs.||In-person workshop: Measurement with the Rasch Model (M. Pampaka, J. Williams, Winsteps), Manchester, UK, website|
|Feb. 27 - June 24, 2017, Mon.-Sat.||On-line: Advanced course in Rasch Measurement Theory (EDUC5606), Website|
|March 31, 2017, Fri.||Conference: 11th UK Rasch Day, Warwick, UK, www.rasch.org.uk|
|April 2-3, 2017, Sun.-Mon.||Conference: Validity Evidence for Measurement in Mathematics Education (V-M2Ed), San Antonio, TX, Information|
|April 26-30, 2017, Wed.-Sun.||NCME, San Antonio, TX, www.ncme.org|
|April 27 - May 1, 2017, Thur.-Mon.||AERA, San Antonio, TX, www.aera.net|
|May 26 - June 23, 2017, Fri.-Fri.||On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com|
|June 30 - July 29, 2017, Fri.-Fri.||On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com|
|July 31 - Aug. 3, 2017, Mon.-Thurs.||Joint IMEKO TC1-TC7-TC13 Symposium 2017: Measurement Science challenges in Natural and Social Sciences, Rio de Janeiro, Brazil, imeko-tc7-rio.org.br|
|Aug. 7-9, 2017, Mon-Wed.||PROMS 2017: Pacific Rim Objective Measurement Symposium, Sabah, Borneo, Malaysia, proms.promsociety.org/2017/|
|Aug. 11 - Sept. 8, 2017, Fri.-Fri.||On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com|
|Aug. 18-21, 2017, Fri.-Mon.||IACAT 2017: International Association for Computerized Adaptive Testing, Niigata, Japan, iacat.org|
|Sept. 15-16, 2017, Fri.-Sat.||IOMC 2017: International Outcome Measurement Conference, Chicago, jampress.org/iomc2017.htm|
|Oct. 13 - Nov. 10, 2017, Fri.-Fri.||On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com|
|Jan. 5 - Feb. 2, 2018, Fri.-Fri.||On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com|
|Jan. 10-16, 2018, Wed.-Tues.||In-person workshop: Advanced Course in Rasch Measurement Theory and the application of RUMM2030, Perth, Australia (D. Andrich), Announcement|
|Jan. 17-19, 2018, Wed.-Fri.||Rasch Conference: Seventh International Conference on Probabilistic Models for Measurement, Matilda Bay Club, Perth, Australia, Website|
|May 25 - June 22, 2018, Fri.-Fri.||On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com|
|June 29 - July 27, 2018, Fri.-Fri.||On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com|
|Aug. 10 - Sept. 7, 2018, Fri.-Fri.||On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com|
|Oct. 12 - Nov. 9, 2018, Fri.-Fri.||On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com|
|The HTML to add "Coming Rasch-related Events" to your webpage is:|