Reliability and Separation

"The reliability of any set of measurements is logically defined as the proportion of their variance that is true variance... We think of the total variance of a set of measures as being made up of two sources of variance: true variance and error variance... The true measure is assumed to be the genuine value of whatever is being measured... The error components occur independently and at random" (Guilford 1965, p.439-40).

Observed Variance = "True" Variance + Error Variance

Reliability = "True" Variance / Observed Variance

In Rasch terms, "True" valiance is the "adjusted" variance (observed variance adjusted for measurement error). Error Variance is a mean-square error (derived from the model) inflated by misfit to the model encountered in the data.

Kubiszyn and Borich (1993, p.353) say "an acceptable standardized test should have reliability coefficients of about .95 for internal consistency". But "reliability depends upon the population measured as well as the measuring instrument... One should speak of the reliability of a certain instrument applied to a certain population under certain conditions" (Guilford p.439). This is because the "true" variance is a characteristic of the sample tested and the "error" variance is a characteristic of the measuring instrument.

Since reliability is restricted to the range 0 to 1, it is convenient to express it as a separation coefficient with range 0 to infinity:

G = Separation = sqrt(Rel./(1-Rel.)) =
"True" Standard Deviation / Error Standard Deviation

In Rasch literature, the "True" S.D. is often called the "Adjusted" (for measurement error) S.D.

Separation is the number of statistically different performance strata that the test can identify in the sample. This can be pictured by placing an error distribution in each stratum (see Examples). A separation of "2" implies that only two levels of performance can be consistently identified by the test for samples like the one tested. Kubiszyn & Borich's 0.95 corresponds to a separation of 4.5, i.e., 4 consistently identifiable strata. (See RMT 6:3 p. 238 for a Table of Reliability - Separation equivalences). Compare the Separation plot with with a different definition of "ability strata" at Number of Person or Item Strata (4G+1)/3 (RMT 16:3).

Example 1: Separation = 2.0, "True" S.D. = 2.0, Error S.D. = 1.0
Reliability = (2.0*2.0) / (2.0*2.0 + 1.0*.1.0) = 0.8

Separation of 2


Example 2:Separation = 3.0, "True" S.D. = 3.0, Error S.D. = 1.0
Reliability = (3.0*3.0) / (3.0*3.0 + 1.0*.1.0) = 0.9
In the graph below, notice that the middle error strata is from -1.5 Error S.D. to +1.5 Error S.D.

Separation of 3


Guilford J. P. (1965) Fundamental Statistics in Psychology and Education. 4th Edn. New York: McGraw-Hill.

Kubiszyn T., Borich G. (1993) Educational Testing and Measurement. New York: Harper Collins.

Note:

  • For separation G, the levels in the true distribution are 3*"True S.D."/G apart, centered on the sample mean.
    The levels in the observed distribution are 3*"Observed S.D."/G apart, centered on the sample mean.


    1. Reliability, separation, strata statistics, Fisher WP Jr. … 6:3 p.238
    2. Reliability and separation nomograms, Linacre JM. … 1995, 9:2 p.421
    3. Reliability and separation, Wright BD. … 1996, 9:4 p.472
    4. Predicting Reliabilities and Separations of Different Length Tests, Linacre, J.M. … 2000, 14:3 p.767
    5. Going beyond Unreliable Reliabilities, Mallinson T., Stelmack J. … 2001, 14:4 p.787-8
    6. Separation, Reliability and Skewed Distributions: Statistically Different Levels of Performance, Wright B.D. … 2001, 14:4 p.786
    7. Number of Person or Item Strata (4G+1)/3, Wright BD, Masters GN. … 2002, 16:3 p.888
    8. Cash value of Reliability, WP Fisher … Rasch Measurement Transactions, 2008, 22:1 p. 1160

    Reliability and separation. Wright BD. … Rasch Measurement Transactions, 1996, 9:4 p.472



    Rasch Publications
    Rasch Measurement Transactions (free, online) Rasch Measurement research papers (free, online) Probabilistic Models for Some Intelligence and Attainment Tests, Georg Rasch Applying the Rasch Model 3rd. Ed., Bond & Fox Best Test Design, Wright & Stone
    Rating Scale Analysis, Wright & Masters Introduction to Rasch Measurement, E. Smith & R. Smith Introduction to Many-Facet Rasch Measurement, Thomas Eckes Invariant Measurement: Using Rasch Models in the Social, Behavioral, and Health Sciences, George Engelhard, Jr. Statistical Analyses for Language Testers, Rita Green
    Rasch Models: Foundations, Recent Developments, and Applications, Fischer & Molenaar Journal of Applied Measurement Rasch models for measurement, David Andrich Constructing Measures, Mark Wilson Rasch Analysis in the Human Sciences, Boone, Stave, Yale
    in Spanish: Análisis de Rasch para todos, Agustín Tristán Mediciones, Posicionamientos y Diagnósticos Competitivos, Juan Ramón Oreja Rodríguez

    To be emailed about new material on www.rasch.org
    please enter your email address here:

    I want to Subscribe: & click below
    I want to Unsubscribe: & click below

    Please set your SPAM filter to accept emails from Rasch.org

    www.rasch.org welcomes your comments:

    Your email address (if you want us to reply):

     

    ForumRasch Measurement Forum to discuss any Rasch-related topic

    Go to Top of Page
    Go to index of all Rasch Measurement Transactions
    AERA members: Join the Rasch Measurement SIG and receive the printed version of RMT
    Some back issues of RMT are available as bound volumes
    Subscribe to Journal of Applied Measurement

    Go to Institute for Objective Measurement Home Page. The Rasch Measurement SIG (AERA) thanks the Institute for Objective Measurement for inviting the publication of Rasch Measurement Transactions on the Institute's website, www.rasch.org.

    Coming Rasch-related Events
    June 29 - July 27, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
    July 25 - July 27, 2018, Wed.-Fri. Pacific-Rim Objective Measurement Symposium (PROMS), (Preconference workshops July 23-24, 2018) Fudan University, Shanghai, China "Applying Rasch Measurement in Language Assessment and across the Human Sciences", www.promsociety.org
    July 29 - August 4, 2018 Vth International Summer School `Applied Psychometrics in Psychology and Education`, Institute of Education at the Higher School of Economics, St. Petersburg, Russia, https://ioe.hse.ru/en/announcements/215681182.html
    July 30 - Nov., 2018Online Introduction to Classical and Rasch Measurement Theories (D.Andrich), University of Western Australia, Perth, Australia, http://www.education.uwa.edu.au/ppl/courses
    Aug. 10 - Sept. 7, 2018, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com
    August 25 - 28, 2018, Sat.-Tue.Análisis de Rasch introductorio (en español). (Agustín Tristán), Instituto de Evaluación e Ingeniería Avanzada. San Luis Potosí, México. www.ieia.com.mx
    Sept. 3 - 6, 2018, Mon.-Thurs. IMEKO World Congress, Belfast, Northern Ireland, www.imeko2018.org
    Oct. 12 - Nov. 9, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com

     

    The URL of this page is www.rasch.org/rmt/rmt94n.htm

    Website: www.rasch.org/rmt/contents.htm