Reliability and Separation

"The reliability of any set of measurements is logically defined as the proportion of their variance that is true variance... We think of the total variance of a set of measures as being made up of two sources of variance: true variance and error variance... The true measure is assumed to be the genuine value of whatever is being measured... The error components occur independently and at random" (Guilford 1965, p.439-40).

Observed Variance = "True" Variance + Error Variance

Reliability = "True" Variance / Observed Variance

In Rasch terms, "True" valiance is the "adjusted" variance (observed variance adjusted for measurement error). Error Variance is a mean-square error (derived from the model) inflated by misfit to the model encountered in the data.

Kubiszyn and Borich (1993, p.353) say "an acceptable standardized test should have reliability coefficients of about .95 for internal consistency". But "reliability depends upon the population measured as well as the measuring instrument... One should speak of the reliability of a certain instrument applied to a certain population under certain conditions" (Guilford p.439). This is because the "true" variance is a characteristic of the sample tested and the "error" variance is a characteristic of the measuring instrument.

Since reliability is restricted to the range 0 to 1, it is convenient to express it as a separation coefficient with range 0 to infinity:

G = Separation = sqrt(Rel./(1-Rel.)) =
"True" Standard Deviation / Error Standard Deviation

In Rasch literature, the "True" S.D. is often called the "Adjusted" (for measurement error) S.D.

Separation is the number of statistically different performance strata that the test can identify in the sample. This can be pictured by placing an error distribution in each stratum (see Examples). A separation of "2" implies that only two levels of performance can be consistently identified by the test for samples like the one tested. Kubiszyn & Borich's 0.95 corresponds to a separation of 4.5, i.e., 4 consistently identifiable strata. (See RMT 6:3 p. 238 for a Table of Reliability - Separation equivalences). Compare the Separation plot with with a different definition of "ability strata" at Number of Person or Item Strata (4G+1)/3 (RMT 16:3).

Example 1: Separation = 2.0, "True" S.D. = 2.0, Error S.D. = 1.0
Reliability = (2.0*2.0) / (2.0*2.0 + 1.0*.1.0) = 0.8

Separation of 2


Example 2:Separation = 3.0, "True" S.D. = 3.0, Error S.D. = 1.0
Reliability = (3.0*3.0) / (3.0*3.0 + 1.0*.1.0) = 0.9
In the graph below, notice that the middle error strata is from -1.5 Error S.D. to +1.5 Error S.D.

Separation of 3


Guilford J. P. (1965) Fundamental Statistics in Psychology and Education. 4th Edn. New York: McGraw-Hill.

Kubiszyn T., Borich G. (1993) Educational Testing and Measurement. New York: Harper Collins.

Note:

  • For separation G, the levels in the true distribution are 3*"True S.D."/G apart, centered on the sample mean.
    The levels in the observed distribution are 3*"Observed S.D."/G apart, centered on the sample mean.


    1. Reliability, separation, strata statistics, Fisher WP Jr. … 6:3 p.238
    2. Reliability and separation nomograms, Linacre JM. … 1995, 9:2 p.421
    3. Reliability and separation, Wright BD. … 1996, 9:4 p.472
    4. Predicting Reliabilities and Separations of Different Length Tests, Linacre, J.M. … 2000, 14:3 p.767
    5. Going beyond Unreliable Reliabilities, Mallinson T., Stelmack J. … 2001, 14:4 p.787-8
    6. Separation, Reliability and Skewed Distributions: Statistically Different Levels of Performance, Wright B.D. … 2001, 14:4 p.786
    7. Number of Person or Item Strata (4G+1)/3, Wright BD, Masters GN. … 2002, 16:3 p.888
    8. Cash value of Reliability, WP Fisher … Rasch Measurement Transactions, 2008, 22:1 p. 1160

    Reliability and separation. Wright BD. … Rasch Measurement Transactions, 1996, 9:4 p.472

    Please help with Standard Dataset 4: Andrich Rating Scale Model



    Rasch Publications
    Rasch Measurement Transactions (free, online) Rasch Measurement research papers (free, online) Probabilistic Models for Some Intelligence and Attainment Tests, Georg Rasch Applying the Rasch Model 3rd. Ed., Bond & Fox Best Test Design, Wright & Stone
    Rating Scale Analysis, Wright & Masters Introduction to Rasch Measurement, E. Smith & R. Smith Introduction to Many-Facet Rasch Measurement, Thomas Eckes Invariant Measurement: Using Rasch Models in the Social, Behavioral, and Health Sciences, George Engelhard, Jr. Statistical Analyses for Language Testers, Rita Green
    Rasch Models: Foundations, Recent Developments, and Applications, Fischer & Molenaar Journal of Applied Measurement Rasch models for measurement, David Andrich Constructing Measures, Mark Wilson Rasch Analysis in the Human Sciences, Boone, Stave, Yale
    in Spanish: Análisis de Rasch para todos, Agustín Tristán Mediciones, Posicionamientos y Diagnósticos Competitivos, Juan Ramón Oreja Rodríguez

    To be emailed about new material on www.rasch.org
    please enter your email address here:

    I want to Subscribe: & click below
    I want to Unsubscribe: & click below

    Please set your SPAM filter to accept emails from Rasch.org

    www.rasch.org welcomes your comments:

    Your email address (if you want us to reply):

     

    ForumRasch Measurement Forum to discuss any Rasch-related topic

    Go to Top of Page
    Go to index of all Rasch Measurement Transactions
    AERA members: Join the Rasch Measurement SIG and receive the printed version of RMT
    Some back issues of RMT are available as bound volumes
    Subscribe to Journal of Applied Measurement

    Go to Institute for Objective Measurement Home Page. The Rasch Measurement SIG (AERA) thanks the Institute for Objective Measurement for inviting the publication of Rasch Measurement Transactions on the Institute's website, www.rasch.org.

    Coming Rasch-related Events
    July 30-31, 2016, Sat.-Sun. PROMS 2016 Pre-Conference Workshop, Xi'an, China, confchina.com
    Aug. 1-3, 2016, Mon.-Wed. PROMS 2016 Conference, Xi'an, China, confchina.com
    Aug. 1 - Nov. 25, 2016, Mon.-Fri. On-line course: Introduction to Classical and Rasch Measurement Theories EDU5638 (D. Andrich, RUMM2030), www.education.uwa.edu.au
    Aug. 3-5, 2016, Wed.-Fri. IMEKO 2016 TC1-TC7-TC13 Joint Symposium, Berkeley, CA, imeko-tc7-berkeley-2016.org
    Aug. 12 - Sept. 9, 2016, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com
    Sept. 2 - Oct. 14, 2016, Fri.-Fri. On-line workshop: Rasch Applications, Part 1: How to Construct a Rasch Scale (W.P. Fisher, Jr.), www.statistics.com
    Sept. 28-30, 2016, Wed.-Fri. In-person workshop: Introductory Rasch (M. Horton, RUMM), Leeds, UK, www.leeds.ac.uk/medicine/rehabmed/psychometric
    Oct. 3-5, 2016, Mon.-Wed. In-person workshop: Intermediate Rasch (M. Horton. Tennant, RUMM), Leeds, UK, www.leeds.ac.uk/medicine/rehabmed/psychometric
    Oct. 6-7, 2016, Thur.-Fri. In-person workshop: Advanced Rasch (M. Horton, RUMM), Leeds, UK, www.leeds.ac.uk/medicine/rehabmed/psychometric
    Oct. 14 - Nov. 11, 2016, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
    Nov. 11, 2016, Fri. In-person workshop: 11th International Workshop on Rasch Models in Business Administration, Tenerife, Spain, www.ull.es
    Dec. 7-9, 2016, Wed.-Fri. In-person workshop: Introductory Rasch (M. Horton, RUMM), Leeds, UK, www.leeds.ac.uk/medicine/rehabmed/psychometric
    Jan. 6 - Feb. 3, 2017, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
    May 26 - June 23, 2017, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
    June 30 - July 29, 2017, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
    Aug. 11 - Sept. 8, 2017, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com
    Aug. 18-21, 2017, Fri.-Mon. IACAT 2017: International Association for Computerized Adaptive Testing, Niigata, Japan, iacat.org
    Oct. 13 - Nov. 10, 2017, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
    Jan. 5 - Feb. 2, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
    Jan. 10-16, 2018, Wed.-Tues. In-person workshop: Advanced Course in Rasch Measurement Theory and the application of RUMM2030, Perth, Australia (D. Andrich), Announcement
    Jan. 17-19, 2018, Wed.-Fri. Rasch Conference: Matilda Bay Club, Perth, Australia, Website
    May 25 - June 22, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
    June 29 - July 27, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
    Aug. 10 - Sept. 7, 2018, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com
    Oct. 12 - Nov. 9, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
    The HTML to add "Coming Rasch-related Events" to your webpage is:
    <script type="text/javascript" src="http://www.rasch.org/events.txt"></script>

    The URL of this page is www.rasch.org/rmt/rmt94n.htm

    Website: www.rasch.org/rmt/contents.htm