Reliability and Separation

"The reliability of any set of measurements is logically defined as the proportion of their variance that is true variance... We think of the total variance of a set of measures as being made up of two sources of variance: true variance and error variance... The true measure is assumed to be the genuine value of whatever is being measured... The error components occur independently and at random" (Guilford 1965, p.439-40).

Observed Variance = "True" Variance + Error Variance

Reliability = "True" Variance / Observed Variance

In Rasch terms, "True" valiance is the "adjusted" variance (observed variance adjusted for measurement error). Error Variance is a mean-square error (derived from the model) inflated by misfit to the model encountered in the data.

Kubiszyn and Borich (1993, p.353) say "an acceptable standardized test should have reliability coefficients of about .95 for internal consistency". But "reliability depends upon the population measured as well as the measuring instrument... One should speak of the reliability of a certain instrument applied to a certain population under certain conditions" (Guilford p.439). This is because the "true" variance is a characteristic of the sample tested and the "error" variance is a characteristic of the measuring instrument.

Since reliability is restricted to the range 0 to 1, it is convenient to express it as a separation coefficient with range 0 to infinity:

G = Separation = sqrt(Rel./(1-Rel.)) =
"True" Standard Deviation / Error Standard Deviation

In Rasch literature, the "True" S.D. is often called the "Adjusted" (for measurement error) S.D.

Separation is the number of statistically different performance strata that the test can identify in the sample. This can be pictured by placing an error distribution in each stratum (see Examples). A separation of "2" implies that only two levels of performance can be consistently identified by the test for samples like the one tested. Kubiszyn & Borich's 0.95 corresponds to a separation of 4.5, i.e., 4 consistently identifiable strata. (See RMT 6:3 p. 238 for a Table of Reliability - Separation equivalences). Compare the Separation plot with with a different definition of "ability strata" at Number of Person or Item Strata (4G+1)/3 (RMT 16:3).

Example 1: Separation = 2.0, "True" S.D. = 2.0, Error S.D. = 1.0
Reliability = (2.0*2.0) / (2.0*2.0 + 1.0*.1.0) = 0.8

Example 2:Separation = 3.0, "True" S.D. = 3.0, Error S.D. = 1.0
Reliability = (3.0*3.0) / (3.0*3.0 + 1.0*.1.0) = 0.9
In the graph below, notice that the middle error strata is from -1.5 Error S.D. to +1.5 Error S.D.

Guilford J. P. (1965) Fundamental Statistics in Psychology and Education. 4th Edn. New York: McGraw-Hill.

Kubiszyn T., Borich G. (1993) Educational Testing and Measurement. New York: Harper Collins.

Note: