Factor Analysis versus Rasch Analysis of Items

Factor Analysis versus Rasch Item Analysis of Items

Theory:
	Factor Analysis	Rasch Analysis
T1	Motivation	Hopes to describe data by covariance	Intends to use data for measurement
T2	Model	Ordinal scores mistaken for interval measures which have been observed without error	Ordinal responses modelled as stochastic manifestations of linear parameters, estimated with measurement error
T3	Statistical basis	Covariance matrix of items over examinee	Probability of responses calculated from item and examinee parameters
T4	Outcome of analysis	Factors summarizing the covariance matrix	Operational definition of variable by item calibration and person measure
T5	Interpretation	Factors named to represent correlated items, but with easily disputed meanings	Variable defined by item text manifests the underlying concept. Unexpected outcomes signal misconceptions
T6	Principal statistics	Factor loadings (covariances of item scores with factor). Factor scores (from regressing item scores on loadings).	A linear measure, error and fit statistics, for each item, examinee, and any other element modelled
T7	Largest variance component	The factor with the most variance (largest eigenvalue)	The empirical manifestation of the underlying variable
T8	Other variance components	As many factors as diagonal elements in correlation matrix	Modelled measurement error and unmodelled misfit
T9	Other variance criterion	Factor eigenvalues greater than, usually, 1.4	Standardized misfit statistics greater than, usually, 2.0
T10	Measurement and sampling error	Factors with small eigenvalues confound these sources of error	Measurement error as standard errors, sampling error as standard deviations of examinee and item distributions
T11	Missing data	List-wise deletion loses data. Pair-wise deletion biases factor structure	Measures, standard errors and fit statistics based on all observed data

Diagnosis:
	Factor Analysis	Rasch Analysis
D1	Multi-modal distribution of examinees or items	A factor for each mode, some with large eigenvalue	Documented in item and person measure distributions. No effect on misfit
D2	Irregular examinee response	Slight variance increase. Not determinable from factors	Large misfit for examinee, and somewhat increased misfit for items to which examinee responded unexpectedly
D3	Identification of item bias (DIF, Differential Item Functioning) on many test items	By discovering factor scores correlated with group membership, and then items with loadings on those factors	Exploratory: By discovering items with significant differences between their group measures Confirmatory: By partitioning residuals of suspect items between groups to estimate bias size, significance and homogeneity
D4	Identification of solitary biased item	Undetectable; bias eigenvalues insignificant	As above, plus item misfit (particularly information-weighted (INFIT) statistics)
D5	Major multi-dimensionality (items: 50% math, 50% reading)	After rotation, one math factor and one reading factor	Variable combines math and reading items with low person separation, reliability, and patterns of poor person fit [PCA of Rasch residuals]
D6	Minor multi-dimensionality (items: 95% math, 5% reading)	One math factor; insignificant reading factor	Variable defined by math items, significant misfit in reading items
D7	Miskeyed multiple-choice item	Undetectable; eigenvalues insignificant.	Large item misfit statistic or item calibration contradicts construct

Theory:

Factor Analysis

Rasch Analysis

Motivation

Hopes to describe data by covariance

Intends to use data for measurement

Model

Ordinal scores mistaken for interval measures which have been observed without error

Ordinal responses modelled as stochastic manifestations of linear parameters, estimated with measurement error

Statistical basis

Covariance matrix of items over examinee

Probability of responses calculated from item and examinee parameters

Outcome of analysis

Factors summarizing the covariance matrix

Operational definition of variable by item calibration and person measure

Interpretation

Factors named to represent correlated items, but with easily disputed meanings

Variable defined by item text manifests the underlying concept. Unexpected outcomes signal misconceptions

Principal statistics

Factor loadings (covariances of item scores with factor). Factor scores (from regressing item scores on loadings).

A linear measure, error and fit statistics, for each item, examinee, and any other element modelled

Largest variance component

The factor with the most variance (largest eigenvalue)

The empirical manifestation of the underlying variable

Other variance components

As many factors as diagonal elements in correlation matrix

Modelled measurement error and unmodelled misfit

Other variance criterion

Factor eigenvalues greater than, usually, 1.4

Standardized misfit statistics greater than, usually, 2.0

T10

Measurement and sampling error

Factors with small eigenvalues confound these sources of error

Measurement error as standard errors, sampling error as standard deviations of examinee and item distributions

T11

Missing data

List-wise deletion loses data. Pair-wise deletion biases factor structure

Measures, standard errors and fit statistics based on all observed data

Diagnosis:

Factor Analysis

Rasch Analysis

Multi-modal distribution of examinees or items

A factor for each mode, some with large eigenvalue

Documented in item and person measure distributions. No effect on misfit

Irregular examinee response

Slight variance increase. Not determinable from factors

Large misfit for examinee, and somewhat increased misfit for items to which examinee responded unexpectedly

Identification of item bias (DIF, Differential Item Functioning) on many test items

By discovering factor scores correlated with group membership, and then items with loadings on those factors

Exploratory: By discovering items with significant differences between their group measures
Confirmatory: By partitioning residuals of suspect items between groups to estimate bias size, significance and homogeneity

Identification of solitary biased item

Undetectable; bias eigenvalues insignificant

As above, plus item misfit (particularly information-weighted (INFIT) statistics)

Major multi-dimensionality (items: 50% math, 50% reading)

After rotation, one math factor and one reading factor

Variable combines math and reading items with low person separation, reliability, and patterns of poor person fit
[PCA of Rasch residuals]

Minor multi-dimensionality (items: 95% math, 5% reading)

One math factor; insignificant reading factor

Variable defined by math items, significant misfit in reading items

Miskeyed multiple-choice item

Undetectable; eigenvalues insignificant.

Large item misfit statistic or item calibration contradicts construct

Factor Analysis versus Rasch Analysis of Items, B Wright … Rasch Measurement Transactions, 1991, 5:1 p. 134-135

Rasch Publications
Rasch Measurement Transactions (free, online)	Rasch Measurement research papers (free, online)	Probabilistic Models for Some Intelligence and Attainment Tests, Georg Rasch	Applying the Rasch Model 3rd. Ed., Bond & Fox	Best Test Design, Wright & Stone
Rating Scale Analysis, Wright & Masters	Introduction to Rasch Measurement, E. Smith & R. Smith	Introduction to Many-Facet Rasch Measurement, Thomas Eckes	Invariant Measurement: Using Rasch Models in the Social, Behavioral, and Health Sciences, George Engelhard, Jr.	Statistical Analyses for Language Testers, Rita Green
Rasch Models: Foundations, Recent Developments, and Applications, Fischer & Molenaar	Journal of Applied Measurement	Rasch models for measurement, David Andrich	Constructing Measures, Mark Wilson	Rasch Analysis in the Human Sciences, Boone, Stave, Yale
in Spanish:	Análisis de Rasch para todos, Agustín Tristán	Mediciones, Posicionamientos y Diagnósticos Competitivos, Juan Ramón Oreja Rodríguez

Rasch Publications

Rasch Measurement Transactions (free, online)

Rasch Measurement research papers (free, online)

Probabilistic Models for Some Intelligence and Attainment Tests, Georg Rasch

Applying the Rasch Model 3rd. Ed., Bond & Fox

Best Test Design, Wright & Stone

Rating Scale Analysis, Wright & Masters

Introduction to Rasch Measurement, E. Smith & R. Smith

Introduction to Many-Facet Rasch Measurement, Thomas Eckes

Invariant Measurement: Using Rasch Models in the Social, Behavioral, and Health Sciences, George Engelhard, Jr.

Statistical Analyses for Language Testers, Rita Green

Rasch Models: Foundations, Recent Developments, and Applications, Fischer & Molenaar

Journal of Applied Measurement

Rasch models for measurement, David Andrich

Constructing Measures, Mark Wilson

Rasch Analysis in the Human Sciences, Boone, Stave, Yale

in Spanish:

Análisis de Rasch para todos, Agustín Tristán

Mediciones, Posicionamientos y Diagnósticos Competitivos, Juan Ramón Oreja Rodríguez

Forum

Rasch Measurement Forum to discuss any Rasch-related topic

Go to Institute for Objective Measurement Home Page. The Rasch Measurement SIG (AERA) thanks the Institute for Objective Measurement for inviting the publication of Rasch Measurement Transactions on the Institute's website, www.rasch.org.

Coming Rasch-related Events
Aug. 11 - Sept. 8, 2023, Fri.-Fri.	On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com
Aug. 29 - 30, 2023, Tue.-Wed.	Pacific Rim Objective Measurement Society (PROMS), World Sports University, Macau, SAR, China https://thewsu.org/en/proms-2023
Oct. 6 - Nov. 3, 2023, Fri.-Fri.	On-line workshop: Rasch Measurement - Core Topics (E. Smith, Facets), www.statistics.com
June 12 - 14, 2024, Wed.-Fri.	1st Scandinavian Applied Measurement Conference, Kristianstad University, Kristianstad, Sweden http://www.hkr.se/samc2024