This course addresses the issues of reliability and validity from the perspectives of Classical True-Score Theory (CTT), Generalizability Theory, Structural Equation Modeling, and Rasch Measurement (in particular). Lab sessions will have students develop data analytic skills using software such as Genova, Winsteps/Bigsteps, Facets, RUMM, BMDP, and EQS. Concentration will be on data from polytomous items. Students will present the results of analyses and prepare a proposal for submission to a regional or national conferences.

Reliability:
True Score Theory
Rasch - Reliability, Separation, Strata, Standard Error

Judge awarded scores:
Generalizability Theory
Many-Facet Rasch Measurement

Construct Validity:
True Score Methods
Rasch Based Methods (variable definition, item/person fit, co-calibration)

Dimensionality:
Factor Analysis
Confirmatory Factor Analysis
Rasch fit statistics and residual analysis

Rasch measurement:
Theory
Assumptions
Models

Score reporting and metric development

This course will prepare students with the skills necessary to develop and administer rating scales/questionnaires (emphasis on rating scales designed to measure latent constructs), assess reliability and validity, and analyze and report results. During the course students may develop, administer, analyze, and report the results of their own rating scale/questionnaire. Computer labs will be used to provide examples and experience with data analysis procedures. Students wanting to administer a scale/questionnaire during the semester should receive IRB approval prior to the beginning of the semester. Topics covered include:

1. Criteria for evaluating rating scale research;
   
2. Guidelines for writing items
   
3. Designing response formats and rating scales (use of neutral, grouping items, positive vs. negative, rating points, etc..)
   
4. Data collection strategies;
   
5. Pretesting techniques
   
6. Establishing evidence for reliability and validity using Rasch methods
   
7. Rasch measurement (Rationale, Models, Principles)
   
8. Evaluating the rating scale (category statistics, recoding categories)
   
9. Item evaluation (ordering, fit)
   
10. Measure (person) evaluation (response sets, fit)
    
11. Rater/Judge evaluation (harshness/leniency/halo, etc.., fit)
    
12. Bias evaluation (item bias)
    
13. Measuring change
    
14. Score Reporting and Metric Development (transforming the logit metric)