Five Steps to Science:
Observing, Scoring, Measuring, Analyzing, and Applying

Science progresses by dialogue between the inner world of abstract ideas and the outer word of concrete experience. Ideas are hypothetical guidelines. Experience brings them to life.

Experience is tangible. But it needs ideas to become useful. Raw experience is chaotic. Guidelines are necessary to organize perceptions of reality, to make them recognizable, and to make some sense of them.

Science is the conversation between ideas and experience. We require ideas to recognize reality. We require experience to nudge ideas into new shapes. This dialogue is the relationship between ideas and experience out of which all knowledge evolves. This is the method of science. Science is not facts but method, not data but interpretation of data, not just experience but also idea.

The scientific tasks of psychometrics are pursued by working through a dialogue of five successive procedural models. Articulating these models enables focus and reduces confusion. Explicating a model for each step focuses attention on the specific problem addressed at that step.

1. The Observing Model sets the standards for data production and established the first level of quality control. When Isherwood (1939) imagines, "I am a camera with its shutter open, quite passive, recording, not thinking," he supposes that we are able to observe without thinking. But to observe and record is to think about what to observe and what to record. Isherwood's camera is pointed in a direction, has a focal length, and responds to a particular wave band. We are unable merely to observe.

To make observations is to select what to attend. Even more important, to observe is to select what to not attend. Data does not exist of itself. It must be conceived and produced. To produce data requires deciding what to address, when and how. Data is not found. It is anticipated in imagination and constructed in action. Like any product, its manufacture needs quality control.

Quality control over data production means continuous monitoring. The Observing Model nominates the qualities to be sought and recorded. It specifies what is to be looked for and what is to be counted.

In psychometrics, the response requirements for collecting data influence item difficulty, e.g. double negatives, reversals, conditionals. Sometimes item intention is so overwhelmed by an ambiguous response process that we do not obtain a reproducible item difficulty, only a variety of local difficulties provoked by interactions between the eccentricities of respondents and the peculiarities of the item.

2. The Scoring Model addresses the observed data as though it were nothing but comparisons of ordered alternatives like 0/1 for dichotomous responses and 0/1/2/3 ... for ranks and rating scales. The stochastic model by which these data are given inferential meaning is based on defining transition odds for successive categories in terms of a few conjointly additive parameters. To proceed in this direction it is necessary to determine which of the various ordinal scorings the particular categories offer is most useful for measurement, which scoring format provides the most information. The Scoring Model chosen specifies the more and less comparisons used to infer measurement from the observations (Wright & Stone, 1996).

3. All useful Measuring Models, however complicated appearing, reduce to a Rasch formulation, an equation connecting logs of category transition odds for observable events to conjointly additive parameters designed to explain these odds (Rasch, 1960, 1980; Wright & Stone, 1979). The stochastic model interprets the data as instances of probabilities.

The simplest Rasch model identifies parameters B_n for person ability and D_i for item difficulty. Their difference ( B_n - D_i ) is defined to govern the probability of what is expected to happen when person n uses their ability B_n against the difficulty D_i of item i. The data are interpreted as independent of the distribution of the other B_n, and the measures of B_n are independent of the distributions of D_i. The log-odds function establishes a linear scale and the parameter separation establishes generality (Wright & Stone 1979).

a) Person separation indicates the ability of the items to separate measures of these persons.

b) Item separation indicates the degree to which a variable has been defined by these persons.

c) Item fit evaluates the relevance of each item to the conjoint variable.

d) Person fit evaluates the validity of each person measure and directs response diagnosis.

4. The Analyzing Model cannot do its work until we have satisfied the requirements of the first three models. Premature analysis of data mistaken for measures without taking into account and using the models for observing, scoring and measuring only confounds results with an inextricable mass of unidentified and uncontrolled interactions.

In the Analyzing Model we study process and relation. We determine the implications of the measures we derive from our observations and investigate how these measures relate to other variables also measured by applying the observing, scoring and measuring models.

5. The Applying Model follows analysis of the measures constructed from observations. In this step, we apply the results obtained to the problems that initially provoked our investigations and also to new situations. The Applying Model brings the prior steps into focus and use. It orients the prior models to an outcome.

The scientific productivity of the five models depends on the vitality of their stepwise reconciliation of idea and experience. The models articulate a dialogue proceeding forwards and backwards as we apply what has been clarified by one model to expediting the tasks of another. Quality control and continuous monitoring is essential.

The organizer that integrates the five models is the MAP of the Variable. The MAP begins an idea about experience, an expectation, and a plan. The results from applying the models are incorporated in the MAP. The MAP coordinates and explains the idea by illustration, conceptually and experientially. The MAP portrays the status of results achieved, pictures what has been accomplished and identifies what remains to be done. Successful mapping brings ideas and experience together in a visual manifestation and synthesis of the dialectic process.

Benjamin D. Wright & Mark H. Stone, May 2003 (original paper 1996)

Isherwood, D. (1939) I am a camera. New York: Random House. Rasch, G. (1980). Probabilistic models for some intelligence and attainment tests. Chicago: The University of Chicago Press. (Original work published in 1960)

Wright, B. D. & Stone, M. H. (1979). Best test design. Chicago: MESA Press.

Wright, B. D. & Stone, M. H. (1996). Measurement essentials. Wilmington, DE: Wide Range, Inc.

Five Steps to Science: Observing, Scoring, Measuring, Analyzing, and Applying. B.D. Wright & M.H. Stone … Rasch Measurement Transactions, 2003, 17:1, 912-913.

Rasch Books and Publications
Invariant Measurement: Using Rasch Models in the Social, Behavioral, and Health Sciences, 2nd Edn. George Engelhard, Jr. & Jue Wang	Applying the Rasch Model (Winsteps, Facets) 4th Ed., Bond, Yan, Heene	Advances in Rasch Analyses in the Human Sciences (Winsteps, Facets) 1st Ed., Boone, Staver	Advances in Applications of Rasch Measurement in Science Education, X. Liu & W. J. Boone	Rasch Analysis in the Human Sciences (Winsteps) Boone, Staver, Yale
Introduction to Many-Facet Rasch Measurement (Facets), Thomas Eckes	Statistical Analyses for Language Testers (Facets), Rita Green	Invariant Measurement with Raters and Rating Scales: Rasch Models for Rater-Mediated Assessments (Facets), George Engelhard, Jr. & Stefanie Wind	Aplicação do Modelo de Rasch (Português), de Bond, Trevor G., Fox, Christine M	Appliquer le modèle de Rasch: Défis et pistes de solution (Winsteps) E. Dionne, S. Béland
Exploring Rating Scale Functioning for Survey Research (R, Facets), Stefanie Wind	Rasch Measurement: Applications, Khine	Winsteps Tutorials - free Facets Tutorials - free	Many-Facet Rasch Measurement (Facets) - free, J.M. Linacre	Fairness, Justice and Language Assessment (Winsteps, Facets), McNamara, Knoch, Fan
Other Rasch-Related Resources: Rasch Measurement YouTube Channel
Rasch Measurement Transactions & Rasch Measurement research papers - free	An Introduction to the Rasch Model with Examples in R (eRm, etc.), Debelak, Strobl, Zeigenfuse	Rasch Measurement Theory Analysis in R, Wind, Hua	Applying the Rasch Model in Social Sciences Using R, Lamprianou	El modelo métrico de Rasch: Fundamentación, implementación e interpretación de la medida en ciencias sociales (Spanish Edition), Manuel González-Montesinos M.
Rasch Models: Foundations, Recent Developments, and Applications, Fischer & Molenaar	Probabilistic Models for Some Intelligence and Attainment Tests, Georg Rasch	Rasch Models for Measurement, David Andrich	Constructing Measures, Mark Wilson	Best Test Design - free, Wright & Stone Rating Scale Analysis - free, Wright & Masters
Virtual Standard Setting: Setting Cut Scores, Charalambos Kollias	Diseño de Mejores Pruebas - free, Spanish Best Test Design	A Course in Rasch Measurement Theory, Andrich, Marais	Rasch Models in Health, Christensen, Kreiner, Mesba	Multivariate and Mixture Distribution Rasch Models, von Davier, Carstensen

Forum

Rasch Measurement Forum to discuss any Rasch-related topic

Go to Top of Page
Go to index of all Rasch Measurement Transactions
AERA members: Join the Rasch Measurement SIG and receive the printed version of RMT
Some back issues of RMT are available as bound volumes
Subscribe to Journal of Applied Measurement

Go to Institute for Objective Measurement Home Page. The Rasch Measurement SIG (AERA) thanks the Institute for Objective Measurement for inviting the publication of Rasch Measurement Transactions on the Institute's website, www.rasch.org.

The URL of this page is www.rasch.org/rmt/rmt171j.htm

Website: www.rasch.org/rmt/contents.htm