Toward Practical Significance of Rasch Scores in International Studies in Education:
More than Statistical Significance and Effect Size

Both in social sciences and in education, the scales that are used can easily and accurately lead to statistical significance, calculation of effect size indices, or computation of the confidence interval . However, unfortunately, practical significance of these scores can scarcely be obtained. For example, what does it mean to a teacher to be two points happier with his job than another one on a Likert scale? Almost worse: What does it mean to a school to be compared to the average national results in mathematics? Just to say that there is a statistical significant difference between the school score and the national average scores is not very informative for the school.

Statisticians proposed a compromise to these kinds of situations. For this purpose, they suggest using effect size indices and confidence intervals in replacement of significance tests. They are interesting devices, though not as much when it comes to the practical significance of scores and to the difference between these scores.

In the context of comparisons made with the results of international and national surveys in education, like the TEIMS, PIRLS and PISA, practical significance of these scores would lead researchers to more useful interpretations. It would lead to more meaningful comparisons of countries and states. Important decisions would be easier to make.

In the context of these international surveys, researchers tried to find a way to lead to practical significance of the obtained mathematics, science, or reading literacy scores. For example, Cartwright, Lalancette, Mussio, and Xing (2003) related these international scores to results at a smaller provincial scale in Canada. They found that the provincial results in reading of British Columbia can be related with a good precision to the international PISA reading comprehension literacy score. While these findings don't permit a direct practical interpretation, they tell us, at least, that it is possible to link results obtained at one survey, the PISA, to those obtained at another one, the British-Columbia.

Figure 1. Relation between Rasch and TEIMS 2003 scores, schooling days and number of books at home

More interesting, and more practical, is Willms' proposition (2004a, 2004b). Willms tries to translate the international Rasch scores results from PISA into a metric of school years and school days. This is an attractive idea, because it leads directly to a metric that allows administrative decision and teaching effect interpretation. Thus, it would be legitimate to say that to reach an augmentation of one point on one of PISA literacy scales, a fixed number of schooling days would be necessary. Willms, in preliminary works, finds out that a difference of one point on the PISA scale is equivalent to about 5 schooling days. He didn't report on which literacy scale the equivalence was computed. He did this estimation by considering the fact that, because of the annual inscription date in each state and country, the 15-years-old students participating in the survey can be on two different school levels. This fact permits him to compute the effect of one schooling year on the PISA scores. The average schooling year in the 12 countries, where he was able to obtain the information about the level and the date of inscription, was equal to 172 days. Wherefore he observed that a schooling year correspond to a difference score of 34.30, he extrapolates that a one point difference is equivalent to about 5 schooling days (172/34.30).

By themselves, Willms' findings are of interest, being now able to give a clear interpretation of differences in literacy scores. However, if we also consider Cartwright et al., we can think that the equivalence formula obtained by Willms can be related to the one of other international and national educational surveys. Consider that the PISA scores are on a scale with a mean of 500 and a standard deviation of 100 and that, for example, the TEIMS Rasch scale has a mean of 150 and a standard deviation of 10. Thus, only using the standard deviation ratio, a one point difference at the TEIMS scale would be equivalent to a 10 points difference at the PISA scale: sPISA / sTIMSS = 100/10. An illustration of the relation between TIMSS 2003 scores, schooling days, and number of books at home is presented in Figure 1. It can be seen that a mean difference of 32 books at home (37 - 5) corresponds approximately to one school year (178.50 days).

A note of caution is to be considered. More research work is still to be done on this topic and it will have to consider the specific literacy scale, how old the students are, as well as their school level. More important, however, we think that the equivalence between the schooling days and the difference in the literacy scores would have to be computed independently for each country, not by an average on the 12 countries used by Willms which are so different in schooling practice.

Gilles Raîche, Université du Québec à Montréal
Claire IsaBelle, Université d'Ottawa
Martin Riopel, Université du Québec à Montréal
Cartwright, F., Lalancette, D., Mussio, J. and Xing, D. (2003). Linking provincial student assessments with national and international assessments. Report no 81-595-MIE2003005. Ottawa, Canada: Statistics Canada.
Willms, J. D. (2004a). Reading achievement in Canada and the United States : Findings from the OECD programme for international student assessment. Final Report no SP-601-05-04E. Ottawa, Canada: Human Resources and Skills Development Canada.
Willms, J. D. (2004b). Variation in literacy skills among Canadian provinces: Findings from the OECD PISA. Report no 81-595-MIE2004012. Ottawa, Canada: Statistics Canada.


Toward Practical Significance of Rasch Scores in International Studies in Education: More than Statistical Significance and Effect Size, Raîche G., IsaBelle C., Riopel M … Rasch Measurement Transactions, 2006, 20:1 p. 1046-7



Rasch Publications
Rasch Measurement Transactions (free, online) Rasch Measurement research papers (free, online) Probabilistic Models for Some Intelligence and Attainment Tests, Georg Rasch Applying the Rasch Model 3rd. Ed., Bond & Fox Best Test Design, Wright & Stone
Rating Scale Analysis, Wright & Masters Introduction to Rasch Measurement, E. Smith & R. Smith Introduction to Many-Facet Rasch Measurement, Thomas Eckes Invariant Measurement: Using Rasch Models in the Social, Behavioral, and Health Sciences, George Engelhard, Jr. Statistical Analyses for Language Testers, Rita Green
Rasch Models: Foundations, Recent Developments, and Applications, Fischer & Molenaar Journal of Applied Measurement Rasch models for measurement, David Andrich Constructing Measures, Mark Wilson Rasch Analysis in the Human Sciences, Boone, Stave, Yale
in Spanish: Análisis de Rasch para todos, Agustín Tristán Mediciones, Posicionamientos y Diagnósticos Competitivos, Juan Ramón Oreja Rodríguez

To be emailed about new material on www.rasch.org
please enter your email address here:

I want to Subscribe: & click below
I want to Unsubscribe: & click below

Please set your SPAM filter to accept emails from Rasch.org

www.rasch.org welcomes your comments:

Your email address (if you want us to reply):

 

ForumRasch Measurement Forum to discuss any Rasch-related topic

Go to Top of Page
Go to index of all Rasch Measurement Transactions
AERA members: Join the Rasch Measurement SIG and receive the printed version of RMT
Some back issues of RMT are available as bound volumes
Subscribe to Journal of Applied Measurement

Go to Institute for Objective Measurement Home Page. The Rasch Measurement SIG (AERA) thanks the Institute for Objective Measurement for inviting the publication of Rasch Measurement Transactions on the Institute's website, www.rasch.org.

Coming Rasch-related Events
May 17 - June 21, 2024, Fri.-Fri. On-line workshop: Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 12 - 14, 2024, Wed.-Fri. 1st Scandinavian Applied Measurement Conference, Kristianstad University, Kristianstad, Sweden http://www.hkr.se/samc2024
June 21 - July 19, 2024, Fri.-Fri. On-line workshop: Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
Aug. 5 - Aug. 6, 2024, Fri.-Fri. 2024 Inaugural Conference of the Society for the Study of Measurement (Berkeley, CA), Call for Proposals
Aug. 9 - Sept. 6, 2024, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com
Oct. 4 - Nov. 8, 2024, Fri.-Fri. On-line workshop: Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
Jan. 17 - Feb. 21, 2025, Fri.-Fri. On-line workshop: Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
May 16 - June 20, 2025, Fri.-Fri. On-line workshop: Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 20 - July 18, 2025, Fri.-Fri. On-line workshop: Rasch Measurement - Further Topics (E. Smith, Facets), www.statistics.com
Oct. 3 - Nov. 7, 2025, Fri.-Fri. On-line workshop: Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com

 

The URL of this page is www.rasch.org/rmt/rmt201b.htm

Website: www.rasch.org/rmt/contents.htm