September 1997
Quick Takes: Using the TIMSS Report
Commentators frequently employ international comparisons of student performances to highlight strengths and weaknesses of the U.S. educational system. The latest international test to come to the attention of the U.S. public is the Third International Mathematics and Science Study (TIMSS). How can TIMSS data provide information that will help teachers, administrators, and parents improve schools?
What is the TIMSS?
Sponsored by a cooperative of independent research centers, TIMSS is a comprehensive ongoing comparison of the mathematics and science knowledge and school experience of students at specified ages in many countries. Data were collected using 31 languages (the language of the test in the United States was English). Although student knowledge and skills were tested, TIMSS is more than an international standardized test. The study investigated curricula, instruction, educational policies, and other aspects of education; researchers also considered written tests, interviews, and videotapes of classrooms in constructing this international overview of mathematics and science education.
The first TIMSS report, released in November 1996, covered the achievements of 13-year-olds (U.S. eighth graders) in 41 countries. The second report, on nine-year-olds (U.S. fourth graders) in 26 countries, was released in June 1997, and the final report, which will cover students in their last year of secondary school, will be released in the spring of 1998. To avoid testing only the best students in a country, schools that participated in the study were randomly selected.
Can we compare students using TIMSS data?
Most popular reports of TIMSS have emphasized the simple ranking of the average scores by country to compare student performance. TIMSS investigators have warned against this practice since the data are estimates of ranges that, from a statistical point of view, likely include each country's real score. If national student performances must be compared, similarities and divergences should be talked about in terms of groups of nations that can be ranked together because their scores are statistically similar. Thus, students in some groups of countries can be said to have performed significantly better than, about the same as, or significantly worse than students in other groups of countries. Each individual country's average score can also be compared to the international average.
Eighth Grade (41 countries). U.S. eighth graders did not perform as well in mathematics and science as did students in many other countries. In science, thirteen year olds in nine countries scored higher than U.S. eighth graders and children in 16 countries performed statistically the same as U.S. eighth graders. U.S. students outperformed students from 15 countries. The average score of U.S. eighth graders in science was significantly higher than the international average.
U.S. eighth graders did not perform as well in mathematics as they did in science. Students in 20 countries performed better than U.S. students and those in 13 were in the same statistical group. U.S. eighth graders did better than students in seven countries. The U.S. mathematics average score is not statistically different from the international average score.
Fourth Grade (26 countries). U.S. fourth graders performed better on their test when compared with their peers in other countries than did U.S. eighth graders when compared with their international peers. In science only Korean nine year olds scored significantly better than the U.S. fourth graders, who tied with Japan for second place. U.S. fourth graders did better in science than students in 19 countries; their average score was significantly above the international average. On the mathematics exam, students in seven countries had average scores above those in the United States; the averages of students in six countries were within the same ranking as those of U.S. fourth graders, who outperformed students in 12 countries.
Can TIMSS identify why some students do better than others?
The short answer to this question is no - no matter whether you are comparing students in different countries or students in the same country but different grades, there is no simple answer to the question of why certain students do better. Over the years many commentators have tried to find a simplistic key to predicting student achievement. So far TIMSS data support none. Although the TIMSS researchers looked for patterns involving many factors related to student performance, they found no single factor or group of factors that consistently account for large differences in student achievement.
For example, some observers claim too much television watching and insufficient homework assignments negatively influence U.S. test scores. Yet TIMSS data show that U.S. and Japanese students spend about the same time watching television. U.S. eighth graders also receive more homework assignments and spend more time in class talking about their homework than Japanese or German students - and more than U.S. fourth-graders.
The same can be said about other simple solutions offered for educational issues - more time in class (U.S. students spend more time in class than students in many countries with higher TIMSS scores), more formal schooling for teachers (U.S. teachers had more class hours in their content specialties than teachers in some higher-scoring countries). The TIMSS data strongly support the idea that quick and easy answers will not solve long-standing questions. To understand why some students perform better than others, we must thoughtfully re-examine our individual educational practices at the local level.
How can we use these data in our schools?
The lack of simple answers from TIMSS forces us to think attentively
about how children learn and are taught. We cannot reduce the
complex social activity of classroom interaction to simple rules
or cookbook answers that work for everyone. With TIMSS data
as a starting point, teachers, parents, students, and administrators
can reflect thoughtfully on their educational assumptions and
habits and begin to clarify their understanding of the forces
that shape children's learning. Instead of reading TIMSS as
a report on winners and losers, we should use it as a mirror
to investigate how each of us contributes to the education of
all our children. Approached in this spirit, TIMSS can become
a resource and guide to answering such complicated questions
as why U.S. students seem to lose interest and aptitude in mathematics
and science between the fourth and eighth grades.
One afternoon Pat and Chris were discussing their students' performances on the recent state mathematics tests. Pat maintained that student homework suffered at the expense of television watching and other outside activities. "If kids watched less TV and did more homework their grades on these tests would go up overnight."
"I don't know, Pat. What are the kids going to do but watch TV? A lot of them don't even have a good place to study. I know they don't have computers at home and very few have calculators. Things like that probably affect their scores more than TV watching," Chris said.
The principal overheard this conversation and talc them she had some reference materials she thought could help answer their questions: "You know about the Third International Mathematics and Science Study? Well, I have a set of books in my office that presents the data on the whole thing. You're welcome to use them any time."
Sensing a challenge in the principal's invitation, the two teachers asked the office secretary for the volumes and soon found themselves lugging four fat books to the office they shared. The four books covered reports an 1 3-year-old and nine-year-old students in mathematics and science from across a broad spectrum of countries. "Oh yeah," Pat said, "I remember reading about this stuff in the paper last spring. Seems like the U.S. did real bad."
"I remember that it was more of a mixed bag," Chris said, "Seems like we did poorly in some things and not in others." Curious, they began flipping through the books.
"Look, see here in this table," Pat said, "17 percent of U.S. fourth graders watch more than five hours of television on a school day. No fourth graders should be watching that much TV." But then Pat began to look at the rest of the table and noticed that in other categories Japanese and Korean students actually watched more television than U.S. students. "What do you think that means?" Pat asked.
"I don't know." Chris replied, "What does the eighth grade stuff say?" The two teachers soon found that the trend was the same among the older students: While a relatively high percentage of the U. S. eighth graders watched five hours or more of television each day, students who watched a lot were common in Japan and Korea also.
"And hey," Pat said, "didn't the fourth graders score higher than the eighth graders in math and science? But look 17 percent of fourth graders watched more than five hours of TV and only 13 percent of eighth graders did. Maybe the fourth graders were all watching Sesame Street?"
As the two teachers went through their list of possible causes for poor performance, they found more surprises and fewer correlations. The data covered many factors the teachers had labeled as causes, such as amount of homework assigned, calculator use, and study aids available in the home. Each time the teachers followed a thread through the tables, however, they became more puzzled. The data offered more questions than conclusions; no one thing or group of things could be fixed as the cause of performance.
Pat and Chris, like many readers of the TIMSS, began to use the data to start conversations about teaching and learning with their colleagues and others concerned about the students' performance. To increase their understanding, they contacted the TIMSS Study (enter in the School of Education at Boston College (Chestnut Hill MA 02167) and went to various TIMSS trainings in their region. Soon they realized that they had been viewing the TIMSS as an international academic Olympics. Being number one was the most important thing. Pat and Chris began to think that the TIMSS could be much more powerful as an aid ta reaching better understanding of the complexities of teaching and learning. Improving student performance, they concluded, would require thoughtful reconsideration of the educational beliefs and practices of everyone in the system. TIMSS could not give simple answers but it did provide a good place to start a conversation.
For more information, try the TIMSS web page: http://wwwcsteep.bc.edu/timss
About Quick Takes
Quick Takes was a publication of the Eisenhower Southwest Consortium for the Improvement of Mathematics and Science Teaching (SCIMAST) project, sponsored by the U.S. Department of Education under grant number R168R50027-95. The content herein does not necessarily reflect the views of the department or any other agency of the U.S. government or any other source. Available in alternative formats.The Eisenhower SCIMAST project was located in the Southwest Educational Development Laboratory (SEDL) at 211 East Seventh Street, Austin, Texas 78701; (512)476-6861/(800)201-7435. SEDL is an Equal Employment Opportunity/Affirmative Action Employer and is committed to affording equal employment opportunities to all individuals in all employment matters.