PISA 2018: Measuring Progress Globally
Programme for International Student Assessment
By: Evan Rhinesmith, Ph.D.
The most recent Programme for International Student Assessment (PISA) results are in and we can see how 15-year-olds in the U.S. performed relative to similar students in 78 other countries.
What does the PISA assess?
The PISA exam occurs every three years with the goal of determining what knowledge 15-year-olds worldwide have acquired and how well they can apply what they have learned. The test is administered by the Organisation for Economic Co-operation and Development (OECD), but participants do not need to be OECD members and participation is voluntary. Students are assessed in reading, math, and science.
A unique aspect of the PISA exam is that students are asked to interpret texts, solve math problems, provide scientific explanations of different phenomena. Additionally, students are assessed on one other domain that varies with each iteration of the assessment. For the most recent iteration, U.S. students were assessed on financial literacy. Scores on the PISA exam range from 0 to 1,000, with a standard deviation of 100 points and score ranges providing proficiency levels. Higher scores indicate greater levels of proficiency and the ability to apply knowledge to more complex tasks.
Who takes the PISA exam?
The PISA assesses only 15-year-olds in countries that volunteer to participate in the test. Similar to the U.S. NAEP assessment, only a sample of students participate in the exam. The test is given every three years and has only been in existence since the year 2000.
Roughly 600,000 15-year-olds took the PISA in the 79 countries that participated in 2018. These students are meant to be representative of about 32 million similar-aged students in the participating countries. In the U.S. 4,838 students in 215 schools nationwide took the test.
What is the point of PISA?
Once a country is approved to participate, individual schools are chosen using a list of criteria to ensure the schools are representative of all 15-year-olds in the country. Once schools are selected, students in each school are randomly selected to participate and take the test in the language used in their classroom. Participating students never see their individual performance reports, but their performance is used to calculate a nationwide average that provides a snapshot of how 15-year-olds in that country perform.
According to PISA, the purpose of the exam is not just to provide a ranking of education performance. Rather, it is to show the following:
• Where school systems are more/less effective at preparing students for work and further education
• Where 15-year-olds are better acquiring the social and emotional skills required to work and communicate with others
• Using the results to provide a view of which systems are providing all students the best opportunity to learn and thrive
In doing so, each participating country is given the resources and information to learn from the experiences of other countries to improve their own systems.
PISA Takeaways - U.S.
The most obvious result of PISA that people will want to know is how the U.S. ranked compared to other countries. Compared to other countries (depending on how you count not statistically significantly different), the U.S. ranked 13th in reading, 18th in science, and 37th in math.
However, rankings mask the really interesting trends that emerge from this test. Here are our main takeaways:
1. Results are flat: Reading and science were slightly above the observed OECD average, while math was below the OECD average. A greater percentage of U.S. students scored in the top performance categories in reading in 2018. Reading has moved all of 1 point since 2000, math has dropped 5 points since 2003, and science is the same as it was 2009.
2. Variation in performance: While PISA generally focuses on average performance at the country level, this masks the variation in overall performance. For each of the tested subjects, 4 in 10 tested U.S. students scored in the lowest two proficiency levels. For math, a quarter of tested students scored at or below the lowest proficiency level.
3. Advantage gaps: Like many other countries, economically advantaged U.S. students were more likely to score in the top proficiency categories as compared to the less advantaged classmates. The U.S.’s socioeconomic advantage gap was larger in reading (99 points) than the OECD average (89 points), meaning advantaged students scored nearly a full standard deviation higher in reading than their disadvantaged peers.
4. Gender gaps: Girls had a 24-point advantage over boys in reading (smaller than the average OECD gap). Boys outpaced by girls by an average of 9 mathematics points. Boys and girls achieved at similar levels in science. Among the top performers in math, 3 in 10 American boys expect to work in engineering or science, while only 1 in 10 top-performing girls share this expectation.
5. (Misplaced?) Educational Expectations: Over three-quarters of the lowest-performing students in the U.S. expect to not only enroll in but complete postsecondary education. In other OECD countries, less than half of the lowest-performing students have the same expectation.
What do these results mean?
The question, then, is what do the results of the PISA exam mean for us? As a general rule, the PISA results should not be considered in a vacuum. Instead, we should view these results as additional evidence for what we found in the national trends on NAEP: there is a growing gap in achievement between socioeconomically advantaged and disadvantaged students and our overall achievement is flat. This is not exactly new information for the U.S. However, it should help to increase the attention that we pay to our education system to ensure we are serving all students equitably. The next iteration of PISA occurs in 2021 with results coming in 2022. We will take another look then to see how PISA’s international results can add to other meaningful national data.