My Vision,     
     My World












Archived Post

The Pitfalls of Comparing Standardized Test Scores

The US is falling behind other countries in standardized test scores.

Surely this story is familiar to you by now. I’ve been hearing it most of my life. So much so that you have to wonder how US students can ever hope to compete with the test-taking all-stars in China, Korea, and India.

My own experience teaching a few courses in Computer Engineering at a moderately prestigious US university has been that students from overseas did not perform notably better in my classes than domestic students.

At first, this surprised me. After all, these were students from countries with extraordinarily competitive educational systems that had the talent and motivation to study abroad. My expectation was that they would be much better than the Americans. I discussed my observation with several colleagues, and they agreed that international students didn’t do any better in their classes.

I asked a few Indian and Chinese grad students for their take on it. I was reminded of that discussion by a discussion I was having with Robert M. on a message board recently:

RM: 3rd world here, and I was 2 years ahead when I started Middle School in US. Our education system was brutal; no multiple choice, meticulous memorization, and teachers don’t give you any breaks on homework or any other assignments. Definitely run like a bootcamp, but I was half-witted dumbass so to me it was always a nightmare.

What they told me was that in China and India, their education was heavily focused on memorization. Partly this was due to issues of practical logistics - with a graduating class of 1000 students, it is difficult to give a personalized education - and partly it was a cultural issue. When these grad students came to the US, they felt that US students were much better at abstract thinking, reasoning, and analysis than they were.

In the field of educational psychometrics there is a pretty common analytical framework known as Bloom’s Taxonomy that classifies different types of learning by their cognitive function. In Bloom’s terminology, the grad students I interviewed were expressing the idea that at the undergraduate level, Indian education focused on the “lower-order” functions - memorizing facts, understanding concepts, and applying problem-solving techniques, whereas the American system focused on “higher-order” functions - analysis, evaluation, and creation of new ideas.

The lower-order skills are the ones that produce good scores on the common standardized tests. This is because it is typically quite difficult to test higher-order functions in an inexpensive way. Memorization and application map well to multiple choice questions; evaluation and creation do not.

Naturally, this provokes us to ask the questions “is learning targeted at Bloom’s lower order functions better than learning targeted at higher order functions”? It might seem from the ‘higher order’ moniker that the answer is supposed to be “no”, but it’s actually more nuanced than that. The lower order learning is probably better suited to the majority of the people who will be applying their knowledge in industry, where application of ideas is prized. Conversely, higher order learning is better suited to research fields, which prize creativity and analytical abilities.

Now, I want to be careful to say that I’m not trying to make a blanket statement that Americans are better abstract thinkers than people from India or any other country; clearly there are first-class researchers from these countries. I emphatically do not want to claim that every (or even most) American has a razor-sharp analytical mind. Nor that all universities and teachers emphasize the same things. And I most definitely don’t want to give the impression that it isn’t important for students to achieve the lower-order thinking being evaluated on standardized tests. The sharpest analytical skills are of little avail in any scientific discipline without a base knowledge of theory and the ability to apply mathematical tools. That said, the point I’m trying to make is that there are limitations in what we can extrapolate about educational success by comparing test scores, and I say that as someone who is probably a much bigger fan of standardized testing than most of the people here. Mastery of memorization and application can take you to the heights of test scores, but a high test scores doesn’t guarantee a great education.

If it appears paradoxical that the US scores abysmally in standardized testing and at the same time has the best research universities in the world…well, it isn’t. It’s a natural result of a system that targets a different type of learning.

collapse serp Says:

This reminds me of some of my dad’s descriptions of working with people from overseas and the various different work cultures. He’s said of India that people are very good about doing exactly what is asked of them promptly and to specification, but they do not “think outside the box” very well, and are prone to being brought to a standstill by any roadblock in the process or divergence from the set plan.


You must be logged in to post a comment.