Skip to main content

Looking at "Discrepant Scores"

Several years ago, The College Board produced a study of "discrepant performance," after studying about 150,000 students and their freshman-year grades in college.  If you want to see the study, you can get a pdf of it here. The title, of course, is interesting. (And before we get too deep, it's important to note that these are old SAT scores, in case you think the new test is better and want to argue that point, nice tight concordance between the old and the new notwithstanding.)

Discrepant performance is defined as standardized test scores that are inconsistent with a student's academic performance in high school.  The distributions of scores and grades were normalized, and then each student's z-score on tests was compared to their z-score on grades.  Just under two-thirds of students had scores and grades that were consistent, and we don't need to talk too much more about them.

The most interesting thing is the other one third of students: Those whose grades were higher than their tests (High GPA) and those whose tests were higher than their grades (High SAT).  Each group was about one-sixth (17.5%) of the sample.

Back to The College Board presentation for a moment.  It suggests that high testers with low grades are a better risk than low testers with high grades, but it also says grades are a better predictor by themselves than tests.  That's not statistically impossible, of course, but it does seem curious.  And it does fly in the face of both my experience and conventional wisdom regarding the best way to make admissions decisions; I think most admissions officers believe high tests/low grades are the weaker bet of the two extremes.

But let's go with that for a minute.  And let's ask why it might be true. But first, let's argue with what little methodology is presented here in this study.  A lot of the conceptual problem in predicting human performance, of course, comes from our own arrogance: In this case, the belief that a limited number of pre-college factors represent the sum total of factors affecting freshman grades.  How limited, in this case?  Two.

If you really wanted to get a good model to predict freshman performance, you'd look at a lot of factors: Family income, parental attainment, ethnicity of the student vis-à-vis the student body and vis-à-vis the high school they came from, just to name a few.  All of those factors are important, and what we find is that students from lower-income families whose parents didn't go to college, and who feel out of place in the college they've enrolled have some struggles.  I don't see any of these factors controlled for in this analysis (if I'm wrong I'll be happy to correct it.)

You can see the table of how discrepant performance breaks out, but you can you really see it?  Let me draw you a picture.  On this chart, (which shows only the students with discrepant performance), the light blue bar on the left chart shows the number of students with high tests and lower GPA (High SAT); the orange bar on the left chart show the number of students with low tests and high GPA (High GPA).  Hover over the bars to see how many there are.  One the right chart, the mauve colored bar shows what percentage of each group had high SAT. (The ones The College Board says you should give the breaks to).

Surprise: Guess who tends to have higher grades and lower scores?  Women (who get better grades at every level of education than me, by the way); poorer students; students from underrepresented ethnic groups, and students whose parents have less education.  This narrative plays smoothly into the prevailing wisdom of 1930, which suggested they just were not suited for higher education, and which some people still seem to believe.

Who has higher scores and lower grades? Men, white and Asian students, wealthier students, and children of well educated parents.  And The College Board statistics tell you these are the students you should give a break to in the admissions process because they did better on a three-hour test.  You see, in the simple approach, only SAT scores and GPA determine your college performance, and it's not at all affected by how much you have to work, or worry about money, or spend time figuring out how college operates, or whether you belong there.  So keep giving the white guys the break.

Two final points: If I took the labels off the bars, and told you "This is the gender chart," or "This is the income chart" you could probably put the labels on in correct order pretty easily.  Second (and this could be a whole other blog post all together) even the poorest students (a B- average) with the lowest test scores ended the first year with an average GPA of 2.0, and the differences between and among the groups are exaggerated by a truncated y-axis on the chart in the presentation.

As always, let me know what you think.


  1. I have intuitively sensed this, but it is good to see the data. Somehow your blog fell of my radar. Good to come back to it. Keep it up. Thanks, Carey


Post a Comment

Popular posts from this blog

2018 Admissions Data

This is always a popular post, it seems, and I've had a couple of people already ask when it was going to be out.  Wait no more.

This is IPEDS 2018 admissions data, visualized for you in two different ways.  You can switch using the tabs across the top.

The first view is the universe of colleges and universities that report data; not every college is required to, and a few leave data out, and test optional colleges are not supposed to report test scores.  But IPEDS is not perfect, so if you find any problems, contact the college.

On the first view, you'll see 1,359 four-year private and public, not-for-profit institutions displayed.  In order to make this as clean as possible, I've taken out some specialty schools (nursing, business, engineering, etc.) as many of those don't have complete data.  But you can put them back in using the filter at top right.

Hover over any bar, and a little chart pops up showing undergraduate enrollment by ethnicity.

You can also choose to…

Another 1000 Words and Ten Charts on First-generation, Low-income, and Minority Students

I have always enjoyed writing, and I consider this and my other blog like a hobby.  Usually, I spend no more than 45 minutes on any post, as I don't make my living by writing, and my blogs are not "monetized." But once in a while, an opportunity presents itself to write for a wider audience, and that's when I see what it takes to make a living putting words to paper. That happened this week.

You may have seen my opinion piece in the Chronicle of Higher Education. If not, you can read it first, read it last, or not at all; I think both this and that stand alone, despite their relationship.  In the end, we ended up with about 40% of my first draft, which is what happens when you write for a print publication. And of course, a print publication makes interactive charts, well, difficult.

I think there is more to say on the topic, because the similarities in recruitment challenges for first-generation, low-income, and minority students tend to look a lot alike, and the mo…

So you think you're going back to the SAT and ACT?

Now that almost every university in the nation has gone test-optional for the 2021 cycle out of necessity, a nagging question remains: How many will go back to requiring tests as soon as it's possible?  No one knows, but some of the announcements some colleges made sounded like the kid who only ate his green beans to get his screen time: They did it, but they sure were not happy about it.  So we have some suspicions about the usual suspects.I don't object to colleges requiring tests, of course, even though I think they're not very helpful, intrinsically biased against certain groups, and a tool of the vain.  You be you, though, and don't let me stop you.However, there is a wild card in all of this: The recent court ruling prohibiting the University of California system from even using--let alone requiring--the SAT or ACT in admissions decisions next fall.  If you remember, the Cal State system had already decided to go test blind, and of course community colleges in th…