University Rankings are Terrible. Now Can We Stop Doing the Same Thing With Countries?

The release of the US News & World Report college rankings is as good an excuse as any to talk about the sheer ridiculousness of organising complex institutions into rank order and pert decimal scores.

The criticisms of the index itself are nicely summarized in this Atlantic article, but for those that don’t have time for the full Gladwell, they are basically of two kinds:

  1. The ranking is flawed. The methodology constantly changes, schools juke their stats, it’s based on bullshit surveys that only measure the school’s established reputation, etc. The data might be good enough to distinguish Harvard University from  Southern Methodist Tech, but there is too much noise to say that Yale is better than Princeton or that Oregon State is better than Penn State.
  2. Most of the information we use to determine the quality of education isn’t readily measurable. How good are the teachers? Do students get enough personal attention? Is the campus social life welcoming or cliquey? If you list everything that made your college experience positive or negative, you won’t find it in the number tables of these rankings.

A few years ago I was working at a human rights NGO, and one of my jobs was preparing reports for multinational corporations telling them about the human rights situation in countries where they were thinking of operating.  You want to open a shoe factory in Kenya? Here’s what you need to know about gender discrimination, corruption, occupational health and safety. Here’s how you make shoes there without violating human rights.

Sometimes companies would ask us for big packages of countries, 10, 20, 50 at a time.

‘Can’t you just give us a ranking?’ they would ask. ‘Tell us which country is the worst of that list, and we won’t make shoes there.’

Or, they would suggest, better yet, give us an index. If you tell us that Bolivia scores 8.2 out of 10 and  Iran scores 8.3 out of 10, we can make our decision on quantitative data rather than just putting our finger in the air.

‘But that is putting your finger in the air,’ I would tell them. ‘It’s just us doing it instead of you. Bolivia and Iran, their politics, their demographics, their economics, they look nothing like each other. Two numbers isn’t going to make that go away.’

This is one of my beefs with the Failed States Index, the Economic Freedom Index, the Human Development Index, the Corruptions Perceptions Index, the dozens of other indices that purport to rank countries according to some difficult-to-measure variable.

The problem is, countries have all the same problems as colleges. The data out there isn’t strong enough to justify precise determinations, only broad tranches. Yeah, Norway is less corrupt than Angola, but I don’t need an index to tell me that.

But is Norway less corrupt than Denmark? More developed than Switzerland? Given the limited data and even more limited number of indicators these indices use, the answers to those questions are a re-statement of your methodology, not a useful analysis of the conditions in those two countries.

I used to try to tell the companies this, that any attempt to rank countries according to their ability to prevent corporate human rights violations would be like trying to rank kittens according to cuteness. After you separate them into the already-obvious tranches, it’s just a judgement call, preferences disguised as data.   

‘But it would be so much easier for me if you could do that anyway.’ Only one corporate person ever actually said it this directly, but afterward I started hearing it, in subtler ways, from the others.

Eventually I realized that the only reason the companies pushed so hard, why they insisted so strongly on  rankings and scores over information and analysis, was because it made it not their problem anymore. They didn’t have the credentials to pull 50 ‘good’ countries from 100 uncategorised ones, so they used us to push the responsibility away. ‘It’s not me saying Bolivia is an 8.2,’ they could tell their boss. ‘A human rights NGO said it was. Making shoes there is totally approved. 

I don’t know if high schoolers use college rankings to decide where they should get educated. And I don’t know if multinational corporations use country indices to decide where they should make shoes. I just hope that in both cases, they know that most of what they’re seeing is either totally obvious or entirely unsubstantiated.


Filed under America, Personal, Work

26 responses to “University Rankings are Terrible. Now Can We Stop Doing the Same Thing With Countries?

  1. Pingback: University Rankings are Awful. Can We Stop Doing the Same Thing With Countries? | Rocketboom

  2. I agree that university rankings suck. So I started a website to fix it:

  3. Thank you for your reminder.

    I was trained as an engineer, working for a large aerospace company. I was charged with working on large, complex problems, and proposing courses of action. Almost invariably, I developed models and equations to help me understand – probably as a kinesthetic exercise, waiting for my intuition to kick in and let me see the pattern.

    When I made recommendations to the decision makers (the money holders), they rightly required that I provide my analyses, which included these models. I was always astonished at the ease with which brilliant, mentally agile minds were so easily seduced by a set of numbers, or scatter plots, or curves.

    Eventually, I reconciled myself to the old saw, of “lies, damn lies, and statistics”, perhaps coined by Disraeli (I had to looked it up). Some would poke and prod my methodology, and apply some form of sensitivity analysis, to see if they could steer the recommendation in a different direction. Most, however, accepted the models. They would even argue with me when I offered alternatives with lower scores, especially in risk avoidance discussions.

    I was not immune to my own creations. I learned to exercise great care, that I keep my options open as long as I could. Fortunately, my Myers Briggs type is ENTJ. While I am drawn by equations and models, I rely on a strong Intuitive streak to see if it “makes sense.” I tell myself I’m not just trying to steer to the answer I want. But, maybe I am, especially when the numbers add up, but don’t seem to make sense.

    Thanks for the trip down memory lane,


  4. Problem is so much is ranked in the judiciary of public polling and opinion. Good luck getting objective truth with that.

  5. Reblogged this on Reformed Reporter and commented:
    I’m impressed with the insight of this post and compare it to rather uninsightful articles I’ve read about “rankings” of my new hometown of El Paso, Texas, compared with its cross-border sister of Ciudad Juarez, Mexico.

  6. You’re so right about the desire to shift responsibility to a set of numbers. Decision makers are supposed to be people skilled in making good judgements. There’s a lot more that goes into a good judgement than a set of numbers. But if something goes wrong, a decision maker knows that a lot of blame can be deflected to the numbers. Perhaps part of the problem is that we are too quick to try blame someone when things go south, thus intimidating the decision makers before the decision is even made. If you know you’ll get a firestorm of blame if your decision blows up, you’ll be careful to build a scapegoat into the decision just in case.

  7. While some of the data behind rankings may be questionable, some rankings of universities do use valuable data. For example, the rankings of the average earnings of university graduates after college to rank the results of attending one university over another.

    Would you want to attend a college where the graduates had better odds of employment and higher pay or one that churned out graduates who had a high risk of being unemployed or under-employed and earning [much] less.

    • While I’m aware that many students attend college these days with job opportunities in mind, the focus on jobs subverts the higher purpose of university education. We should use universities to emphasize good thinking skills (those that lead to people being able to make good judgements), and put the job training aspects in community colleges, where less student debt is piled up. Society needs a small cadre of trained people who pursue truth and knowledge wheresoever it leads, and should be willing to bear some of the cost of educating those people. But whether or not we need another Wall St. banker with an Ivy League MBA is a very debatable question. How many of those MBA bankers who led us into the Great Recession showed good judgement?

  8. Reblogged this on whiteness betrayed and commented:
    A wonderful analogy to an ever-growing desire to quantify the qualitative realities of this world.

  9. Excellent points – and well done on being Fresh Pressed.

    Part of my job involves looking at country indexes and they are usefull if you don’t peer too closely at them.

    A more sensible way of looking at data is to group it into three bubbles where the middle bubble overlaps the top and bottom bubbles.
    Then (say for countries v corruption) you can lump all the obviously good countries in the top bubble and the obviously bad countries in the bottom bubble. The overlap areas are for countries going down or up the scale.
    Comparing countries on the corrution index works if you do it this way.

  10. Agree. there are really qualitative factors that are just hard to quantify and then do the comparing. And of course, many don’t like being accountable of their decisions.

  11. Interesting analogy….but it is a little too simplistic to assume that decisions are solely made on investment a d production, using these economic freedom index for example…..there are other qualitative issues to be considered…I agree with companies wanting to shift responsibility to numbers, but will disagree to a complete “ridicule” of the system of using indices to rank the Socio-economic performance of countries.

  12. I am an Independent Educational Consultant (to college students, mostly first time international grads/professionals sponsored by their jobs) and I hear, something along the lines of “please suggest a list of programs in the top 100 U.S. News ranking.” In order to receive company and university sponsorships, these students must gain acceptance into a high ranking university. Of course, I understand this need and try to help them accordingly, but I always tell students to seriously consider their personality types, learning styles and their vision of a great graduate school experience when looking for a program. Unfortunately, the ones who foot the tuition bill (employers seeking bragging rights) get to set the rules regarding who they will sponsor or not based on the university’s name that the employer will add to their company.

  13. nice analogy!!

  14. The whole point of university rankings is to instil a “zero sum” competitive mindset into academia that distracts people from either trying to genuinely improve teaching and research OR effectively liaising to present a coherent sectoral argument.

    Own meditations are here –

  15. University rankings are terrible because theyre artificial and hollow. When I chose my school, I did it based on cost, reputation and size of the student body.

  16. Wow, what a great article. I don’t want to add anything and spoil it.

  17. Excellent article! I studied at a university in South Africa that did not even make the top 500 and today has a good job in a global company! Life is much more than the ranking of a university or college!

  18. This is really insightful. Good analogy.

  19. Quartiles are valuable, and mostly accurate, for the discerning parent or student contemplating a College education. Some methodology must be used to separate the choices, and almost any conceivable method of determining the schools’ placement will generate a rank. Yes, it’s noise, but it sells magazines and creates competition, so it’s not all bad.

  20. Math is overrated 🙂

  21. blueroselady

    Thank you for your reminding and eye-opening post. While I was a kid and having no family members who went to colleges before, university rankings played important roles in shaping my dream (and choice of university). I was hoping that by going to a good university, it will be easier for me to find a job, to earn a living, and to support my younger siblings. Only after I have gone through the system, I realize that every college / university has its own strengths and weaknessess, it is more important to find the right match for students’ needs.

  22. I agree with the fact that quality is hard to rank. That being said it don’t think it will ever change. Simply because we will always try to measure things, all though it may be for different reasons. We like to compare ourselves to others to feel better, we rank things in order to justify our choices etc. And if others aren’t doing it for us, we will do it on our own. It ranges from restaurant reviews to university comparisons to ranking the happiest country on earth, And as you know, when you live in Denmark, we love the “happiest on earth” ranking for the very simple reason, that we are always on top of it. So we can agree to dislike all other rankings – but that one is the real thing!