Legal Commentary

Over at Concurring Opinions, Prof. Solove asked tough questions about how deans are supposed to rank law schools on a 1 to 5 scale when there are so many schools and certainly there are 5 schools that are demonstrably better than each other and other schools. A rational, all-knowing dean would run out of numbers pretty quickly. I joke in a comment that third and fourth tier schools are stuck with negative reputation/quality scores.

Robert Morse responded that the rankings match up with the level of knowledge deans have, and that the rankings come out right in the aggregate.  Prof. Solove responded with questions about whether that means that outliers and gaming drive the rankings.

I, in turn, posted a comment on that thread, but thought it worthy of discussing further in a separate post while I have the opportunity to do so here. In short, I think Morse has it right, and the likelihood of gaming or outlier effect is low.

I tend to think it is difficult to tell the difference between the quality of JD programs (which, I believe, is the new question - not reputation). If this is the case, there is an argument that the 5 point scale is exactly what you want in terms of granularity. 

What the coarseness means is that deans aren?t being asked to put schools in order. They are being asked to put them in groups. Top 20%, Second 20%, etc. It is the variations among the groupings from a large sample that creates the ordering. If 90% of the voters put a school in the top 20%, and 10% put it in the second 20%, then that school will receive a higher score than the school with an 80/20 split.

The beauty is that voters don?t have to choose an ordering between two (obviously close) schools. Indeed, complaints that voters simply lack the information to do so is answered somewhat. Voters may not be able to rank in order, but perhaps they are better able to rank in groups (though even that is somewhat debatable - an issue I will leave for others to argue).

After all ? each of the top 10 programs are all outstanding ? give them all 5?s. 10-40 ? well they?re pretty good, so 4, with maybe some fives. 40-100 ? ?average? ? so 3, with some 4 and some 2. Third tier? All about a 2. Fourth Tier? 1 (ouch). Indeed, voters need not put schools into 20 percentile groups - perhaps all third and fourth tier schools are 2's - then all wind up being in the lowest grouping for that voter.

As a result, it is only those that would put any particular school into a completely different category that determine the ordering. 

So, among the schools listed in Prof. Solove's post, I suspect that the groupings look like this:

Yale 5 (top 20%)

Michigan 5 (top 20%, with a few putting in second 20%)

Cornell 5 (top 20%, with some putting in second 20%)

USC 4 (top 40%, with some putting in top 20%)

Emory 4 (top 40%, with few putting in top 20%)

American 3/4 (top 60%, with many putting in top 40%)

And so forth. This, I think is not unreasonable, and could yield more robust results than would a straight ranking of 180 schools because voters are not expected to have perfect knowledge.