Home -> Law Blog Directory -> Academic Blogs -> The Legal Workshop

Academic

: The Legal Workshop

The Costs Of Judging Judges By The Numbers

By Marin K. Levy

What is to be gained by using empirical evidence to rank or ?judge? judges? These empirical studies claim two major benefits. First, because the criteria are assertedly apolitical, the resulting rankings will identify the ?best? judges across the political spectrum and thereby improve, for instance, the Supreme Court nomination process.¹ Second, because the criteria are ?objective?² rather than ?subjective,?³ the possibility of unconscious bias is greatly reduced and the studies may at last yield answers to two difficult questions: whether female judges are better than male judges⁴ and whether appointed judges are better than their elected counterparts.⁵ These claims in turn rest on two assumptions: that the legal empiricists have selected the right qualities to measure and that their methodology for doing so is accurate.

In the years since the first empirical rankings were published, both judges and scholars have cast grave doubt on the accuracy of these assumptions,⁶ and we seek to contribute to this discussion.⁷ We believe that there is now a general consensus that (1) the judicial virtues the legal empiricists set out to measure probably have little bearing on what actually makes for a good judge, and (2) even if they did, the empiricists? variables have not measured those virtues accurately. As matters now stand, the benefits professed by the empiricists have not been achieved by their models.⁸

The failure of empirical rankings to deliver the benefits promised by their proponents is enough to raise serious doubts about whether the enterprise is worthwhile. But even more importantly, these rankings stand to create (and, indeed, may have already created) several significant harms. By generating and then publicizing unreliable claims about the relative quality of judges, these studies mislead both decisionmakers and the public, degrade discussion of judging, and could?if taken seriously?perniciously alter the behavior of judges themselves. Far from improving the selection process of would-be judges and justices, reliance on such empirical rankings would distract all involved from serious inquiry and debate about the nature of judging, the process by which judges should be selected, and contested visions of justice itself.

In this Essay, we focus on the presently dominant models of judging judges, which have been advanced by Professors Stephen Choi, Mitu Gulati, and Eric Posner, among others [hereinafter, ?Choi et al.?].⁹ We briefly explain why these models are deeply flawed?revisiting arguments made by others and positing new ones?and why there is little hope for meaningful improvement. We then discuss the harms of relying on imperfect models, and conclude that the current enterprise of ?judging judges? has significant costs.

1. Flaws in the Current Models

Choi et al. posit three ?objective? measures of judicial merit: (1) productivity; (2) opinion quality (or what they sometimes call influence); and (3) independence. Although Choi and his coauthors have revised their models from time to time, they consistently have professed to be able to measure these qualities from datasets of appellate¹⁰ merit [of appellate judges] are less likely to affect a tournament of U.S. district court judges. Because federal district judges generally make decisions alone, their opinions are less likely to be affected by the preferences of their colleagues. In addition, they have almost complete discretion over choices such as whether to publish an opinion in a particular case. Thus quantitative measures of judicial performance will be measures only of a particular judge?s behavior rather than also of the influence of other judges on a panel.? (footnotes omitted)). Professor Mitu Gulati and Duke Law School Dean David Levi (who is a former United States District Judge), have argued that in light of the fact that ?the literature on judicial measurement has largely ignored the question of how best to measure trial court performance,? it would be beneficial for empiricists to study trial judges. David F. Levi & Mitu Gulati, Judging Measures, 77 UMKC L. REV. 381, 403?04 (2008). Gulati and Levi note, however, that empiricists would need to develop different measures to do so because, for example, publication rates are less meaningful for trial judges than for appellate judges. Id. at 404?05.] judicial opinions by looking to, respectively: (1) how many signed opinions a judge publishes; (2) how many times the judge?s opinions have been cited in other jurisdictions; and (3) how many times the judge has dissented in cases in which the majority opinion is written by a member of the same political party. The authors then use their measures to rank judges, and also to answer whether female judges are ?better? than male judges, and whether appointed judges are ?better? than elected judges. As noted above, several commentators have identified serious flaws in this general model.

First, the judicial virtues selected by Choi et al. do not adequately capture the qualities good judges possess and the activities in which they engage. Professors Scott Baker, Adam Feibelman, and William P. Marshall point out that ?because of data limitations, [Choi et al.?s] study ignores aspects of judicial behavior that are arguably more important than the ones proxied, such as integrity, fairness, open-mindedness, thoroughness, and temperament.?¹¹ As Chief Justice Laura Denvir Stith of the Supreme Court of Missouri has explained: ?[n]o matter how much one breaks down the criteria used as measures, those criteria still miss the essence of judging.?¹² Moreover, a model with only the three mentioned virtues fails to take into account the other important work that appellate judges do in addition to opinion writing, ?including judicial administration and lower court supervision, serving on government commissions, teaching at law schools, writing scholarly articles, and training other judges,?¹³ and, we would add, expending significant time editing or commenting on other opinions that will be published under the name of a colleague. All of this work is both critical for the functioning of courts and a significant part of what ?good? judges do.

Second, even if we could agree that Choi et al. had selected judicial qualities that are possessed by the ?best? judges, they have failed to measure those qualities accurately. Beginning with productivity, it seems obvious beyond peradventure that a judge may be productive by doing things beyond publishing her own signed opinions. In many courts, judges issue far more unpublished opinions than published ones. Furthermore, the measure fails to account for another common and important form of judicial production: the per curiam opinion (which is published without the identification of the author). While practices with respect to per curiam opinions differ among state courts and among federal circuit courts, in many jurisdictions they constitute a significant portion of all published opinions. And although per curiam opinions tend, on average, to be shorter than their signed counterparts, they still consume much of the time and energy that signed published opinions require. Because the datasets used by Choi et al. do not include per curiam opinions, and because the writing of these opinions is not spread evenly across a given court, the statistical analysis undertaken by Choi et al. is incomplete and quite possibly misleading. Indeed, judges in the state dataset published on average only twenty-two signed opinions per year;¹⁴ issuing even a few per curiam opinions more than average should significantly affect the judge?s ?productivity? ranking.

The point is not simply that per curiam opinions (along with unpublished opinions) are a major category of written judicial production that Choi et al. have failed to account for in their methodology. The more critical point is that their methodology will never be able to account for these opinions. Because, as their name suggests, per curiam opinions are the opinions ?by the court,? they are not signed by the principal drafter (and, similarly, unpublished opinions are unsigned). In short, there is no way to know which judge should get credit for the effort of writing these opinions. This important category of data is simply missing, and it cannot be recovered by Choi et al. or anyone else.

?Opinion quality,? too, is an ambiguous and potentially broad-ranging concept. Yet all that Choi et al. measure is the number of citations that a judge?s opinions receive in published opinions in other jurisdictions. Citation counts need not, of course, reflect quality. Even as a rough measure of one type of judicial influence, the chosen proxy is wanting. There are many factors that contribute to why an opinion might be cited that have nothing to do with the insight or persuasiveness of the opinion and nothing to do with the ongoing influence of the opinion writer.¹⁵ If a novel issue arises in one state court that has been addressed in only a few other states, the opinions from those few other state courts are likely to be cited whether or not they are persuasive or otherwise of high quality. There is a simple temporal component to citations as well. Unless there is a seminal case for a particular proposition, many judges prefer to cite recent cases articulating the governing standard or principle. Accordingly, there may frequently be situations in which Judge A publishes an opinion enunciating a particular rule or standard, and a few years later, Judge B follows and restates this very same principle of law. Simply by virtue of being more recent, the later opinion may garner more out-of-state citations going forward.¹⁶

Moreover, using citation counts as a measure of opinion quality would appear to introduce a bias in favor of those decisions (and those courts) that reach out to resolve issues that courts in other jurisdictions have left to legislative determination.¹⁷ Once one court identifies or articulates a new right or rule of law, litigants in other states or circuits will urge their courts to quite literally follow suit?resulting perhaps in opinions that cite the novel judicial proposition even if that proposition is thoroughly rejected. This situation reflects neither judicial influence nor opinion quality, much less ?good? judicial behavior that should be rewarded in judicial rankings.

Finally, there are serious flaws in the way that Choi et al. measure ?independence.? Although we would readily agree that independence?in the sense of not being influenced by certain personal, political, or other inappropriate considerations?is a valuable attribute in a judge, that quality should not be confused with what the model actually measures, which is a disposition to dissent. Dissenting frequently, regardless of the political affiliation of the other panel members, might well be due to characteristics associated with ?bad? judging.¹⁸here plainly is a point at which dissenters cross the line from enriching thought into either intellectual preening or obstructionist polemicism.?).] As Professor Baker and his coauthors have noted, ?independence defined in this way may actually measure a lack of quality in judging, including deficiencies in collegiality and leadership, and a propensity for judicial activism outside the ideological mainstream.?¹⁹udicial activism at the trial level is not necessarily an ideological question or a political question or a jurisprudential question?it is, rather, a psychological question; whether a judge is ?active? or ?passive? is often simply a reflection of the judge?s psychological disposition rather than his political worldview. The most important ?ism? in a trial judge [isn?t liberalism or conservatism but] . . . metabolism. . . . ?[J]udicial activism? at the trial level often may mean little more than that a particular judge is assertive and demanding?even overbearing?in the assertion of his will and his authority. In some such circumstances, the assertion of judicial power may as readily be a function of an individual jurist?s disposition to command and to take charge . . . .?).]
We believe this problem deserves even greater emphasis. There is an obvious aspect of good judges that is in tension with the measure of independence chosen by Choi et al.: the ability to build a coalition or achieve consensus. Nowhere does their analysis credit the judge who is skilled in convincing other judges to join his or her opinions. Perhaps such a judge writes particularly sound opinions and therefore has an easier time convincing other judges to sign on. Or perhaps the judge is simply good at finding compromises or alternative routes to reaching a decision that command greater agreement. Yet such a judge will dissent less frequently, regardless of the political affiliation of the other panel members.²⁰

Here again, the fundamental problem with the approach of Choi et al. is that there is no clear way to solve the measurement problem. To credit dissents as evidence of independence necessarily discredits coalition-building as relevant to judicial quality. Without additional information, there is no way of knowing whether judges who frequently dissent are independent, are poor at persuasion and coalition building, or simply have idiosyncratic views.

1. Why the Current Models Cannot Be Improved

In response to these objections, the empiricists? natural reaction might well be to continue to adjust or add to their proxy variables and to conduct new studies. They may seek to improve the enterprise of judging judges by refining or expanding their model, in the hope that they can eventually fix whatever deficiencies currently exist. We hope that this is not the direction taken.

The overarching problem here is not that the methodology is flawed (though it is), but that the enterprise itself is misguided. Choi et al. cannot simply tweak their model to take into account many of the concerns we and others have noted because the data inevitably conflate countervailing qualities. Because of the complexity of judging, it is not surprising that a single data point (a published opinion, a citation, or a dissent) may be both consistent and inconsistent with admirable characteristics. Most importantly, the very meaning of ?good? judging is contested in a heterogeneous, pluralistic society. For some opinion elites (including, apparently, Choi et al.),²¹he goal for our tournament was to provide an improvement over the current system. The harder question is whether our measures could play a role in a more bipartisan, but subjective, selection process of the type that some would claim we have had in the past. Our view is that they should; objective measures will serve as a check on the inevitable biases that any system of subjective analysis will possess (and vice versa).?).] the Supreme Court nomination process would be improved if it were less partisan and if it were less dependent on a nominee?s views on politically charged issues. For others, this development would be worrisome, betraying the ultimately political character of both presidential nomination and senatorial consent.²² Similarly, analyses purporting to ascertain whether female judges are better than male judges and whether appointed judges are better than elected ones founder not just on data and methodological limitations, but also (and unavoidably) on disputed understandings of what it means to be ?better.?

1. The Harms of Relying on Imperfect Models

While the project of Choi et al. is unsustainable on its own terms, its false promise?that the quality of a judge (indeed, of judging) can be captured by a composite ranking?threatens to do real damage. First, far from improving the Supreme Court nomination process, increased reliance on anything like these rankings could arbitrarily derail or encourage particular nominations. Consider the appellate judge who writes more than her share of per curiam or unpublished opinions, who hesitates to propound novel legal theories, and who seldom dissents because she knows how to build a majority. This judge will rank toward the bottom of all three measures employed by Choi et al.: signed opinions, citations by other courts, and dissents from others appointed by Presidents of her party. Is there now a presumption?one that is irrebuttable as a practical matter²³?that she is unqualified? If the available empirical models are treated as actually indicating something important, will we lose such judges?either because the media will be unduly influenced by this seemingly ?expert? and ?objective? ranking or because political actors will make the calculation, in part on the basis of media reports, simply not to nominate them?

The delusive allure of the empirical judicial ranking model has already distorted critical debates about the judiciary. Recently, several articles in the popular media have reported the findings of Choi et al. as definitive conclusions. In May 2009, The New York Times published an article entitled, ?Assessing Sotomayor?s Influence,? which purported to reveal how influential Sonia Sotomayor had been as a judge based on how many citations her opinions had received.²⁴ The article did not mention any criticisms of the use of citation counts as a measure of either quality or influence.

Indeed, it is instructive to examine how Judge Sotomayor and Judge Samuel Alito, both nominated to the Supreme Court after the first article by Professors Choi and Gulati was published, have fared in the wake of this scholarship. Judge Alito, sitting on the Third Circuit from January 1, 1998 through December 31, 2001 (the time period for the study) ranked sixteenth out of the seventy-four judges. Considering the company, perhaps that is ?not too shabby,? though it does not sound as good when stated slightly differently?that he was in the seventy-eighth percentile of federal appellate judges.²⁵ In any event, the media did not trumpet his overall ranking, but rather, that he was in the ?top ten? on the ?independence? ranking.²⁶ Like all numbers, the Choi et al. rankings can be spun.

In fact, the numbers have been spun not only by the media but by one of Professor Choi?s coauthors. Judge Sotomayor had not been included in Professors Choi?s and Gulati?s ranking of federal judges because she was not appointed to the Court of Appeals until mid-1998.²⁷ Shortly after Judge Sotomayor was nominated to the Supreme Court in 2009, Professor Eric Posner published online an analysis that sought to extrapolate from Sotomayor?s 1999?2001 data where she would have placed in the original ranking. Posner concluded:

The bottom line is that Judge Sotomayor is about average, or maybe a bit below average, for a federal appellate judge. These results are far from conclusive, but one might think that put the burden on Judge Sotomayor?s defenders to come forward with stronger reasons for her nomination than they have so far.²⁸

Not surprisingly, the defenders did come forth. Two weeks later, Professor Posner announced a different conclusion: ?Sotomayor looks good.?²⁹ Working with Professors Gulati and Choi, he examined Judge Sotomayor?s opinions between 2004 and 2006, and compared her not to all other federal judges but to a subgroup consisting of ?other court of appeals judges who were rumored to be on President Obama?s shortlist, and with a kind of control group consisting of court of appeals judges rumored to be on President Bush?s shortlist in 2005.?³⁰ Compared to this group, she ranked highly both in total number of citation counts and in citations per opinion (the latter being a measure discussed, but not employed, in the rankings previously published by Choi and Gulati);³¹ Posner referred to these citation-count data as ?a measure of general influence.?³² He noted that the use of citation counts had been criticized, but addressed that concern only with the non sequitur: ?Still, the numbers seem pretty robust, and the comparison here is with well-respected judges, not with ordinary judges.?³³ Of course, if the measure does not make sense, it is not clear what ?robust? numbers might mean. Moreover, the second posting seemed to undercut the core of the larger empirical project, for it both downplayed the other two measures?productivity and independence³⁴?and reverted to subjective general reputation (?not ordinary?) as a measure of quality.

Even though commentators have powerfully criticized their model, the authors cannot seem to resist trumpeting its claims?without acknowledging the model?s severe limitations. In October 2009, Professor Choi and his coauthors published an article in Slate, titled ?Do Women Make Better Judges? Asked and Answered?With Data,? ³⁵ which The New York Times picked up in an article titled, ?Women Judges Less Credentialed, but Equally Good.?³⁶ Thus both the media and the model?s authors proactively publicize the ?results? of what the authors know to be a limited or flawed model. This looks more like false advertising than scholarship.

IV.  Conclusion

Studying judges?both what they do and what it means for them to do their job well?is important. The common, informal practice of judging judges on the basis of general ?reputation? within the bar or among other judges may be inadequate, fraught with misunderstanding, and subject to manipulation by judges and by those who comment on the work of judges, among others. But a more rigorous study of judges must be approached with an appreciation that the job of judging is complicated and multifaceted; judging judges should not be reduced to the counting of discrete data points that yield ordinal rankings. Given the limitations of available data, the model developed by Choi et al. cannot accurately measure even what it purports to measure?productivity, opinion quality, and independence. Given the nature of judging, we doubt that any such data-driven model can ever describe what constitutes ?good? or ?bad? judging, at least not to the satisfaction of those who take the time to study the models.

Whatever ?good judging? may be, it is not to be found in how individual judges score on a law-professor-generated rating system based on crude data and a shallow understanding of the art and science of judging. The biggest cost of judging judges by these numbers would be if the public and political decisionmakers (not to mention judges themselves) are influenced by the rankings. In that event, the entire rankings project not only will have failed to identify good judging, but will be counterproductive to that end.

Acknowledgments:

Copyright © 2010 Duke Law Journal.

Marin K. Levy is a Lecturing Fellow at the Duke University School of Law.

Kate Stith is the Lafayette S. Foster Professor of Law at Yale Law School.

José A. Cabranes is a Judge on the U.S. Court of Appeals for the Second Circuit.

Full post as published by The Legal Workshop on February 25, 2010 (boomark / email).