Legal Commentary

Why I'd Stick With Yale Clerks-- Some Econometric Ruminations

By Jack M. Balkin

In my view, Justice Scalia blundered badly last week in his concurring opinion in Baze v. Rees when he ineptly attempted to suggest that the empirical evidence supported the view that the death penalty in the U.S. has deterred murder. My coauthor Justin Wolfers and I had shown that the evidence upon which Scalia rested his beliefs did not in fact support his conclusion. The empirical debate on the death penalty perfectly illustrates the conclusion that it is very easy to draw erroneous conclusions from statistical data. Donohue and Wolfers, ?Uses and Abuses of Empirical Evidence in the Death Penalty Debate,? 58 Stanford Law Review 791 (2005).

Another illustration of empiricism gone astray is provided by a new working paper by Royce de Rohan Barondes, which adopts the following provocative title: "Want Your Opinions Questioned or Reversed? Hire a Yale Clerk." The man bites dog nature of the claim is sure to raise interest in the paper, since Yale is obviously one of the most elite law schools in the U.S., and the hardest to get into. Unfortunately, counterintuitive empirical results almost always turn out to be wrong if they are not based on an appropriate empirical methodology for the inquiry at hand. In my opinion, the methodology of the Barondes is flawed, and the conclusions drawn from this research are either incorrect or unfounded. My review of the Barondes paper (as well as my own personal experience with Yale Law students) affords little reason to believe that the value of a Yale Law clerk is less than the law school?s preeminent ranking would suggest.

Before turning to the problems in the paper, let me mention some good points. Barondes has collected data on judicial decisions by federal district court judges, along with information about the law clerks for these judges, and creatively thought to see if Shepherd?s Signals could give some inexpensive insight into the quality of the judicial decisions. This is all to the good. However, there are no shortcuts to knowledge, and I would encourage Barondes to spend more time examining the nature of the cases that are getting questioned by the Shepherd?s indicator (about which more will be said below). The primary problem with this study involves the nature of the econometric specification. Papers that fail adequately to address the difficult specification issues of endogeneity and omitted variable bias simply do not provide reliable estimates and therefore cannot be relied upon.

Barondes claims that he has found evidence that federal district court judicial opinions are reversed or questioned more when the authoring judge has a greater number of Yale Law clerks. The paper presents some interesting data concerning the roughly 13,000 judicial opinions written by 95 district court judges over a 56 month period, but it makes two major mis-steps. First, it ignores all of the recent lessons of why panel data analysis is a superior method for estimating causal influence than a simple cross section analysis. Second, even if the cross-sectional approach of the Barondes paper were not flawed, Barondes has overstated his results and has failed to control properly for various important factors, such as political affiliation of the judge that could well undermine his claims.

Let?s begin with the big problem of causal influence. Barondes uses ?Shepherd?s Signals? as a quick device for identifying something problematic in the 13,000 opinions in his data set. Barondes concludes that judges with more Yale Law clerks tended to have more of these ?problematic? signals, then we might expect given the high ranking of Yale Law School. Unfortunately, Barondes has failed to understand the likely causal relationships between behavior of judges in writing decisions that are more likely to be questioned and the process of selection of clerks. The judges almost surely drive the error rate and the clerks show up as ?significant? in the regression because there is small, but possibly significant relationship between the type of judge that gets reversed and those who will select (and be attractive to) Yale law students.

Simple cross-section studies of the type that Barondes provides are not well-suited to teasing out causal relationships given the underlying links when some judges are both more likely to have these ostensibly negative signals, and somewhat more likely to hire Yale Law clerks. Barondes? error is a bit like concluding that because the death penalty is almost nonexistent in the Northeast, which has the lowest murder rate in the country, and widespread in the South, which has the highest murder rate, this means that the death penalty causes murder.

While it is true that a simple cross-section correlation of execution rates and murders by state will naively suggest that more executions lead to more murder, this regression will almost certainly generate the wrong causal answer in suggesting that the death penalty massively increases the murder rate. A better approach would control for the fact that some states (e.g., Southern states) have persistently higher murders rates and then look to see what happens to murders in those states when executions rise or fall.

Similarly, to tease out the effect of a Yale Law clerk, Barondes needs to hold the judge constant in the same way that we just said we have to hold the state constant to tease out the effect of an increase or decrease in executions. Interestingly, Barondes did have this data available, but he failed to use it. Specifically, he collected information on district court decisions written in two different 28 month time periods. A first cut at the question he wants to answer ? what is the impact of having an additional Yale Law clerk on the likelihood of having an opinion questioned or reversed? ? would simply compare for each judge the rate of negative signals in the second period minus the rate in the first period with the percentage of Yale Law clerks in the second period minus this percentage in the first term. A positive relationship might be taken as suggesting that more Yale Law students led to an increase in negative signals and a negative relationship would suggest the opposite ? negative signals fall as Yale Law clerks area added.

While this test would be better than what Barondes did, it is still not the best approach. To see why, note that Barondes shows that the 95 judges had more Yale law clerks in the second period than in the first (5 percent versus 2 percent). My prior suggested approach would control for the constant or fixed effect of each judge on the likelihood of negative ratings, but it would not control for any varying effect. Judges were getting older over the course of the two time periods, and they were also gaining experience. Might these factors influence the rate of negative signals? They well might, suggesting that a control for judge?s age should likely be included, which Barondes fails to do.

A 35 year old judge at the start of the first period may be getting better (and therefore less likely in the second time period to make mistakes that could lead to a negative signal) while a much older judge may be slowing down and thus experience an increase in errors in the second time period. There also may be constant effects each year (or each period) that make it more likely that cases written at that time will be reversed, and a control for those time fixed effects would be helpful. Only when the fixed effect of the judge, as well as the aging effect and time fixed effects have been controlled for, would we expect to be able to identify the impact of having a Yale law clerk on the rate of negative signals. Yet, Barondes controls for none of these factors.

With the large jump in Yale clerks in the second period, if aging or time effects (such as the switch from the Clinton to the Bush Administration which occurs during his second time period) could well be driving up reversals, which happen to correlate with the greater number of Yale Law clerks in the second time period. Thus, Barondes? Yale Law clerk dummy is picking up three effects ? aging of the judge, change in political administration and other time period effects, and any influence of the Yale law clerk ? when he only wants to capture the last. Since Barondes also fails to correct for the dominant influence of the fixed attributes of the judge (judicial philosophy, political affiliation or ideology), the noise in Barondes? coefficient estimate is great relative to the signal that he hopes to capture.

For those who thirst for econometric terminology, Barondes should have run a panel data model with judge and year fixed effects to see if the presence of Yale clerks influences the negative signal rate. (For example, if the judges with no Yale clerks had a 10 percent decline in their negative signal rate across the two periods (perhaps because they gained experience and were better judges), then if we saw a similar 10 percent decline in the negative signal rate of the judges who increased (or decreased) their number of Yale clerks, we would concluded that the clerks were not influencing the judges. On the other hand, if the judges who increased their hiring of Yale clerks over the period experienced a larger decline in the negative signal rate ? say 15 percent ? then we would conclude that having Yale clerks reduces the negative signal rate.) I would be extremely surprised if his result held up in such a model. Instead, his model simply correlates the negative signal rate with the presence of Yale clerks, which leaves us with the same problem of correlating high executions with high murder rates. We can have no confidence that a causal relationship was identified.

The probing reader might respond to this discussion by contending that it suggests that the problem is not that Yale clerks degrade the quality of judicial opinions but that judges who select Yale clerks are themselves defective. At this point another deficient aspect of the Barondes paper must be acknowledged: the paper is insufficiently nuanced in its failure to note that the term "negative" is imprecise. Indeed, the Shepherd?s Signals can identify two very different phenomena. This identifier might signal errors in understanding legal doctrine or following precedent -- the Barondes? meaning -- as well as outcomes that would not necessarily be seen as pejorative. For example, a liberal judge like Judge Jon Newman in the early 1970s might have gotten lots of "negative" ratings as he was trying to push the law in a direction about which the Burger court was skeptical. Yet Newman is universally acknowledged to be an outstanding judge (who hired many Yale clerks). Ironically, then, some of the best judges might have high negative ratings. If these are the judges who are selecting Yale Law students (as Newman for one tended to do), then the true fact would be "pioneering judges tend to select Yale Law students," instead of "if you want to get reversed, choose a Yale Law clerk." The story is really that certain judges choose Yale Law students, not that ill-informed law clerks are leading judges into error. If Yale clerks tend to be liberal and prefer to work for liberal judges at a time when the Supreme Court is or becomes more conservative, it is not surprising that a somewhat higher proportion of opinions by judges have Yale clerks would be questioned by other (more conservative) judges.

Another way to highlight the dangers of the simple cross-section regression that Barondes runs is to consider a hypothetical (poorly designed) study of police effectiveness. The empiricist looks at all 911 calls and measures bad outcomes that occur during each police response. The bad researcher notices that in the calls for officer assistance in which the police commander sent the SWAT team in to deal with the problem, more people died than when the commander sent one of his school crossing guards. From this, the bad researcher then publishes a study with the headline: Want to deal with that hostage crisis? Forget the SWAT team and send Harry the crossing guard. Of course, it is the deadly situation that leads to the SWAT team being sent and to the high risk of fatalities; the SWAT team doesn't create the trouble, but is trying to deal with it. Conversely, no one is ever killed when Harry shows up because he is usually asked to go over when a cat is stuck up in a tree, not when terrorists have taken 35 people hostage and are threatening to blow them up.

Apart from its muddled causal story, the Barondes paper both oversells its own naïve findings and suffers from some other specification defects, in any event. The paper looks at the roughly 13,000 judicial decisions written by 95 federal district court judges (across two time spans of 2 years and four months) who provided school information for at least one law clerk in surveys of clerks done in 1997 and 2001. Roughly 8 percent of these opinions are given a Shepherd's "warning" or "questioned" indication. The author concludes that the likelihood of receiving this negative assessment will rise to 9.5 percent (based on my calculations from Barondes? fn 48), even though a 95 percent confidence interval around this point estimate would include the possibility of a REDUCTION (rather than an increase) in the negative outcomes (because his point estimate is not statistically significant at the .05 level). In other words, the author's own finding is statistically weak.

Moreover, the advice to shun Yale Law students is unpersuasive even if the causal story were not muddled and the statistical evidence were strong rather than weak. The above estimate of a bump up in the "negative" rating assumes that you are selecting one extra Yale Law clerk and holding everything else constant, where some of the other factors in the model really can't be held constant. Specifically, if you are selecting a Yale Law clerk, it means you are not selecting some other clerk. Barondes found that the better ranked the law school from which the clerk comes, the LOWER the negative ratings. Obviously when a judge chooses a Yale Law clerk he or she is getting a clerk from a highly ranked law school. In fact, the "high ranked law school" effect leads to lower rates of negative indicators. Superficially, the paper claims that if you could get a student from a school other than Yale THAT IS AS HIGHLY RANKED AS YALE LAW SCHOOL, then you would get the benefit of the better school effect without the negative effect the author attributes to Yale. One could add controls for other top schools ? perhaps Harvard and Stanford ? and then test whether their estimated effects are different from the Yale control. Again, I doubt there will be a difference in the direction the author states. The provocative title of Barondes? paper suggests that if you want the train to go off the tracks, hire a Yale law clerk, but this is just silly. The estimated effect is small and not robust, even if one accepts the author's interpretation (without the qualification of the better-school effect, which swamps the estimated Yale effect).

Barondes attempts in his Table 5 to deal with the issue raised in the previous paragraph (to be technical -- non-linearity in the quality of law school effect biasing the estimate of the top-ranked law school), but unlike Tables 4 and 6, which show 6 models, the author only shows 1 model in Table 5 (was this because the other models went against his thesis?). The author wants to compare Chicago favorably with Yale, but eyeballing it, I suspect that in almost all the models there is no statistical difference in the estimated effects for these two schools. Moreover, one un-named judge wrote about 1300 decisions (!) in the 4 year 8 month study period (about 10 percent of all the opinions written by the 95 judges), and when that judge is dropped, the Yale effect goes away entirely (Table 6, column 3) or loses significance (Table 4, column 3). (Why not name the judge? That would have been one of the most interesting tidbits of the paper.) In Table 6, column 3 the positive Yale coefficient is actually smaller than the positive Chicago coefficient (although both are statistically insignificant). Again, this suggests to me that judges, not clerks are likely driving the story.

My colleague Roberta Romano notes that Barondes speculates that Yale law clerks may know less legal doctrine because of the school?s famous emphasis on theory. But Romano points out that bar review passage rates would at least give a sense of whether Yale Law students are deficient in acquiring knowledge of legal doctrine. To test this I thought one might look at July 2007 bar passage rates by school for the single largest state. As it turns out, across all non-California law schools with at least 15 applicants, Yale had the highest bar passage rate (94.1 percent). California bar exam takers from the University of Chicago and Harvard did quite well, but their passage rates of 86 and 87 percent were clearly lower than that of Yale students. Yale law graduates are looking better all the time!

There are some features and anomalies that appear in the data that the author does not comment on. First, who are these 95 federal district court judges? Is there something unusual about them? Table 2 suggests that in the early year only 2 percent of their law clerks were from Yale and 1 percent were from Chicago. Four years later the percent had rise to 5 percent Yale and 6 percent Chicago. That is a large jump and should have been examined.

Second, while the paper is not entirely clear, there may be an odd matching of clerks to opinions. Barondes doesn't know with certainty that the Yalies are the ones writing the opinions that get reversed. (To follow in the fanciful vein of the paper, perhaps I should hypothesize that the Yale Law clerks are do dazzling that the other clerks fall apart and mess up more since they know they can't compete with the very best.). Also, it appears that the author looks at judicial opinions from 9/96 - 12/98 and 9/2000 - 12/2002 and links data on the clerks working for what I suspect is 9/96 - 8/98 and 9/2000 - 8/2002. In other words, the judicial decisions four months after the law clerk data ends are attributed to the prior law clerks (presumably on the theory that the previous clerks worked on those cases), but at the start of the clerkship period it is implicitly assumed that all judicial decisions are attributable to the current clerks. Again, no mention is made of this apparent inconsistency.

Third, roughly 15% of the time, cases receive a caution but the author doesn't show those results. Again, one wonders if these results were dropped because Yale effect did not appear there.

Fourth, Barondes controls for an interaction of his second time period and whether the Judge is a Republican appointee and a second interaction of the second time period and Democratic appointee. But when interaction terms are used one must include both of the constituent terms in the model (that is, separate controls for the second time period and for political affiliation). What would be better is to have a second period time dummy, a Republican time dummy, and an interaction of these two terms. See, Brambor, Thomas, William Roberts Clark and Matt Golder, ?Understanding Interaction Models: Improving Empirical Analyses,? Political Analysis (2006) 14:63?82.

In sum, I am confident that a more suitable methodology than the one employed by Barondes would reveal that Yale Law clerks are extraordinarily capable and effective public servants. All judges will likely be pleased to hire them.

Full post as published by Balkinization on April 22, 2008 (boomark / email).