OR PHONE (866) 635-1838 for Bankruptcy Help, (866) 635-6190 for Divorce,
(866) 635-2689 for Personal Injury or (866) 635-9402 for Criminal Defense

Find a Local Lawyer

Close This Box!

During my first year as a economics graduate student, I spent at most two minutes thinking about the philosophy behind empirical work. On the first day of my year-long econometrics sequence, our professor quickly reminded us that hypotheses cannot be proven, only disproven. That was it. I don't even think Karl Popper's name came up. This is simply not an issue that social scientists wrestle with. Which is a problem, since what we do is not what we think we do.

We think we are engaged in Popperian refutation. Popper's theory is relatively simple to explain: Induction and confirmation are impossible, and all we can do is refute hypotheses. In other words, it is impossible to prove that all swans are white, no matter how many white swans I see. In fact, to Popper, each additional white swan provides no additional confirmation of that hypothesis. But all it takes is one black swan to refute the hypothesis. To Popper, then, science consists of proposing daring hypotheses, testing them, rejecting them, and then moving on to the next, and even more daring, hypothesis. The best thing one can ever say about a theory was that it has not yet been refuted: surviving a test does not, to Popper, provide any indication of accuracy.

The appeal of Popper's apporach is that it avoids the problems of induction, known to us since the time of David Hume. Popper's is a purely deductive logic. Our theory makes prediction X, we see that X is not so, so our theory is--and logically must be--wrong. (Logically speaking, this is modus tollens--the first time I've used that word since math class during my junior year in high school.)

There are, of course, some obvious problems with Popper's approach on its own terms. For example, he didn't entirely believe it himself (Susan Haack's work on Popper lays out his internal contradictions well). At one point he talks about science as being a cathedral built brick by brick over the centuries. But his theory is more about dynamiting that cathedral over and over and over again. And it may be impossible to know what you've refuted or why. Any meaningful scientific theory is a collection of assumptions and sub-theories. If the model predicts 6 and returns -12, it has been refuted, but what exact is the "it"? What part of the theory is wrong? Popper gives us little guidance here.

But the real problem for me is that the social sciences are not engaged in the Popperian endeavor. If nothing else, our theoretical models are incompatible with it. Compare criminology to physics. In criminology, we may be able to make a guess about the direction of the effect, but that is all: "more people in prison will lead to less crime" is the best we can do. There is no theoretical reason to say "a 4% increase in the prison population will lead to a 7% decrease in crime." Physics theory, however, can do just this. The Big Bang model, for example, predicts that cosmic background radiation will have a particular pattern--not "it will get louder for a while then softer for a bit," but the precise pattern it should hold. So the Big Bang theory can be tested: observe the patterns of background radiation and see if it rejects the prediction. NASA did just this with the CoBE satellite. The observed data fit the theoretical predictions almost perfectly; had it not, physicists would have known that some part of the Big Bang theory was wrong. 

Physics produces genuinely testable predictions. The social sciences do not.

When we take a closer look, the Potemkin Village-ness of our allegedly Popperian approach becomes even clearer. Popper calls on us to test ever-more-daring hypotheses. We not only test the same hypothesis over and over--that of no effect--but the hypothesis we test is the least daring one possible. Superficially, it looks like we're doing what Popper asked: we're trying to reject our hypothesis. But unlike the physicists, we actually want to reject it. (The physics community was very happy the CoBE results were what they were.) 

We've inverted the whole point of the process, in doing so we are trying to disguise confirmation as rejection.

And, most obviously, we treat our results as meaning something more than just that the null hypothesis was wrong. We draw policy conclusions from them. In effect, we treat our coefficients as confirming a true effect size, not as refuting the hypothesis of no effect. And maybe we have to. After all, in many cases the null hypothesis must be wrong. An increase in the prison population must have some effect on crime. Maybe a small one, maybe one too small to be detected in our data given our current techniques, but there must be an effect. In other words, from the very start we essentially know that our hypothesis is wrong, so refutation can't really be the goal of the process. And it isn't. What we're trying to do is estimate an effect size. And this is induction.

Perhaps this seems like a semantic debate. Okay, fine, we don't do what Popper tells us to do. But we've created some sort of hybrid system that works for us. Does it matter what we call it?

Absolutely. Because the hybrid system we have come up with doesn't actually work.

By thinking we're engaged in deductive falsification, we adopt a lone-wolf way of determining what we know. After all, it only takes one black swan to refute the hypothesis that all swans are white. So when my paper returns a statistically significant result, my paper has found the black swan. It doesn't matter what other people find. (Of course, especially in the social sciences, our instrumentation is imperfect, so we may want outside confirmation to make sure the swan really is black instead of our dirty instruments making a white swan look black.)

But this approach is completely antithetical to the inductive project we're actually engaged in. Induction requires us to add up a lot of results to put together the big picture. It lacks the logical purity of falsification, but so be it. This is what we do. Too often we hear empiricists say "My study shows that...." Not really. An individual study in an inductive world tells us very, very little--as compared to in a Popperian world, where it can tell us everything. Knowledge does not reside in a single study but in the synthesis of an entire literature. By convincing ourselves that we are falsifying, we relegate rigorous reviews to the second or third tier of research, rather than bringing them to the forefront where they belong.

Susan Haack provides a nice way to think about how we should approach knowledge production in the social sciences. She equates knowledge to a crossword puzzle. A single word, all alone in the grid, is not well-supported (especially if, as most scientific questions likely are, it is a Saturday NY Times). If 35-down is a seven-letter word for "Oceanographers' references," SEAMAPS fits, but we might not be all that confident it is right. But then we look at 46-across, which starts on the M, and we think that MATED could be the right answer for "Like shoes and socks;" we become a bit more sure that SEAMAPS is right. And our faith in MATED grows when we realize that 47-down, which starts on the D, is probably "DOSE," since the clue is "Recommended intake." So DOSE provides warrant for MATED, which in turn warrants SEAMAPS. Once the whole puzzle is filled, we're pretty sure we're right. We could be completely wrong--I've certainly erased entire quadrants of a crossword puzzle before--but the odds are low.

So too with empirical knowledge. When induction is the goal, we must see how all the pieces fit together. Our Popperian pretenses leave us poorly equipped to do this. 

In fact, while fields like epidemiology and medicine are developing rigorous evidence based practices, including powerful techniques to measure study quality and synthesize entire literatures, the social sciences appear to be taking almost no steps whatsoever in this direction. We occasionally dabble in meta-analyses, but no real effort at systematic reviewing has occurred. I may be wrong--one reason for these posts is find out what articles or points I have missed--but as far as I can tell there is no, absolutely no, work in economics on developing quality guidelines or systematic reviews.

Thus the costs of our failure to think about our philosophy of science are great, and the growth of ELS is only exacerbating the problem. I want to conclude by just briefly enumerating two key concerns:

1. A failure to synthesize. Too often we think one study tells us something. And when we do attempt to take a broader view, we tend to do so in a non-rigorous manner. This can lead to serious problems. For example: I attended a conference several years ago in which one of the speakers said that if we added up every paper that said "factor x is responsible for y% of the crime drop in the 1990s," we would see that we've explained about 250% of the crime decline. Everyone laughed but thought little of it. But this is a huge problem: the words in the crossword puzzle are not crossing properly. Something, somewhere, is wrong, but by taking a corpuscular view of empirical evidence, we miss it.

2. We use the wrong tools. I'll have a slightly wonkier post on this issue later today, but by thinking in terms of refutation, we provide the wrong results in our papers. In an inductionist world, p-values and t-statistics mean very little. What matters are confidence intervals, and these are almost never repoted. Moreover, what we need are the right confidence intervals. Those given by Stata and other packages implicitly assume that the null hypothesis is correct. For small effect sizes these intervals may be sufficiently good approximations. But for larger effects, they are increasingly inaccurate, and alternate confidence intervals, build around different distributional assumptions, are needed.

So it should be clear now why we need rigorous reviews, and why our failure to produce them is a fundamental epistemological failure. A single study can refute, but only an overview can confirm. And confirmation is what we do. The explosion in empirical work makes such overviews all the more important, since the larger the literature the harder it is to see the big picture, especially with an ever-growing pool of poorly-designed studies muddying the waters. Review essays are the very heart of empirical knowledge, and they should be treated as such. 

Looking at how to develop these reviews in the social sciences will, not surprisingly, be the focus of several of my upcoming posts.