Did the stimulus really destroy a million private-sector jobs?

Hey non-economists, want to see what's inside the guts of one of those econ papers you keep hearing about? Well that's why you have grad student bloggers like me. We read the papers so you don't have to.

This week, Greg Mankiw links us to a paper by Timothy Conley of Western Ontario and Bill Dupor of Ohio State University. The paper's eye-popping finding is that the American Recovery and Reinvestment Act (ARRA), also known as the Stimulus, was responsible for a net loss in jobs. No, really! From the paper's abstract:

Our benchmark results suggest that the ARRA created/saved approximately 450 thousand state and local government jobs and destroyed/forestalled roughly one million private sector jobs...The majority of destroyed/forestalled jobs were in growth industries including health, education, professional and business services.

Wow! Seriously? The stimulus directly resulted in a net loss of five hundred and fifty thousand jobs? That's it, I'm voting Republican from now on...

But wait. A still small voice is nagging me from the back of my mind, urging me to read beyond the abstract. And so my dissertation will have to wait 40 minutes while I wade through three dozen pages of PDF in search of an answer to my nagging doubts.

Because I do have some doubts about this result. Stimulus spending destroys jobs? How the heck is that supposed to work? I mean, maybe you believe in full Ricardian Equivalence, but that would just predict that stimulus is a wash. Perhaps people are cutting back spending in anticipation of the deadweight losses caused by the future taxes needed to pay back the stimulus-related borrowing*? Hmm, maybe, but that sounds so preposterous that I kind of expected something to be fishy about this paper from the get-go.

What Conley and Dupor do is to run a state-by-state regression. Different states received different amounts of ARRA spending, so looking at the differences in employment growth rate between those states after the passage of ARRA should tell us how many jobs ARRA created or destroyed. This should lead to a regression of the type:

Employment growth = A + B*Stimulus + C*Other Stuff + e

Now, you may say: "Wait, but states where employment goes down should be expected to get more stimulus money, since those are just the states that were hardest-hit by the recession!" And you'd be right: there is a big endogeneity problem here. After all, the fact that there's a bunch of sick people in the doctor's office doesn't mean that doctors make you sick. Messrs. Conley & Dupor deal with this problem by finding some "instruments" - natural sources of variation in the amount of stimulus money that a state gets, that have nothing to do with how bad the state's economy was doing. Usually, critics of an empirical paper like this will try to say that the instruments used are bad ones - that they actually can be affected by the business cycle, or that they don't give rise to enough variation in stimulus funding. 

I am not going to do that. I am going to give Conley & Dupor a free pass on their instrumental variables, because I already see one and possibly two gaping hole in their analysis that makes the instrument problem somewhat of a sideshow.

(Update: I had initially written about a second possible problem with this paper, but commenter Ivan found evidence that (thankfully) that problem didn't exist. So, in the interests of not making people read several pointless paragraphs, I've deleted the section that was previously here. Thanks, Ivan!!)

On page 20 of their paper (Table 4), Conley and Dupor have a table that shows their main result: the number of jobs that they estimate to have been created or destroyed by the stimulus. In all private sectors, the estimates are negative. BUT, check out the confidence intervals in Table 4. With one exception, the upper limits of all the confidence intervals are highly positive. This despite the fact that they use a less-rigorous 90% confidence interval (instead of the standard 95%).

This means that Conley and Dupor's results are statistically insignificant. Bluntly, what they have found is nothing. Formally, if we use their model to test the hypothesis that the stimulus caused a net increase in private-sector jobs, we will not be able to reject the hypothesis.

Conley and Dupor tweak their model with some alternative specifications. No change. As you can see in Table 7 and Table 9 (p.23-4), upper 90% confidence limits continue to be strongly positive. If the authors really did leave the intercept term in their regression equation, then that's probably why they got insignificant results; if not, then there's some other problem with their instruments or their specification, or maybe just the data itself.

But, given the lack of any statistically significant findings, this paper does not deliver the results that it advertised. Conley and Dupor's abstract should read "We find no evidence for a significant effect of the ARRA on job creation." That would be scientifically honest, but would not turn a lot of heads. Instead, the abstract makes the more politically incendiary claim that the ARRA destroyed jobs, which the authors actually did not find. They do leave themselves an escape rout by using the word "suggest," but I am not satisfied. In my opinion this is a paper that overstates its findings. (Note: Conley and Dupor have since revised their abstract significantly to more accurately reflect their results, for which I commend them!)

My guess is that papers like this get attention because of politics, not because of science. Dr. Mankiw linked to this paper without comment, evaluation, or qualification. But he could have just as easily linked to this paper by Daniel J. Wilson, which uses a methodology similar to that of Conley and Dupor, but finds strongly positive (and often strongly significant) effects of the stimulus.

Update: Arnold Kling is also not a fan of Conley-Dupor. 

* Actually, it's worse than that. You have to also assume that future deadweight losses from stimulus-payback taxation will be highly concentrated in the states that received the most stimulus funding; i.e., that taxes will be specifically targeted at those states!