counterfactuals

Nightcap

Brandon Christensen Links , , ,
  1. What counterfactuals (don’t) tell us Chris Dillow, Stumbling & Mumbling
  2. The counterfactual and the factual Mark Koyama, NOL
  3. Anti-clerical movements in Mexico Madeleine Olson, Not Even Past
  4. A World Cup for the world’s stateless Pete Kiehart, ESPN

The Incomplete Counterfactual Fallacy

Rick Weber Economics, Liberty, Political Thought, Politics , , , ,

Mariana Mazzucato has some interesting ideas, but her basic thesis (which I’m guessing based only on her recent Freakonomics interview) forgets a simple fact: had the government not made GPS, not only would we not have GPS, resources would have been allocated somewhere else.

In other words, she’s right that without government involvement we’d miss all sorts of valuable things like the basic research it turns out government has financed, physical infrastructure (no matter how neglected), GPS, the specific form the Internet we got, etc., etc.

But that’s only half the story. What happens if DARPA closes shop just before starting on GPS? Those engineers end up somewhere else, the money for the project goes somewhere else, the steel and titanium for the satellites they would have built end up in some other shop.

If we could know that nothing else would have happened at that point, then GPS is a free lunch–some bold committee scooped up a handful of clay, handed it to brilliant engineers, and created something from nothing.

More likely some of those engineers would have made dozens of other projects a little bit better, and one or two of them might even have, through some series of unpredictable events, ended up creating some totally different innovation that would have put us on to an entirely different path.

Which is the better path? There’s no way to know. The people on that other path exist in a completely different paradigm from our own. It’s comparing meta-apples with meta-oranges. We are where we are and we ought to appreciate all the costs and benefits of the system we’ve inherited. I think Mazzucato is right that many of us have underrated the government. (Certainly 23-year-old-Rick was guilty of the over-complete counterfactual fallacy.)

She makes a compelling argument that Solyndra doesn’t look like such a boondoggle when you take the wider view we would expect of the smallest venture capital investor. But if it’s true that pointing at government failures is invalid, then so is pointing at government success.

What’s important is improving the future performance of our current institutional mess. To do that, I think we’re better off backing away from her broad claims and focusing on her more sensible (and jarring) arguments about how the current system encourages destructive rent seeking through intellectual property protection of basic research that might be better left in the public domain.

The Counterfactual and the Factual

Mark Koyama Economics, History , , ,

Historians often appear skeptical of counterfactual arguments. E.H. Carr argued that “a historian should never deal in speculation about what did not happen” (Carr, 1961, 127). Michael Oakeshott described counterfactual reasoning as ‘a monstrous incursion of science into the world of history’ (quoted in Ferguson, 1999). More recently, Eric Foner is reported to have found “counterfactuals absurd. A historian’s job is not to speculate about alternative universes …It’s to figure out what happened and why” (cited in Parry, 2016, here).

Such skepticism is striking to the modern economic historian, who since Robert Fogel’s work on the impact of the railroad on American economic growth has been trained to think explicitly in terms of counterfactuals. Far from being the absurdity Foner suggests, counterfactuals represent the gold standard in economic history today. Why? Because they are the sine qua of causal analysis. As David Hume noted, a counterfactual is exactly what we invoke whenever we use the word “cause”: “an object, followed by another, . . . where, if the first object had not been, the second would had never existed” (Hume, 1748, Part II).

Hume’s reasoning can best be understood in the context of a controlled experiment. Suppose a group of randomly selected patients are treated with a new drug while another randomly selected group are assigned a placebo. If the treatment and control groups were ex ante indistinguishable, then the difference between the outcomes for these two groups is the causal effect of the drug. The outcome for the control group provides the relevant counterfactual which enables us to assess the effectiveness of the drug.

The modern revival of economic history is based largely on the skill with which economic historians have been able to use econometric tools to replicate this style of experimental design using observational data. Such techniques enable economic historians to assess such counterfactuals as how much did slavery contribute to Africa’s underdevelopment?, what was the impact of the Peruvian Mita? or the effects of the Dust bowl?

The rejection of the counterfactual approach by historians such as Foner seems to run deep and constitutes a major divide between historians and economic historians; it is therefore well worth exploring its source.

To begin with, let’s set aside some of the reasons why historians have dismissed counterfactuals in the past. We need not, for instance, pay too much attention to the attachment of Marxists (like Carr) and Hegelian idealists (like Oakeshott) to teleological history. Of course, if history represents the unfolding of a dialectical process, then events that did not occur cannot, by definition, constitute the subject of historical analysis. Crude Marxism (and Hegelianism) is, I hope, still out of favor. But another reason why historians are skeptical of the counterfactual seems better grounded. And this is historians’ attachment to the factual.

Consider, Niall Ferguson’s edited volume Virtual History. It provides an excellent defense of counterfactual history. The counterfactuals considered by Ferguson and co, however, are largely in military or diplomatic history: what would have happened had the Nazis’ invaded Britain? etc.

These counterfactuals are a useful way to think through a question. But their power typically depends on reversing a single decision or event, i.e. suppose Hitler doesn’t issue his Stop Order in June 1940 or Edward Grey decides not to defend Belgium neutrality, what then? To be plausible everything else has to be held constant. This means that counterfactuals in diplomatic and military history shed light on the short term consequences of particular events. But the ceteris paribus assumption becomes harder to maintain as we consider events further removed from the initial counterfactual intervention. Thus, we have a reasonable idea of what Nazi rule of Britain in 1940 might have looked like — with the SS hunting down Jews, liberals, and intellectuals and restoring Edward VIII to the throne. But once we consider the outcomes of a Nazi ruled Britain into the 1950s and 1960s, we have much less guidance. Lacking any documentary evidence of the intentions of Britain’s Nazi rulers in the post-war era leaves us in the realm of historical fiction like Robert Harris’ Fatherland or CJ Sansom’s Dominion; there are simply too many degrees of freedom to do conduct historical analysis. Counterfactuals become problematic once we run out of facts to discipline our analysis.

This is the one fact it a valid reason for historians to be skeptical of counterfactuals. The actual historical record has to serve as a constant constraint on historical writing. This goes back to Leopold von Ranke, the scholar responsible for history’s emergence as an academic discipline in the 19th century. Ranke and his followers insisted on rigorous documentation and established the idea that the craft of the historian lay in the discovery, assembly, and analysis of primary sources. Ranke urged historians to focus on what actually happened; simply put, the facts ma’am, just the facts. Many criticisms have been levied at Ranke in the intervening 150 years, and to jaded post-modern eyes this approach no doubt appears hopeless naïve. But we should not dismiss Ranke’s strictures too quickly given what happens when historians abandon them (here and here). What is important here is that the same Rankian strictures that helped form history as an academic discipline, also rule out speculating about things that didn’t happen. They instill in historians a natural skepticism of counterfactual, alternative, history.

Moreover, while military history lends itself naturally to counterfactual analysis, other areas of history such as social or economic history where change is typically more gradual appear less suitable. After all: how is one to assess such complex counterfactuals as the fate of slavery in the US South in the absence of the Civil War?

These are questions which benefit from counterfactual reasoning but which, unlike diplomatic, political or military history, often requires training in the social sciences to answer. For example, take a question that is of interest to historians of capitalism: would slavery have disappeared quickly without the civil war?

From the 1950s to the 1970s, cliometric historians utilized economic theory to try to answer this. They employed economic models to assess the profitably of slavery and to infer the expectations of slave owners in the south (here). The main finding was that, contrary to the suppositions of historians (who at the time were often sympathetic to the southern cause): slavery was extremely profitable in 1860 and slaveholders foresaw the institution lasting indefinitely. In this case, their use of counterfactual reasoning overturned the previous historical orthodoxy.

The issue of the economic importance of slavery to the American economy in the early nineteenth century is also a counterfactual question. Implicitly it asks what would GDP have been in the absence of the slave-produced cotton. Here it is not only economic historians who are making counterfactual arguments. Foner championed Ed Baptist’s book The Half Has Never Been Told. But in it, Baptist argued that almost 50% of GDP in 1836 was due to slavery, itself a counterfactual argument. He is arguing that, in the absence of slavery, the American economy would have been roughly half the size that it was. This claim is certainly false based as it is on double-counting. But the problem with Baptist’s argument is not that he had made a counterfactual claim, but that he conducted counterfactual analysis ineptly and that his estimates are riddled with errors (see here and here).

All of this sheds light on why counterfactuals are so often dismissed by historians. There is an important and deeply shared sense that the counterfactual approach is ahistorical and an unfamiliarity with the techniques involved. A natural lesson from the Baptist affair is that historians should become more familiar with the powerful tools social scientists have to assess counterfactual questions. Taking counterfactuals seriously is a way to make progress on uncovering answers to important historical questions. But there is also a sense in which the historians’ suspicion of counterfactual may be justified.

There remain many questions where counterfactuals are not especially useful. The more complex the event, the harder it is to isolate the relevant counterfactual. Recently Bruno Gonçalves Rosi at Notes on Liberty suggested such a counterfactual: “no Protestant Reformation, no freedom of conscience as we know today”.

But in comparison to what we have considered thus far, this is a tricky counterfactual to assess. Suppose Bruno had said, “no Martin Luther, no freedom of conscience as we know it today”. This would be easier to argue against as one could simply note that absent Luther there probably won’t have been a Reformation starting in 1517, but at some point in the 1520s-1530s, it is likely that someone else would have taken Luther’s place and overthrown the Catholic Church. But taking the entire Reformation as a single treatment and assessing its causal effect is much harder to do.

In particular, we have to assess two separate probabilities: (i) the probability of freedom of conscience emerging in Europe in the absence of the Reformation (P(Freedom of conscience|No Reformation)); and (ii) the probability of freedom of conscience emerging in Europe in the presence of the Reformation (P(Freedom of conscience| Reformation)). For Bruno’s argument to hold we don’t just need P(FC|R) > P (FC|NR), which is eminently plausible. We also need P(FC|NR) to equal zero. This seems implausible.

The problem becomes still more complex once one recognizes that the Protestant Reformation was itself the product of economic, social, political and technological changes taking place in Europe. If our counterfactual analysis takes away the Reformation but leaves in place the factors that helped to give rise to it (urbanization, the printing press, political fragmentation, corruption etc.), then it is unclear what the counterfactual actually tells us. This problem can be illustrated by considering a causal diagram of the sort developed by Judea Perle (2000).

Here we are interested in the effect of D (the Reformation) on Y (freedom of conscience). The problem is that if we observe a correlation between D and Y, we don’t know if it is causal. This is because of the presence of A, B, and F. Perhaps these can be controlled for. But there is also C. We can think of C as the printing press.

The printing press has a large role in the success of the Reformation (Rubin 2014). But it also stimulated urbanization and economic growth and plausibly had an independent role in stimulating the developments that eventually gave rise to modern liberalism, rule of law, and freedom of conscience. The endogeneity problem here seems intractable.

Absent some way to control for all these potential confounders, we are unable to estimate the causal effects of the Protestant Reformation on something like freedom of conscience. In contrast to the purely economic questions considered above, we don’t have a good theoretical understanding of the emergence of religious freedom. Counterfactual reasoning only gets us so far.

Historians need economic history (and this means economic theory and econometrics). And economists need historians. They need historians to make sense of the complexity of the world and because of their expertise and skill in handling evidence.