The Counterfactual and the Factual

Mark Koyama Economics, History, , ,

Historians often appear skeptical of counterfactual arguments. E.H. Carr argued that “a historian should never deal in speculation about what did not happen” (Carr, 1961, 127). Michael Oakeshott described counterfactual reasoning as ‘a monstrous incursion of science into the world of history’ (quoted in Ferguson, 1999). More recently, Eric Foner is reported to have found “counterfactuals absurd. A historian’s job is not to speculate about alternative universes …It’s to figure out what happened and why” (cited in Parry, 2016, here).

Such skepticism is striking to the modern economic historian, who since Robert Fogel’s work on the impact of the railroad on American economic growth has been trained to think explicitly in terms of counterfactuals. Far from being the absurdity Foner suggests, counterfactuals represent the gold standard in economic history today. Why? Because they are the sine qua of causal analysis. As David Hume noted, a counterfactual is exactly what we invoke whenever we use the word “cause”: “an object, followed by another, . . . where, if the first object had not been, the second would had never existed” (Hume, 1748, Part II).

Hume’s reasoning can best be understood in the context of a controlled experiment. Suppose a group of randomly selected patients are treated with a new drug while another randomly selected group are assigned a placebo. If the treatment and control groups were ex ante indistinguishable, then the difference between the outcomes for these two groups is the causal effect of the drug. The outcome for the control group provides the relevant counterfactual which enables us to assess the effectiveness of the drug.

The modern revival of economic history is based largely on the skill with which economic historians have been able to use econometric tools to replicate this style of experimental design using observational data. Such techniques enable economic historians to assess such counterfactuals as how much did slavery contribute to Africa’s underdevelopment?, what was the impact of the Peruvian Mita? or the effects of the Dust bowl?

The rejection of the counterfactual approach by historians such as Foner seems to run deep and constitutes a major divide between historians and economic historians; it is therefore well worth exploring its source.

To begin with, let’s set aside some of the reasons why historians have dismissed counterfactuals in the past. We need not, for instance, pay too much attention to the attachment of Marxists (like Carr) and Hegelian idealists (like Oakeshott) to teleological history. Of course, if history represents the unfolding of a dialectical process, then events that did not occur cannot, by definition, constitute the subject of historical analysis. Crude Marxism (and Hegelianism) is, I hope, still out of favor. But another reason why historians are skeptical of the counterfactual seems better grounded. And this is historians’ attachment to the factual.

Consider, Niall Ferguson’s edited volume Virtual History. It provides an excellent defense of counterfactual history. The counterfactuals considered by Ferguson and co, however, are largely in military or diplomatic history: what would have happened had the Nazis’ invaded Britain? etc.

These counterfactuals are a useful way to think through a question. But their power typically depends on reversing a single decision or event, i.e. suppose Hitler doesn’t issue his Stop Order in June 1940 or Edward Grey decides not to defend Belgium neutrality, what then? To be plausible everything else has to be held constant. This means that counterfactuals in diplomatic and military history shed light on the short term consequences of particular events. But the ceteris paribus assumption becomes harder to maintain as we consider events further removed from the initial counterfactual intervention. Thus, we have a reasonable idea of what Nazi rule of Britain in 1940 might have looked like — with the SS hunting down Jews, liberals, and intellectuals and restoring Edward VIII to the throne. But once we consider the outcomes of a Nazi ruled Britain into the 1950s and 1960s, we have much less guidance. Lacking any documentary evidence of the intentions of Britain’s Nazi rulers in the post-war era leaves us in the realm of historical fiction like Robert Harris’ Fatherland or CJ Sansom’s Dominion; there are simply too many degrees of freedom to do conduct historical analysis. Counterfactuals become problematic once we run out of facts to discipline our analysis.

This is the one fact it a valid reason for historians to be skeptical of counterfactuals. The actual historical record has to serve as a constant constraint on historical writing. This goes back to Leopold von Ranke, the scholar responsible for history’s emergence as an academic discipline in the 19th century. Ranke and his followers insisted on rigorous documentation and established the idea that the craft of the historian lay in the discovery, assembly, and analysis of primary sources. Ranke urged historians to focus on what actually happened; simply put, the facts ma’am, just the facts. Many criticisms have been levied at Ranke in the intervening 150 years, and to jaded post-modern eyes this approach no doubt appears hopeless naïve. But we should not dismiss Ranke’s strictures too quickly given what happens when historians abandon them (here and here). What is important here is that the same Rankian strictures that helped form history as an academic discipline, also rule out speculating about things that didn’t happen. They instill in historians a natural skepticism of counterfactual, alternative, history.

Moreover, while military history lends itself naturally to counterfactual analysis, other areas of history such as social or economic history where change is typically more gradual appear less suitable. After all: how is one to assess such complex counterfactuals as the fate of slavery in the US South in the absence of the Civil War?

These are questions which benefit from counterfactual reasoning but which, unlike diplomatic, political or military history, often requires training in the social sciences to answer. For example, take a question that is of interest to historians of capitalism: would slavery have disappeared quickly without the civil war?

From the 1950s to the 1970s, cliometric historians utilized economic theory to try to answer this. They employed economic models to assess the profitably of slavery and to infer the expectations of slave owners in the south (here). The main finding was that, contrary to the suppositions of historians (who at the time were often sympathetic to the southern cause): slavery was extremely profitable in 1860 and slaveholders foresaw the institution lasting indefinitely. In this case, their use of counterfactual reasoning overturned the previous historical orthodoxy.

The issue of the economic importance of slavery to the American economy in the early nineteenth century is also a counterfactual question. Implicitly it asks what would GDP have been in the absence of the slave-produced cotton. Here it is not only economic historians who are making counterfactual arguments. Foner championed Ed Baptist’s book The Half Has Never Been Told. But in it, Baptist argued that almost 50% of GDP in 1836 was due to slavery, itself a counterfactual argument. He is arguing that, in the absence of slavery, the American economy would have been roughly half the size that it was. This claim is certainly false based as it is on double-counting. But the problem with Baptist’s argument is not that he had made a counterfactual claim, but that he conducted counterfactual analysis ineptly and that his estimates are riddled with errors (see here and here).

All of this sheds light on why counterfactuals are so often dismissed by historians. There is an important and deeply shared sense that the counterfactual approach is ahistorical and an unfamiliarity with the techniques involved. A natural lesson from the Baptist affair is that historians should become more familiar with the powerful tools social scientists have to assess counterfactual questions. Taking counterfactuals seriously is a way to make progress on uncovering answers to important historical questions. But there is also a sense in which the historians’ suspicion of counterfactual may be justified.

There remain many questions where counterfactuals are not especially useful. The more complex the event, the harder it is to isolate the relevant counterfactual. Recently Bruno Gonçalves Rosi at Notes on Liberty suggested such a counterfactual: “no Protestant Reformation, no freedom of conscience as we know today”.

But in comparison to what we have considered thus far, this is a tricky counterfactual to assess. Suppose Bruno had said, “no Martin Luther, no freedom of conscience as we know it today”. This would be easier to argue against as one could simply note that absent Luther there probably won’t have been a Reformation starting in 1517, but at some point in the 1520s-1530s, it is likely that someone else would have taken Luther’s place and overthrown the Catholic Church. But taking the entire Reformation as a single treatment and assessing its causal effect is much harder to do.

In particular, we have to assess two separate probabilities: (i) the probability of freedom of conscience emerging in Europe in the absence of the Reformation (P(Freedom of conscience|No Reformation)); and (ii) the probability of freedom of conscience emerging in Europe in the presence of the Reformation (P(Freedom of conscience| Reformation)). For Bruno’s argument to hold we don’t just need P(FC|R) > P (FC|NR), which is eminently plausible. We also need P(FC|NR) to equal zero. This seems implausible.

The problem becomes still more complex once one recognizes that the Protestant Reformation was itself the product of economic, social, political and technological changes taking place in Europe. If our counterfactual analysis takes away the Reformation but leaves in place the factors that helped to give rise to it (urbanization, the printing press, political fragmentation, corruption etc.), then it is unclear what the counterfactual actually tells us. This problem can be illustrated by considering a causal diagram of the sort developed by Judea Perle (2000).

Here we are interested in the effect of D (the Reformation) on Y (freedom of conscience). The problem is that if we observe a correlation between D and Y, we don’t know if it is causal. This is because of the presence of A, B, and F. Perhaps these can be controlled for. But there is also C. We can think of C as the printing press.

The printing press has a large role in the success of the Reformation (Rubin 2014). But it also stimulated urbanization and economic growth and plausibly had an independent role in stimulating the developments that eventually gave rise to modern liberalism, rule of law, and freedom of conscience. The endogeneity problem here seems intractable.

Absent some way to control for all these potential confounders, we are unable to estimate the causal effects of the Protestant Reformation on something like freedom of conscience. In contrast to the purely economic questions considered above, we don’t have a good theoretical understanding of the emergence of religious freedom. Counterfactual reasoning only gets us so far.

Historians need economic history (and this means economic theory and econometrics). And economists need historians. They need historians to make sense of the complexity of the world and because of their expertise and skill in handling evidence.

9 thoughts on “The Counterfactual and the Factual”

  1. It seems to me that stating any factual involves necessarily making a counterfactual. What is the difference between saying that “X happened because of Y” and saying “no Y, no X”? Other than that, what is the problem in saying “no Martin Luther, no freedom of conscience as we know it today”? As history happened, Luther started the Reformation. No Martin Luther, no Reformation, no freedom of conscience as we know today. How can you possibly say that “this would be easier to argue against”? How can you possibly say that “one could simply note that absent Luther there probably won’t have been a Reformation starting in 1517, but at some point in the 1520s-1530s”? Really? How do you calculate probabilities in a scenario like that? The only history we have is the history that we have. And in the history that we have Martin Luther started the Protestant Reformation. To finish, how can you tell that “the Protestant Reformation was itself the product of economic, social, political and technological changes taking place in Europe”? Really? Had it nothing to do with theological developments long in development? Give me a break.

    Reply

    • “What is the difference between saying that “X happened because of Y” and saying “no Y, no X”?” The difference is if you say “no Y, no X” you are saying that X could only happened if Y happened and nothing else could cause X to happen. But saying “X happened because of Y” doesn’t mean that Y is the only cause. It just means it is the proximate cause.

      So when you say “Luther started the Reformation. No Martin Luther, no Reformation, no freedom of conscience as we know today,” you’re first sentence is of the nature “X happened because of Y” and your second sentence is of the nature “no Y, no X”. And given the number of proto-reformers before Luther and the number of contemporary reformers in Luther’s time (e.g. Zwingli), it seems a bit daft to say only Luther could have caused the Reformation. I think there is something to “great man” theories in history and the power of various personalities, but I’m just not sure how far you can push it in the case of the Reformation.

      And when Mark says, “the Protestant Reformation was itself the product of economic, social, political and technological changes taking place in Europe”, I don’t think he’s denying that theological developments had nothing to do with the Reformation (Mark’s far too careful a thinker to make such a claim). All he’s saying is that the Reformation was underpinned by many causes. And if you are trying to make a causal claim like “the Reformation caused freedom of conscience” (i.e. No Reformation, No Freedom of Conscience), you actually have to show why all those other factors wouldn’t have independently led to freedom of conscience.

      “No Martin Luther, no Reformation, no freedom of conscience as we know today. How can you possibly say that “this would be easier to argue against”?” It’s easy to say this. It is easier to argue against “No Luther, No Reformation, No Freedom of Conscience” than “No Reformation, No Freedom of Conscience” because the causal chain is shorter.

      As a last observation, I am actually surprised that any of the observations in Mark’s post (and also (I imagine) in this comment) are controversial. It shows a deep need of not just of social science education for historians, but also elementary courses in logic.

    • JFA made exactly the two points I would have made in response.
      I think the Reformation mattered a lot. But how it mattered is itself a complex question (given the difficulties of doing counterfactual analysis on complicated historical processes).

    • I was reading your answer with great interest and a varying level of agreement and educated discordance. Until I came to the last paragraph and I noted you just want to offend me implying I’m not well educated in social science and logic. So I believe I can dis-consider everything else. Good day, sir. Just as a last observation for Mark (not for you), I started this conversation saying that I’m really not sure how much we can identify with confidence causality in history. But with all things considered, it seems to me a fair statement that the Reformation made a great contribution to the development of freedom of conscience as we now today. Definitely, after some fight, that’s true, reformed theologians came to the position of accepting disagreements. That was the whole point. But I believe I had enough of this conversation for now.

    • Mr. Rosi,

      I honestly wasn’t singling you out and did not mean it as an insult. I think the unawareness of logical rules and social science techniques are something endemic to (most of) academic history (which Mark pointed out the latter problem but I think the plethora of logical fallacies I see in academic history is also damning (see Beckert’s theory of war capitalism for a prime example)). What you see in a lot of history writing is captured in your statement “The only history we have is the history that we have.” This leads to a lot of history writing that is of the “post hoc ergo propter hoc” variety. What many historians seem to do is to follow the existing causal chain and assume that is the only way things could turn out, which leads to sweeping claims like “the US economy would have been far smaller if it had not been for [insert product or production process here]”.

      Your statement “it seems to me a fair statement that the Reformation made a great contribution to the development of freedom of conscience as we now today” is vastly different from “No Martin Luther, no Reformation, no freedom of conscience as we know today”. (As a PhD in Poli Sci and having taught Methodology of Scientific Research, I’m sure you can see the difference.) I think you can defend the former with gusto and tell an interesting story that links various historical events. I don’t think you can defend the latter statement. If you were to try, you would have to bring the best tools of social science to bear, and I don’t think (most) historians get that type of training or even realize they need it.

Please keep it civil