On the point of quantifying in general and quantifying for policy purposes

Recently, I stumbled on this piece in Chronicle by Jerry Muller. It made my blood boil. In the piece, the author basically argues that, in the world of education, we are fixated with quantitative indicators of performance. This fixation has led to miss (or forget) some important truths about education and the transmission of knowledge. I wholeheartedly disagree because the author of the piece is confounding two things.

We need to measure things! Measurements are crucial to our understandings of causal relations and outcomes.  Like Diane Coyle, I am a big fan of the “dashboard” of indicators to get an idea of what is broadly happening.  However, I agree with the authors that very often the statistics lose their entire meaning. And that’s when we start targeting them!

Once we know that this variable becomes the object of target, we act in ways that increase this variable. As soon as it is selected, we modify our behavior to achieve fixed targets and the variable loses some of its meaning. This is also known as Goodhart’s law whereby “when a measure becomes a target, it ceases to be a good measure” (note: it also looks a lot like the Lucas critique).

Although Goodhart made this point in the context of monetary policy, it applies to any sphere of policy – including education. When an education department decides that this is the metric they care about (e.g. completion rates, minority admission, average grade point, completion times, balanced curriculum, ratio of professors to pupils, etc.), they are inducing a change in behavior which alters the significance carried by this variable.  This is not an original point. Just go to google scholar and type “Goodhart’s law and education” and you end up with papers such as these two (here and here) that make exactly the point I am making here.

In his Chronicle piece, Muller actually makes note of this without realizing how important it is. He notes that “what the advocates of greater accountability metrics overlook is how the increasing cost of college is due in part to the expanding cadres of administrators, many of whom are required to comply with government mandates(emphasis mine).

The problem he is complaining about is not metrics per se, but rather the effects of having policy-makers decide a metric of relevance. This is a problem about selection bias, not measurement. If statistics are collected without an intent to be a benchmark for the attribution of funds or special privileges (i.e. that there are no incentives to change behavior that affects the reporting of a particular statistics), then there is no problem.

I understand that complaining about a “tyranny of metrics” is fashionable, but in that case the fashion looks like crocs (and I really hate crocs) with white socks.

More Musings on Colonialism

I recently attended an excellent lecture at Cabrillo College, located in central California, by an International Relations scholar who focused on the effects of colonialism. We took a solid look at the ‘World Systems Theory’ of why the developing world is, well, developing, and it was great to go over this school of thought’s main arguments.

For those of you who don’t know, World Systems Theory is a Marxian analysis that basically states that poor countries are poor because of the effects of colonialism, and the evidence supporting their claims is pretty damn solid. Basically, the World Systems theorists argue that when the various European powers gained outright control of non-European lands (this process in itself took centuries, by the way, and I deplore the historical narrative that argues Europeans set out to conquer foreign lands and divide up the spoils of war for reasons outlined in the link provided), the European powers set up states that were designed specifically to export raw agricultural materials to European factories, to be produced by European workers, and to be consumed by European (and elite non-European) consumers.

This is pretty much what happened, and explains why most of the developing world is dependent upon raw commodity exports (that are shipped to European markets) for most of their well-being. Unfortunately, the very solutions that the World Systems theorists propose to dismantle the structural inequalities that exist in this world will (and have) actually led to more of the same structural inequality.

Allow me to explain. Continue reading