A paradox

You know those little floaters on the surface of your eyes? They drift into view, catch your attention, then when you try to focus directly on one it disappears from view. They’re only really there if you don’t look straight at them.

Goodhart’s Law tells us that “When a measure becomes a target, it ceases to be a good measure.” The same basic logic applies to two of my favorite things: the Internet and college.

The Internet is still a magical thing, but we’ve killed some of the magic by trying to take the Internet seriously. The Internet ceases to provide output worth taking seriously when people actually take the Internet seriously. When you only keep it in your periphery, it’s actually worth taking seriously

Ditto for college. The basic problem with the current system is that we’re all taking it too seriously. That leads to all sorts of specific bad behavior. But it all comes from this root problem. College is only worth taking seriously if we don’t. When college is back in the ivory tower, separated from the “real” world, it’s a place where people can be creative and make non-obvious connections. But once we recognize “hey, that’s a pretty neat thing, let’s make it a one-size-fits-all solution to all of our problems” we kill the goose that lays the golden eggs.

My advice for getting the most out of the Internet: don’t take it too seriously. It was only ever meant to be a place for weirdos to do weird stuff.

My advice for getting the most out of college: don’t take it too seriously. It was only ever meant to be a place for weirdos to do the sort of stuff that the rest of the world doesn’t have time for.

On the point of quantifying in general and quantifying for policy purposes

Recently, I stumbled on this piece in Chronicle by Jerry Muller. It made my blood boil. In the piece, the author basically argues that, in the world of education, we are fixated with quantitative indicators of performance. This fixation has led to miss (or forget) some important truths about education and the transmission of knowledge. I wholeheartedly disagree because the author of the piece is confounding two things.

We need to measure things! Measurements are crucial to our understandings of causal relations and outcomes.  Like Diane Coyle, I am a big fan of the “dashboard” of indicators to get an idea of what is broadly happening.  However, I agree with the authors that very often the statistics lose their entire meaning. And that’s when we start targeting them!

Once we know that this variable becomes the object of target, we act in ways that increase this variable. As soon as it is selected, we modify our behavior to achieve fixed targets and the variable loses some of its meaning. This is also known as Goodhart’s law whereby “when a measure becomes a target, it ceases to be a good measure” (note: it also looks a lot like the Lucas critique).

Although Goodhart made this point in the context of monetary policy, it applies to any sphere of policy – including education. When an education department decides that this is the metric they care about (e.g. completion rates, minority admission, average grade point, completion times, balanced curriculum, ratio of professors to pupils, etc.), they are inducing a change in behavior which alters the significance carried by this variable.  This is not an original point. Just go to google scholar and type “Goodhart’s law and education” and you end up with papers such as these two (here and here) that make exactly the point I am making here.

In his Chronicle piece, Muller actually makes note of this without realizing how important it is. He notes that “what the advocates of greater accountability metrics overlook is how the increasing cost of college is due in part to the expanding cadres of administrators, many of whom are required to comply with government mandates(emphasis mine).

The problem he is complaining about is not metrics per se, but rather the effects of having policy-makers decide a metric of relevance. This is a problem about selection bias, not measurement. If statistics are collected without an intent to be a benchmark for the attribution of funds or special privileges (i.e. that there are no incentives to change behavior that affects the reporting of a particular statistics), then there is no problem.

I understand that complaining about a “tyranny of metrics” is fashionable, but in that case the fashion looks like crocs (and I really hate crocs) with white socks.