What makes it science?

When I hear the phrase “I experimented with drugs/diet/habit/whatever [on myself],” I tend to call bullshit. (A good exception is the author at Gwern.net who does blind, randomized trials on himself sometimes.) Without a control group you aren’t doing an experiment.

But I heard some interesting phrasing that is making me reconsider. Scott Adams was talking about experimenting with changes to his diet by isolating one thing and seeing if he can observe a change after a week. It’s clear that he understands the limitations of this approach. And that clarity makes me think that he’s really properly experimenting. He’s not going so far as running a double-blind study, he’s just taking a serious look at imperfect evidence and being epistemically honest.

Like any good scientific thinker of our time, Adams knows that the outcomes he observes can be affected by any number of variables he’s failed to account for. He knows that his estimates need an error term. He almost certainly knows that time isn’t on his side and ever so slightly affects his results. He almost certainly also knows that path-dependency plays a role. But he corrects for all that in his interpretation. It’s this considered approach to the evidence that makes me view him as operating on a scientific basis. So even if his trials do not provide powerful evidence, his interpretation and application of the evidence is what makes it science.

I suppose this would mean that an experiment can’t be considered scientific until the data is interpreted.

And of course all of this is to say that he’s definitely right about Donald Trump.

4 thoughts on “What makes it science?

  1. Experiment: combine metallic sodium and water. Result: a violent exothermic reaction. No control group needed, no interpretation.

  2. Warren makes a good point about the physical sciences versus the social sciences here, and one that can sometimes grate social scientists. How scientific is social science, after all?

    I’m too lazy and too unqualified to answer that in this thread, but I hope the question is one we’ll continue to come back and revisit, or at least keep in the back of our minds.

    • To add on to Warren’s point:

      Sometimes the results of a trial are so clear that you only need one data point. More complex phenomena would ideally come with more data points, but our most complex phenomena (society) only yield highly autocorrelated data so we need to substitute statistical clarity for wishy-washy interpretation of historical facts.

      In the two-dimensional space of sample-size/causal-obviousness, the ease of learning the truth is highest on the 45-degree line. More complex phenomena require more observations. And in all cases effort must be put into pulling the truth out of the data (in the case of metallic sodium + water, interpretive-effort=epsilon). The further we get away from that 45-degree line, the more effort must be put into interpretation, and the less certain the results.

      Any fact or theory might be certain enough for practical application, even if the science behind it isn’t perfect. But I’m thinking like an engineer; facts and theories often work like Keynes’s beauty contest. The working version of the truth is the one that most other people think most other people want to see.

Please keep it civil

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s