When I hear the phrase “I experimented with drugs/diet/habit/whatever [on myself],” I tend to call bullshit. (A good exception is the author at Gwern.net who does blind, randomized trials on himself sometimes.) Without a control group you aren’t doing an experiment.
But I heard some interesting phrasing that is making me reconsider. Scott Adams was talking about experimenting with changes to his diet by isolating one thing and seeing if he can observe a change after a week. It’s clear that he understands the limitations of this approach. And that clarity makes me think that he’s really properly experimenting. He’s not going so far as running a double-blind study, he’s just taking a serious look at imperfect evidence and being epistemically honest.
Like any good scientific thinker of our time, Adams knows that the outcomes he observes can be affected by any number of variables he’s failed to account for. He knows that his estimates need an error term. He almost certainly knows that time isn’t on his side and ever so slightly affects his results. He almost certainly also knows that path-dependency plays a role. But he corrects for all that in his interpretation. It’s this considered approach to the evidence that makes me view him as operating on a scientific basis. So even if his trials do not provide powerful evidence, his interpretation and application of the evidence is what makes it science.
I suppose this would mean that an experiment can’t be considered scientific until the data is interpreted.
And of course all of this is to say that he’s definitely right about Donald Trump.