Broken incentives in medical research

Last week, I sat down with Scott Johnson of the Device Alliance to discuss how medical research is communicated only through archaic and disorganized methods, and how the root of this is the “economy” of Impact Factor, citations, and tenure-seeking as opposed to an exercise in scientific communication.

We also discussed a vision of the future of medical publishing, where the basic method of communicating knowledge was no longer uploading a PDF but contributing structured data to a living, growing database.

You can listen here:

As background, I recommend the recent work by Patrick Collison and Tyler Cowen on broken incentives in medical research funding (as opposed to publishing), as I think their research on funding shows that a great slow-down in medical innovation has resulted from systematic errors in organizing knowledge gathering. Mark Zuckerberg actually interviewed them about it here:

Launching our COVID-19 visualization

I know everyone is buried in COVID-19 news, updates, and theories. To me, that makes it difficult to cut through the opinions and see the evidence that should actually guide physicians, policymakers, and the public.

To me, the most important thing is the ability to find the answer to my research question easily and to know that this answer is reasonably complete and evidence-driven. That means getting organized access to the scientific literature. Many sources (including a National Library of Medicine database) present thousands of articles, but the organization is the piece that is missing.

That is why I launched StudyViz, a new product that enables physicians to build an updatable visualization of all studies related to a topic of interest. Then, my physician collaborators built just such a visual for COVID-19 research, presenting a sunburst diagram that users can select to identify their research question of interest.

Studyviz sunburst

For instance, if you are interested in the impact of COVID-19 on pregnant patients, just go to “Subpopulations” and find “Pregnancy” (or neonates, if that is your concern). We nested the tags so that you can “drill down” on your question, and so that related concepts are close to each other. Then, to view the studies themselves, just click on them to see an abstract with the key info (patients, interventions, and outcomes) highlighted:


This is based on a complex concept hierarchy built by our collaborators and that is constantly evolving as the literature does:


Even beyond that, we opened up our software to let any researchers who are interested build similar visuals on any disease state, as COVID-19 is not the only disease for which organizing and accessing the scientific literature is important!

We are seeking medical coinvestigators–any physician interested in working with us, simply email or contact us on our website!

There is no Bloomberg for medicine

When I began working in medical research, I was shocked to find that no one in the medical industry has actually collected and compared all of the clinical outcomes data that has been published. With Big Data in Healthcare as such a major initiative, it was incomprehensible to me that the highest-value data–the data that is directly used to clear therapies, recommend them to the medical community, and assess their efficacy–were being managed in the following way:

  1. Physician completes study, and then spends up to a year writing it up and submitting it,
  2. Journal sits on the study for months, then publishes (in some cases), but without ensuring that it matches similar studies in the data it reports.
  3. Oh, by the way, the journal does not make the data available in a structured format!
  4. Then, if you want to see how that one study compares to related studies, you have to either find a recent, comprehensive, on-point meta-analysis (which is a very low chance in my experience), or comb the literature and extract the data by hand.
  5. That’s it.

This strikes me as mismanagement of data that are relevant to lifechanging healthcare decisions. Effectively, no one in the medical field has anything like what the financial industry has had for decades–the Bloomberg terminal, which presents comprehensive information on an updatable basis by pulling data from centralized repositories. If we can do it for stocks, we can do it for medical studies, and in fact that is what I am trying to do. I recently wrote an article on the topic for the Minneapolis-St Paul Business Journal, calling for the medical community to support a centralized, constantly-updated, data-centric platform to enable not only physicians but also insurers, policymakers, and even patients examine the actual scientific consensus, and the data that support it, in a single interface.

Read the full article at!

Some interesting links on post secondary schooling

A Conservative Defense of Tenure

This article raises the important point that tenure is a form of compensation, and one that can reduce budget pressure. It also raises the point that tenure allows a more open-ended approach to schooling which, in my mind, frees teachers and students to engage in genuinely educational but non-measurable activities. At the end the author writes, “we conservatives are especially alive to what is lost when we transform all of our institutions according to the logic of the market.” I agree that conservatives (properly understood) are not pro-market, but as a pro-market libertarian, I also agree with him on the value of tenure. Really what it boils down to is that education (the result we hope students will attain in schools) really is unmeasurable and so can’t be neatly provided in a market or a bureaucracy; schools can be provided on a market, but there is an important civil-society element to them.

From Tennessee, a Solution for Mission Creep

One of the core insights of economics, simple though it appears, is apparently not understood by schools (or even economics departments): everyone doing the same thing is unproductive. Diversity (no, not diversity of melanin content) is the basis of gains from trade, and product differentiation is the way to advance oneself. But what schools tend to do is try to imitate “better” schools by doing a worse version of the same thing. Imagine if restaurants did this; McDonald’s would sell budget foie gras, Applebee’s would sell slightly better foie gras in a kitschy atmosphere, and the only places you’d actually want to eat foie gras would (still) be the same restaurants that sell it in the world we actually live in.

The state of Tennessee has set up an incentive structure that ties funding to measurable outcomes, but makes that funding contingent on a school’s Carnegie rating. The effect is that trying to move up the prestige ladder will result in reduced funding unless a school is actually able to deliver results. “Take one of the state’s regional colleges, Austin Peay State University. If it tried to become more like Middle Tennessee State University by awarding doctorates, Austin Peay would very likely lose 4 percent of its state funds.”

Competency-Based Degrees: Coming Soon to a Campus Near You

If more institutions gravitate toward competency-based models, more and more students will earn degrees from institutions at which they take few courses and perhaps interact minimally with professors. Then what will a college degree mean?

It may no longer mean that a student has taken predetermined required and elective courses taught by approved faculty members. Rather, it would mean that a student has demonstrated a defined set of proficiencies and mastery of knowledge and content.

Sounds good to me! Although, as the author points out, we’re still left with the problem of how to evaluate students. It makes sense to allow someone to test out of an accounting class, but certification of competency isn’t the whole story for a liberal arts program.

If you want to learn skills, then a technical college with a competency-based degree makes a lot of sense. If you’re looking for an immersive environment  that expands your appreciation for philosophy, art, and deep thought then you’re dealing with something unmeasurable. A BA from Wesleyan should communicate that you’ve experienced something like that, but that’s a different product than what most students are looking for (a piece of paper to help them get a good job). This goes back to the conflation of education and schooling. I’m not sure that credentials for liberal arts even makes sense; a better measure of a student’s success in lib arts would be the books and essays they write.

Betting on Vetting

The author is concerned with the current state of affairs in social sciences where hiring and tenure decisions are based on a cumbersome publication process resulting in new research being kept unavailable until it has finally survived the publication process. But there’s an unexploited opportunity: have outside experts evaluate unpublished manuscripts and assign grades. These grades can be used for faculty evaluation, but they can also reduce transaction costs on the publishing end. Instead of a round robin, manuscripts (or articles) are evaluated once, and publishers compete for publishing rights. “The new slogan for upward academic mobility would be ‘produce or perish.’… Publishing was yesterday’s problem, vetting is tomorrow’s.”