The slowdown in idea diffusion

Our regular roundup from earlier this week is posted here.

It’s been six years now since the publication of Ideas are getting harder to find, a famous paper in the economics of innovation which documented that vast increases in research effort are not resulting in materially faster rates of discovery (since then, a number of other papers point to similar conclusions). Why not? My preferred version is that as sciences mature, a variety of stresses emerge. A partial list of those stressors is that, as a discipline matures…

• Ideas that can be quickly or cheaply discovered are the first to be found

• Further progress requires marshalling more detailed knowledge, which requires more years of training to acquire, and older scientists have a harder time with some kinds of breakthrough innovation

• Marshalling more knowledge also increases reliance on research teams, and teams have a harder time developing breakthroughs than individuals

• The requirement to bring more knowledge to bear on outstanding problems, leads people to specialize more. But this impedes making connections across different fields, which is an important part of innovation (see Jordan’s latest blog post on this, in the context of AI that differentially accelerates different fields!)

The last point - that specialization impedes the diffusion of ideas - has seemed likely true to me, but I haven’t ever seen it measured, until this week. A new paper by Gaetani and Berkes tackles this measurement problem in medicine. To measure the spread of ideas, they compare the abstracts of papers that mention a medical phrase in the first year in which it was coined, and then in the subsequent ten years. They then use natural language processing to measure how different the abstracts are that first mention a concept and the ones that use it in later years, with the idea being that a greater difference between the abstracts implies the idea is being used in fields that are further afield from where an idea was developed.

Gaetani and Berkes’ paper suggest that specialization has made it harder for ideas to spread: they find new medical phrases increasingly come from scientists who have worked in fewer fields, and that phrases originating from more specialized scientists don’t diffuse as far. But they also document a dynamic that compounds this issue: over time, scientists have been introducing new ideas in less and less accessible language.

Our living literature review program was motivated in part by precisely these concerns - accessible literature syntheses could help ideas diffuse more quickly over time.