Downweighing review scores at NIH

Last week the NIH released an RFI proposing a change to how it uses and reports peer review scores. Instead of receiving a numerical impact score and percentile, applicants would learn only which of three bins their proposal landed in: “most competitive” for the top 25%, “competitive” for 26-50%, and “not discussed” for the rest. Overall impact scores and percentiles would also no longer be shared with program staff and advisory councils, who would instead see the relevant bin, the written critiques, and the specific criterion scores.

The stated rationale is that the numerical scores falsely suggest a high level of quantitative precision, and are middlingly predictive of downstream outcomes; thus, more qualitative buckets can give applicants and program staff a higher-level sense of relative quality while mitigating the likelihood of either actor overindexing on score-based rankings in their assessment of proposal quality. Indeed, I have heard from NIH institute directors that program officers’ decisions often hew too closely to percentile scores, rather than incorporating reviewer scores, reviewer comments, and PO discretion into a more holistic portfolio creation process. It is reasonable that removing the opportunity to simply rank and fund to the cutoff could encourage more consideration of written comments and disciplinary priorities (NSF already approaches their review and decision-making in something closer to this manner). But many scientists and science policy practitioners are reasonably wary, given the broader context of the OMB proposed rule, ongoing grant terminations, and politicization of review.

In my view, the most useful piece on this proposal is from DrugMonkey, a longtime pseudonymous blogger on biomedical science. They generally support a balanced approach, and point out that many institutes have historically deviated from the strict percentile order; as a result, this change mostly forces institutes that have historically funded more strictly on paylines to adopt the practices of the more balanced institutes. By DrugMonkey’s lights, the reason to be concerned is not that peer review scores are sacred and deviating from a strict order is itself harmful to science, but that the proposed change could give political appointees more authority to override scientific review while simultaneously obscuring that process. Comments are due October 13.