Research Metrics: Responsible Metrics

Perhaps the best known initiative in the Responsible Metrics Movement is DORA.

The San Francisco Declaration on Research Assessment (DORA) originated from the 2012 meeting of the American Society for Cell Biology. Participants discussed their concerns with the way that research is being evaluated by institutions, funding agencies, and other entities. In particular, they were alarmed by the heavy reliance on citation based metrics such as the the Journal Impact Factor. 

Adapted from the Declaration:

"The Journal Impact Factor has a number of well-documented deficiencies as a tool for research assessment. These limitations include:

• citation distributions within journals are highly skewed
• the properties of the Journal Impact Factor are field-specific: it is a composite of multiple, highly diverse article types, including primary research papers and reviews
• Journal Impact Factors can be manipulated (or “gamed”) by editorial policy
• data used to calculate the Journal Impact Factors are neither transparent nor openly available to the public"

DORA makes a number of recommendations for improving the way in which the quality of research output is evaluated:

• the need to eliminate the use of journal-based metrics, such as Journal Impact Factors, in funding, appointment, and promotion considerations;
• the need to assess research on its own merits rather than on the basis of the journal in which the research is published; and
• the need to capitalize on the opportunities provided by online publication (such as relaxing unnecessary limits on the number of words, figures, and references in articles, and exploring new indicators of significance and impact).

There are now more than 20,000 signatories to DORA including research funders, publishers, institutions, and individuals.

"Research evaluation has become routine and often relies on metrics. But it is increasingly driven by data and not by expert judgement. As a result, the procedures that were designed to increase the quality of research are now threatening to damage the scientific system. To support researchers and managers, five experts led by Diana Hicks, professor in the School of Public Policy at Georgia Institute of Technology, and Paul Wouters, director of CWTS at Leiden University, have proposed 10 principles for the measurement of research performance: the Leiden Manifesto for Research Metrics published as a comment in Nature."

The Coalition for Advancing Research Assessment (CoARA) is a member organization seeking to reform research assessment processes. They have developed the Agreement on Reforming Research Assessment with ten Commitments to help establish a common direction for reform, while respecting organizations’ autonomy. Member organizations are encouraged to share Action Plans on how they will implement the commitments.