{"id":432,"date":"2021-10-26T14:47:07","date_gmt":"2021-10-26T14:47:07","guid":{"rendered":"http:\/\/blogs.plymouth.ac.uk\/research\/?p=432"},"modified":"2021-10-26T14:47:07","modified_gmt":"2021-10-26T14:47:07","slug":"an-introduction-to-responsible-metrics-open-access-week","status":"publish","type":"post","link":"https:\/\/blogs.plymouth.ac.uk\/research\/2021\/10\/26\/an-introduction-to-responsible-metrics-open-access-week\/","title":{"rendered":"An introduction to responsible metrics (Open Access Week)"},"content":{"rendered":"

\"theThis post is part of a series of blogs in celebration of Open Access Week 2021<\/a>. Keep an eye out for more posts throughout the week or follow our Twitter account,\u00a0@OpenResPlym<\/a>, to keep up to date with OA Week events.<\/strong><\/p>\n

<\/h2>\n

Responsible metrics in a nutshell<\/strong><\/h2>\n

Research metrics\u00a0are used to ‘measure’ the influence or impact\u00a0of researchers and their publications. Authors use journal metrics to decide where they want to publish; article metrics are used to assess the ‘quality’ of a research output, or group of outputs; institutions use author metrics to inform the recruitment, probation, or promotion of researchers.<\/p>\n

Most research metrics come in the form of quantitative measurements. Many well-known metrics are citation-based, from citation counts and percentiles, Field-Weighted Citation Impact<\/a> (FWCI), the h-index<\/a>, and the Journal Impact Factor (JIF)<\/a>; beyond these, metrics might include anything from views, downloads, mentions, or sales, to collaboration metrics or research grant income.<\/p>\n

Problems with the use of metrics arise when these quantitative<\/em> metrics are used as a proxy for measuring something more complex and qualitative<\/em>, such as the overall calibre of a researcher or a research output. Qualitative metrics can also be biased or manipulated, particularly if too much emphasis is placed on specific metrics as evaluation criteria.<\/p>\n

Responsible metrics is a movement which advocates for the ethical, appropriate use of numerical metrics when evaluating research. The idea is not to do away with quantitative metrics, but rather to ensure that they are used in appropriate situations, applied alongside qualitative information wherever possible, and that they are not used as inadequate proxies or arbitrary measures.<\/p>\n

 <\/p>\n

What\u2019s so bad about quantitative metrics?<\/strong><\/h2>\n

Quantitative metrics are not inherently \u2018bad\u2019. The problems arise when these metrics are used<\/em> badly.<\/p>\n

Citation metrics, for example, measure exactly what they set out to measure (within the parameters of the available data). They indicate how many citations were received by an output or group of outputs within a certain period of time; this measurement may be weighted, expressed as a percentile, or calculated in relation to another metric. These data certainly tell a part of a research output\u2019s story, and they can be useful. They do not, however, tell the whole<\/em> story: a high volume of citations does not guarantee that a piece of research is of high quality, nor vice versa. It is therefore not the metrics that are the problem, but rather the assumptions which are made about what these metrics can sufficiently \u2018measure\u2019.<\/p>\n

Some of the ways in which research metrics can be insufficient, biased, or misconstrued include:<\/p>\n

Lack of context<\/h4>\n

There are many reasons publications get cited, and not all of them are good. Generally, citation metrics make no distinction between \u2018good\u2019 citations and neutral or negative ones. Similar problems can arise when using social media attention as an indicator of research quality.<\/p>\n

False proxies<\/h4>\n

The Journal Impact Factor (JIF) measures the average citations per document in an entire journal over two years. It is unreasonable to use the JIF of the journal an article was published in as a surrogate for its quality as an individual publication \u2013 or even for its citation impact: papers published in high JIF journals are not guaranteed to be more highly cited.<\/p>\n

Bias and gaming<\/h4>\n

There are many ways in which research metrics can be biased. Some speak to broader problems (for example, some studies have shown female authors are less likely to be cited than their male colleagues),[1]<\/a> while other metrics can introduce bias when applied in the wrong situations (for example, the h-index disfavours younger researchers, and doesn\u2019t account for author order). Metrics can be deliberately manipulated, too \u2013 publishers can game their impact factor through publishing only in particular areas, avoiding certain output types, or even through participating in citation coercion and \u2018citation cartels\u2019.[2]<\/a><\/p>\n

Skewed incentives<\/h4>\n

Too much emphasis on one metric encourages goal displacement and distortion of behaviour. Researchers may feel the need to prioritise what they are being measured by (e.g. impact factor) over anything else (e.g. more suitable publishing venues and\/or open access opportunities).<\/p>\n

Suitability<\/h4>\n

Some metrics may be used as appropriate indicators in certain situations but are entirely inappropriate in others. The FWCI, for example, becomes less stable the smaller the sample size, so it is not a suitable metric for smaller groups of outputs.[3]<\/a> It also takes time to stabilise (since citations accrue over time), so it is more inaccurate when applied to newer publications.<\/p>\n

Other common problems with metrics can include database reliance (since different databases will produce varying results) and failure to account for variance in practise between different disciplines.
\n <\/p>\n

So, what\u2019s the solution?<\/strong><\/h2>\n

It is not practical to do away with the use of research metrics entirely, nor does the responsible metrics movement advocate for this. It is possible, however, to avoid certain practises that are more likely to discriminate, and to introduce measures which can help to offset some of the problems outlined above.<\/p>\n

Some general rules for good practice in research assessment include:<\/p>\n