Altmetric Blog

Altmetrics linked to 3* and 4* impact scores in REF2014 impact case studies

Stacy Konkiel, 20th January 2017

Though altmetrics’ ability to directly predict “real world” impact is still very much under debate, one research team has found that the presence of altmetrics is related to the broader impacts of their REF2014-submitted research.

We spoke to Dr. Jenny Wooldridge, a member of the National Physical Laboratory (NPL) Impact Evaluation team, to learn more about her team’s research that uses expert peer review, citations, and altmetrics to understand the attention and related impacts that UK research has received.

Exploring the data

Initially, Jenny’s team used the Altmetric Explorer for Institutions platform to run reports on altmetrics for NPL publications that were identified as the “high impact” via the organization’s annual awards program.

The Impact Evaluation team found initial evidence that the “high impact” NPL-authored publications may be correlated with the presence of Altmetric attention scores, but the sample was too small to yield statistically significant results. So, the team was keen to expand the study to see how well altmetrics aligned with peer review evaluations of research impact, using REF2014 peer review data from universities across the UK.

How do altmetrics relate to 3* and 4* research?

To understand if the presence of altmetrics was correlated with expert peer review, the NPL Impact Evaluation team analyzed a total of 41,409 DOIs of publications submitted to REF2014 main panel B (relating to mathematics, engineering and the physical sciences) using Altmetric Explorer for Institutions.

Each UK university department’s research outputs and impact case studies were scored by REF2014 assessment units according to the REF2014 “star” system for quality and impact. The Impact Evaluation team associated peer review scores with individual research outputs in two ways: scores related to research quality were associated with the lists of papers provided as evidence for departmental excellence, and scores for impact case studies were related to references extracted from each case study’s cited references lists.

3* and 4* NPL research outputs and impact case studies were then evaluated with respect to the presence (or lack thereof) of Altmetric attention scores, with controls applied for citation rates (the average number of traditional bibliometric citations accrued per year post publication), the age of the publications, and the research subject areas.

Jenny’s team ran the following regressions on the collected Altmetric attention score and reference data:

  1. Quality scores vs paper citations, paper age and paper subject
  2. Quality scores vs non-zero Altmetric attention scores, paper citations, paper age and paper subject
  3. Impact case study scores vs paper citations, paper age and paper subject
  4. Impact case study scores vs non-zero Altmetric attention scores, paper citations, paper age and paper subject

Whether or not an output has an Altmetric attention score (i.e., whether research was being discussed online) did not seem to affect the REF2014 quality scores. However, the presence of Altmetric attention scores were found to be highly correlated with REF2014 impact case study scores. Put another way, the presence of altmetrics can potentially provide an early indicator of societal impact as compared with bibliometrics, with the vast majority of mentions occurring within the same year the paper was published.


Figure 1. Citation and Altmetric mention latency for the publications submitted as evidence to the REF 2014 exercise within main panel B. Altmetric mentions data have been restricted to period after the launch of The anomalous data in 2004-2005 relate to two publications on graphene that attracted large numbers of citations after the announcement of the 2010 Nobel prize.

NPL’s research on the REF2014 data provides direct evidence for the how online discussions of research are related to the potential impact of research, and Jenny plans to publish the results of this work. Jenny is very interested to see where UK research has been picked up by news outlets and within public policy documents, and so her team will be further analysing the REF2014 publications using Altmetric data to focus on these aspects.

To learn more about this study, email Jenny at

Leave a Reply

Your email address will not be published. Required fields are marked *