Altmetric Blog

Papers, policy documents and patterns of attention: is it possible to identify which papers make it into policy by tracking their altmetric attention?

Stacy Konkiel, 5th July 2016

The following post is authored by Dr. Lauren Cadwallader, Open Access Research Advisor in the Office of Scholarly Communication at Cambridge University Library and recipient of the first annual Altmetric Research Grant. Following, she explains what she plans to study in the coming months, thanks to the funding she’s received.

Being from a UK institution, the words “REF” and “impact” are often heard at work so these seemed like a natural starting point for thinking about a possible research project for the Altmetric grant. is a great tool for finding out who has been saying what about your paper and it has a lot of potential in helping in research evaluation. When I was reading around the subject I came across Stacy Konkiel’s blog posts that looked at the altmetrics for papers that had been put forward in the 2014 REF exercise. One line in particular got me thinking:

“the value of research changes over time, with what we consider “REF-worthy” impact possibly only being measurable in the longer term”.

Institutions rarely have the opportunity to provide evidence of impact in the longer term, instead they have to either pick papers that have had shorter term impact or select papers as a best guess of their potential for impact because the assessment exercises happen on medium term timescales (the REF is roughly every 5 years). Policy documents are one very tangible piece of evidence of the impact research has had on society but research has shown (1,2,3) that it can take a decade or so for research to be included in a policy document.

I started thinking: what if there is a way to identify the papers that we think will go on to have longer term impact based on shorter term Altmetric attention data? This what my project aims to investigate, using policy documents as a proxy measure for long term impact.

It will be interesting to see if papers that make into policy receive a certain type (or types) of online attention and what timeframes are involved. If correlations can be found, then this might provide a good indicator of which research papers are more likely to get included in policy documents. If no correlations are found then it will also be interesting as we try to understand how and why one paper is more likely to make it into policy than another. Ultimately, the findings might have bearings on the strategies that universities and research groups can use to maximise the policy impact of their research.

What I plan to do:

Scale: This is going to be a pilot project, testing the proof of concept. The grant from Altmetric will cover the time I need away from my main job so I can dedicate myself to this research. Although a small study, I wanted to propose a realistic project that I could complete (or at least have completed the analysis for!) over the summer so I can present the results at the Altmetric 3:AM conference.

Methods – collection, data disambiguation, analysis: This project will use papers produced by researchers at the University of Cambridge. I’ll collect Altmetric data from papers that have been cited in education and environmental science policy documents and a comparative sample that have not made it into any policy. I aim to assess and identify patterns in the type of attention they received, such as attention from particular audiences and the time frame of attention. I will look at the dataset as a whole as well as by policy subjects using stats to tease out any correlations or significant relationships.

Disseminating results: The results of my project will be disseminated in a couple of ways – blog posts, an academic paper and a presentation at the Altmetric 3:AM conference. In keeping with my job as an Open Access Research Advisor, the data will be made publicly available in Cambridge’s OA repository and the paper will be published in an open access journal.

Grand plans for the future: Hopefully, the results of this research will prove to be interesting and will lead onto further research into the Altmetric attention received by papers that make it into policy documents. Interesting correlations might be found that weren’t expected that could lead onto new lines of research. Once I’m finished I’ll be sure to share my thoughts about future possibilities with you!

Leave a Reply

Your email address will not be published. Required fields are marked *