This post was contributed by Dr. Lauren Cadwallader, winner of Altmetric’s first annual Research Grant and Open Access Research Advisor in the Office of Scholarly Communication at Cambridge University Library.
Earlier this year I was awarded the first annual Altmetric.com Research Grant to carry out a proof-of-concept study into the patterns of online attention received by journal articles that are incorporated into policy documents. I was planning to look at the types and timings of attention that papers received before they were incorporated into a policy document, to see if there was some way to help research administrators make an educated guess rather than a best guess at which papers will have high impact for the next REF exercise in the UK.
The results have rather taken me by surprise – I was half expecting that there would be no patterning and that altmetrics would be no help whatsoever! This work hasn’t come up with a formula to predict which papers will be incorporated into policy, but it has come up with some interesting insights, which I think merit further research.
The Data Used
I selected papers authored by researchers from the University of Cambridge that originated from the Department of Geography, the Department of Public Health and Primary Care and the Faculty of Education in order to cover a broad range of policy subjects. The sample I used contained both papers that had been incorporated into a policy document (based on Altmetric.com’s policy sources) and a comparative set of papers that hadn’t been included in any policy (the data is shared here).
I looked at the type and timings of the attention that articles received, in relation to both the publication date and the policy inclusion date (if applicable. I wanted to see if there was any patterning, especially in the period of time before papers were included in policies.
Papers are incorporated into policy quicker than thought
In my research proposal I cited a couple of studies that had looked at the time lag between a paper being published and its findings being incorporated into a policy or practice. Those studies quote between 8 and 20 years for biomedical research to be adopted. My work used papers published since 2011 and I was expecting that the papers would have taken four or five years to be included in a policy.
However, I found that nearly all of the papers were included in a policy document within the first two years after publication. This was true regardless of the discipline of the research. This raises really interesting questions to do with modern day scholarship and research dissemination: Are researchers writing specifically with policy makers in mind? Have policy makers changed the way they find material? Is open access having any effect?
Duration of attention lasts for longer for (some of) the policy papers
One of the things that interested me was how long any online attention lasted for papers. There are two parts to this finding. Firstly, some papers get very little online attention and are still cited by a policy document. Roughly equal numbers from both subsets received attention that lasted less than a week (typically just one tweet). This shows that any formula for predicting high impact papers (in terms of policy inclusion) that is based solely upon sustained online attention would fail to identify all impactful papers.
Secondly, for papers that did receive more than a week’s worth of attention, the duration of that attention was on average longer for papers that made it into a policy compared to those that didn’t (61 weeks versus 33 weeks respectively). The sustained online attention doesn’t appear to be related to the length of time since publication, nor is the extension of the attention due to the policy incorporation. There seems to be something about these papers unrelated to policy that warrants sustained attention rather than a concentrated amount of attention close to publication.
News and blogs have sustained attention for policy papers
Looking at the different types of attention – tweets, news stories, blogs and Facebook mentions – it appears that sustained news and blog attention could be a key indicator for identifying those papers likely to make it into policy. When the first year’s worth of attention was plotted into 4 week periods, a trend emerged that saw news and blog stories crop out throughout the first year for many of the papers that made it into policy. On the other hand, with the comparative set, most news stories were published with 3 months of the article’s date of publication with the journal. The patterning for tweets was much less clear, but it’s possible that with a larger sample size that trends might emerge.
This research has raised some interesting questions about the relationship between research publication, dissemination, and policy adoption, questions that I did not expect. It also shows that by examining and visualizing the patterning of research attention over time using altmetrics data, some really interesting observations can be made about the types and timings of attention that papers receive.