Altmetric Blog

Attention! A study of open access vs non-open access articles

Euan Adie, 23rd October 2014

You can get the dataset from and the PDF of this post here:
Adie, Euan (2014): Attention! A study of open access vs non-open access articles. figshare.
http://dx.doi.org/10.6084/m9.figshare.1213690

There are lots of good reasons to publish in open access journals. Two of the most commonly given ones are the beliefs that OA articles are read more widely and that they generate higher citations (for more on this check out slide 5 of Macmillan’s Author Insights Survey, which is up on figshare).

Do open access articles get higher altmetric counts?

In celebration of Open Access week we decided we’d take a look at some hybrid journals to see if there was any discernible difference in the quantitative altmetrics between their open access and reader pays articles. We picked Nature Communications to look at first as it’s a relatively high volume, multi-disciplinary-within-STM hybrid journal (at least it was during our study period – it has gone fully OA now), selects articles for publication blind to OA / non-OA status and clearly marks up authors, license and subject areas in its metadata. Plus we sit in the same building.

Coincidentally Nature Publishing Group recently commissioned a study from RIN that indicates that the OA articles in Communications get downloaded more often than their reader pays counterparts. So does that hold true when looking at other altmetrics sources?

Prepping the data & first impressions

Using a combination of the Altmetric API and web scraping we pulled together data on all the Communications papers published between 1st October 2013 and 21st October 2014. You can find all of it on figshare.

The short answer is that yes, there does seem to be a significant difference in the attention received. We’re going to cover some of the highlights below, but feel free to take the dataset and delve deeper – there’s only so much we can cover in a blog post.

First let’s characterize the dataset. It contains 2,012 articles of which 1,395 (70%) are reader pays. The bulk of articles – 1,181 (59%) – are tagged ‘Biological sciences’ by the journal. 519 (26%) are ‘Physical sciences’, 193 (10%) ‘Chemical sciences’ and 104 (5%) ‘Earth sciences’. Only 4 of the 2,012 are reviews.

We grouped articles by month of publication so that we can control for the fact that some kinds of altmetric data accrues over time. You can see this clearly in the graph below – the median number of Mendeley readers for articles published in each month is the line in red.

Screen Shot 2014-10-23 at 17.28.28

A tangent: every source is different

“Older articles have more” doesn’t hold true for all sources. I’ve plotted the median number of unique Twitter accounts talking about each paper by month of publication above too, in blue. Notice that the median actually trends down very slightly as we look at older papers.

This is because: (1) most tweeting happens very quickly after publication and (2) the Twitter userbase is growing incredibly rapidly so there are more people tweeting papers each month.

Think about it this way: if you compared a paper published in 2009 to a paper published in 2014, the 2009 paper would have lots of citations (accrued over time) and hardly any tweets (as not many researchers were tweeting when it was first published – Twitter was still very new). The 2014 paper would have hardly any citations but lots of tweets (as there is now a large number of tweeting researchers).

This is sometimes addressed in novel ways in altmetrics research: Mike Thelwall’s paper in PLoS One presents one elegant solution to a similar issue.

An initial hypothesis

Let’s get back to OA vs reader pays. Here in the office our initial hypothesis was that there would be an OA advantage for tweets in general as a larger audience would be more inclined to read and tweet the paper, but that the effect would be much less pronounced in Mendeley readership and amongst people who regularly tweet scientific papers.

Here’s the median number of tweeters over time, comparing the two cohorts in each month of publication:

Screen Shot 2014-10-23 at 17.37.57

And the median number of Mendeley readers (remember that newer articles won’t have many Mendeley readers yet):

Screen Shot 2014-10-23 at 17.38.07

To get a feel for the data we graphed means and 3rd quartiles too. Here’s the mean number of tweeters who regularly tweet scientific papers:

Screen Shot 2014-10-23 at 17.46.30

There’s a lot of light blue in these graphs and just eyeballing the data does seem to indicate an advantage for OA papers. But is it significant? Once we establish that we can start considering confounding factors.

If we look at all of the articles published in Q4 ’13 (to give ourselves a decent sized sample) we can compare the two cohorts in detail and do some sanity checking with an independent t-test. We’ll look at average author and references counts too in case they’re wildly different, which might indicate an avenue for future investigation.

Here are the results:

Screen Shot 2014-10-23 at 18.13.32

It seems like there is a difference between the number of tweets, the number of tweets by ‘frequent article tweeters’ and Mendeley readers.

The idea that the effect may be less pronounced for Mendeley doesn’t really hold water – a median of 23 readers for OA articles vs 13 for the reader pays is a pretty big difference.

Interestingly we didn’t see much difference in the number of news outlets or blogs covering papers in the two cohorts. A lot of news coverage is driven by press releases, and on the Nature side there is no preference for OA over reader pays when picking papers to press release (we checked).

Confounders

If we accept that the articles published as open access did get more Twitter and Mendeley attention the next obvious question is why?

Two things to check spring immediately to mind:

  1. Do authors select open access for their ‘best’ papers, or papers they think will be of broader appeal?
  2. People tweet about life sciences papers more than they do physical sciences ones. Perhaps the OA cohort has a higher number of biomedical papers in it? Notice that the OA cohort also has more authors, on average, than reader pays cohort. Might that be an indicator of something?

Do authors select only their ‘best’ papers for open access?

It doesn’t seem like we can discount this possibility. Macmillan’s author insight survey (warning: PDF link) have 48% of scientists saying “I believe that research should be OA” as a reason to publish open access, which leaves 52% who presumably have some other reason for wanting to do so. 32% have “I am not willing to pay an APC” as a reason not to go OA. The APC for Nature Communications is $5,200.

Are the higher altmetrics counts a reflection of subject area biases?

Screen Shot 2014-10-23 at 21.21.01

There doesn’t seem to be that much difference when we look at top level subjects, though it might be worth pulling out the Earth Sciences articles for a closer look.

That said, some disciplines definitely see more activity than others: if we look only at articles with the keyword ‘Genetics’ across our entire dataset, taking the median of unique tweeters per article each month then the ‘median of medians’ for OA is 21 and 6 for reader pays.

Compare that to ‘Chemical Sciences’ where the OA median of medians is only 3, and for reader pays it’s 2.

Wrapping up

Open access articles, at least those in Nature Communications, do seem to generate significantly more tweets – including tweets from people who tweet research semi-regularly – and attract more Mendeley readers than articles that are reader pays.

It seems likely that the reasons behind this aren’t as simple as just a broader audience. We’ve also only been looking at STM content.

Would we find the same thing in other journals? We deliberately looked within a single journal to account for things like differences in how sharing buttons are presented and to control for different acceptance criteria, and the downside to this is we can’t generalise, only contribute some extra datapoints to the discussion.

We’ll leave further analysis on those fronts as an exercise to the reader. Again, all the data is up on figshare. Let us know what you find out and we’ll follow up with another blog post!

52 Responses to “Attention! A study of open access vs non-open access articles”

Graham Steel (@McDawg)
October 24, 2014 at 12:00 am

Attention! A study of open access vs non-open access articles - blog post by @Stew http://t.co/yMXJjiduwg

Stacy
October 24, 2014 at 12:00 am

I was JUST asking this question earlier this week. Thanks for pulling this study together to backup my hunch with a bit of data! :)

Impactstory (@Impactstory)
October 24, 2014 at 12:00 am

.@stew backs up our hunch that #OpenAccess articles get more online attention & #altmetrics http://t.co/kQkvkvEVku #OAWeek2014

Mogens H. Sørensen (@mhsorens)
October 24, 2014 at 12:00 am

Do #openaccess articles get high #altmetric counts? Study by @altmetric of @naturecomms articles from the last year: http://t.co/XlzmCZtszT

@AnnalisaManca
October 24, 2014 at 12:00 am

Attention! A study of open access vs non-open access articles - http://t.co/DkFAaCbt9S

@NazlinBhimani
October 24, 2014 at 12:00 am

A study of open access vs non-open access articles via #almetrics.com http://t.co/c1q3ap5ll4

Fabrice Leclerc (@leclercfl)
October 24, 2014 at 12:00 am

Top #openedu story: Attention! A study of open access vs non-open access articl… http://t.co/cK5wRVUD0p, see more http://t.co/BLfaziYYRV

@tarabrabazon
October 24, 2014 at 12:00 am

Do open access journals receive more readers and tweeters? For STM - the answer is YES. (http://t.co/8Qaco8C43b). #oa

@jimtill
October 24, 2014 at 12:00 am

Do #openaccess articles get higher #altmetric counts? A study of @naturecomms articles from the last year http://t.co/bh2eeYKCcW @altmetric

@wolframseidler
October 25, 2014 at 12:00 am

Attention! A study of open access vs non-open access articles http://t.co/iUiVE7pYDZ

Richard Poynder (@RickyPo)
October 25, 2014 at 12:00 am

Attention! A study of open access vs non-open access articles http://t.co/vB6a6zebVj

Joseph Esposito
October 25, 2014 at 12:00 am

Why is this site not optimized for sharing through social media?

@PhilipGardiner
October 25, 2014 at 12:00 am

Open access articles get more tweets than paid articles: analysis http://t.co/NdZYRmzWPH #altmetric

Xavier Lasauca (@xavierlasauca)
October 26, 2014 at 12:00 am

A study of #openaccess vs non-open access articles, by @altmetric http://t.co/hjPGxcSYn7 #impact #research

Lou Woodley (@LouWoodley)
October 26, 2014 at 12:00 am

Do Open Access articles get higher altmetric counts? http://t.co/ELpWxmlUiy #OA

Simon Harold (@sid_or_simon)
October 26, 2014 at 12:00 am

Interesting! #Openaccess articles get more tweets than non-#OA articles, in Nature Communications at least http://t.co/k7Y1vTs1rZ

@LeonieMueck
October 27, 2014 at 12:00 am

Do #openaccess articles get more attention online? @altmetric analyzes OA and non-OA @NatureComms articles http://t.co/vUQeLGEiPj

@AnoopSimha
October 27, 2014 at 12:00 am

#OA articles get more Tweets - great @altmetric article (and data) @figshare http://t.co/nlHGtEB2yD @NTULibraries Interesting read indeed!

@LIBREAS
October 27, 2014 at 12:00 am

Attention! A study of #OpenAccess vs non-open access articles / http://t.co/9e0q3IjFCK - http://t.co/iUcVsggCsY

@theladythinks
October 27, 2014 at 12:00 am

Attention! A study of open access vs non-open access articles

http://t.co/40mBn3EnaW

Kirk Englehardt (@kirkenglehardt)
October 27, 2014 at 12:00 am

#openaccess vs. non-open access papers: which are read more? (people tweet the newer stuff) http://t.co/txpQiHGqHP @Stew #scicomm

[…] Do open access article get higher altmetric counts? […]

Keita Bando (@KeitaBando)
October 28, 2014 at 12:00 am

Do #openaccess articles get higher @altmetric counts? http://t.co/me0C3upDrZ A study of open access vs non-open access articles #altmetrics

@N_Kwanele
October 28, 2014 at 12:00 am

Attention! A study of open access vs non-open access articles - http://t.co/b1vfrMWBF8

Frank Nuijens (@FrankNu)
October 28, 2014 at 12:00 am

A study of open access vs non-open access articles | http://t.co/hprqsad8B6 http://t.co/xTCeLWpf6m

Science journalism (@sci_journalism)
October 28, 2014 at 12:00 am

A study of open access vs non-open access articles | http://t.co/wdZRWDhLGR http://t.co/8eaQzhdBi4

Science journalism (@sci_journalism)
October 28, 2014 at 12:00 am

A study of open access vs non-open access articles | http://t.co/wdZRWDhLGR http://t.co/8eaQzhdBi4

@OAJoe
October 28, 2014 at 12:00 am

[tag] Attention! A study of open access vs non-open access articles | http://t.co/bxQIGOBjla http://t.co/wm7Xe2kgzd

donatella gentili (@do_ge)
October 29, 2014 at 12:00 am

Attention! A study of open access vs non-open access articles - http://t.co/vCjWy3vy1c

@melbaek
October 29, 2014 at 12:00 am

#OpenAccess articles, at least those in Nature Communications, do seem to generate significantly more tweets
http://t.co/2JnefFvxMi

@RitaNeko1
October 30, 2014 at 12:00 am

A study of open access vs non-open access articles http://t.co/D5ynhyAgHI #altmetrics

@juliaKhpi
October 30, 2014 at 12:00 am

A study of open access vs non-open access articles
#qr8-14 #Altmetric
http://t.co/aDymtI8n5O

@MDPIOpenAccess
October 31, 2014 at 12:00 am

Looking at #OpenAccess versus #NonOpenAccess http://t.co/PlUAcskQCy

@FabTinti
November 1, 2014 at 12:00 am

A study of #openaccess vs non-open access articles http://t.co/6eg2grNsiA @altmetric

Freddie
November 1, 2014 at 12:00 am

Hello,
I think the results would be the same for any kind of stuff on the internet. As far as I am concerned, I only tweet about matter that is accessible to my audience, whether it is scientific articles or not. I would never tweet about something I know I benefit privileged access or about something that requires authentification to access the full content. For example, I don't tweet about a newspaper article that is not free to anyone, neither a post on a private Facebook account.
I feel my audience/network has the same behaviour because I think I never came across a tweet mentioning something I can't access to. I would remember the frustration it would have aroused.

Thank you for this very interesting study that conforts me in my belief and will help me to convince research fellows I work for.

Fb

@freddie2310
November 1, 2014 at 12:00 am

Les articles OA semblent déclencher plus de tweets que les non-OA #altmetrics #OA http://t.co/53Z5xFZqL8

@KUnlatched
November 4, 2014 at 12:00 am

Attention! A study of open access vs non-open access articles | http://t.co/B9EfHTpSQ1 http://t.co/GmOXHikjED

@Sebdupre
November 7, 2014 at 12:00 am

Attention! A study of open access vs non-open access articles http://t.co/dIYrzvZEbR via @freddie2310

[…] Attention! A study of open access vs non-open access articles  […]

Altmetric Top 100 2014 | Altmetric.com
December 9, 2014 at 12:00 am

[…] to get shared and discussed than those that are published behind a paywall – indeed we recently undertook a small scale study of our own to look at this. At the time the results we got proved positive, however we do not see the same […]

@mrkwr
January 2, 2015 at 12:00 am

@CameronNeylon Thx I agree, just wondering what was out there. I’ve seen http://t.co/wouQt27UDQ @Stew @Impactstory @GrowKudos

Altmetric (@altmetric)
February 24, 2015 at 12:00 am

@TrevorABranch we've found that people often share open access content more, so that might be another factor? http://t.co/pmcFbsyMeT

[…] interested in the correlation between open access papers and altmetrics attention. In Euan’s recent blog post, we found that open access papers from a set of Nature Communications articles generated […]

Open Access | Living and Working on the Web
April 28, 2015 at 12:00 am

[…] there are advantages for you personally: the research shows that by obeying Open Access policy, scientific articles gather more views and citations. Which makes perfect sense – certain group of people that normally would not buy the […]

Altmetric (@altmetric)
May 7, 2015 at 12:00 am

@mrnick be interested to hear what you find! ICYMI, we did a v small scale study: http://t.co/SdzA8tPmLz

@OABrunel
September 17, 2015 at 12:00 am

Attention! A study of open access vs non-open access articles http://t.co/fIE60vW5wb via @altmetric

@DeGruyterOpen
October 8, 2015 at 12:00 am

#OpenAccess articles are tweeted more. The article backed with #opendata http://t.co/fsLN5XUat9 #altmetrics http://t.co/R7maZjqdj7

@Aisaonlus
October 8, 2015 at 12:00 am

An altmetric study of open access vs non-open access articles - http://t.co/jqt6ISSjc3 v. @RitaNeko1

[…] and the use of social media for research dissemination is also closely allied to open access (see this study from last year’s Open Access Week). Other links can also be made, such as the role of open access with regard to Wikipedia in terms […]

[…] available for all to read, and it has the added advantage of getting you more citations, views, Mendeley readers and Twitter mentions. What’s not to love about […]

Free Resources – WEB IS WHERE THE HEART IS
November 15, 2016 at 12:00 am

[…] (Source: Adie, 2014) […]

Leave a Reply

Your email address will not be published. Required fields are marked *