Médecins Sans Frontières Scientific Day
As a charitable organisation that contributes to research, how do you measure the impact of your publications? Being able to track societal and academic impact has been particularly important for Médecins Sans Frontières/Doctors Without Borders (MSF), a humanitarian aid organisation that provides medical care during international emergencies, conflicts, and disasters. Each year, MSF staff publish findings in peer-reviewed journals, and the scholarly papers are then deposited in the open-access MSF Field Research repository. Since MSF research has the potential to influence domains such as policy, clinical practice, and humanitarian advocacy, it’s necessary for the organisation to continually assess its own impact.
MSF research, which spans numerous health-related topics (including HIV/AIDS and tuberculosis), takes centre stage each year during a conference called MSF Scientific Day. Altmetric was invited to contribute a poster to the 2013 MSF Scientific Day, which took place last Friday at the Royal Society of Medicine in London. I attended the conference and presented an altmetrics research poster entitled “Using altmetrics and citation counts to assess the social and academic impact of Médecins Sans Frontières publications”. (The poster was authored by myself and Euan Adie, along with 2 members of the MSF Manson Unit, Louise Bishop and Sarah Venis.)
Assessing the social and academic impact of MSF publications
The purpose of our poster was to describe the online and academic impacts of recent MSF publications (from July 2011 to December 2012) using altmetrics and citation counts. We retrieved altmetrics data (including the Altmetric score) and Web of Science citation counts for 155 articles in the MSF Field Research, and divided the articles into 2 categories: research (primary research articles) and other (editorials, perspectives, systematic reviews, etc.). Check out the detailed methodology and limitations of the study here.
These were our key observations:
- Online attention tends to be low across all article types for MSF publications; nearly all of the attention comes from Twitter.
- Non-research (“Other”) articles tend to be tweeted by a higher average number of people than research articles.
- The number of articles mentioned online at least once increased in 2012 but was still only just over 50%.
- Citations are slower to accrue, reaching 60% for 2011 research articles but only around 20% for 2012 equivalents.
Overall, MSF publications didn’t tend to receive very high Altmetric scores (the paper that received the most attention had a score of 44; see case #1 in the case studies section below). Twitter was by far the largest source of online attention, as any papers that had non-zero Altmetric scores were all mentioned at least once on Twitter. Surprisingly, there were very few mentions of MSF publications in other sources such as Facebook and blogs, and there were no mentions in mainstream news outlets.
We didn’t see much attention paid to MSF papers on Facebook, but this may have had something to do with the way that we usually quantify attention from this source. During MSF Scientific Day, I had an interesting chat with Ginny Barbour, the Medicine Editorial Director at PLOS, who told me that PLOS Article-Level Metrics had picked up quite a bit of Facebook attention surrounding MSF papers that were published in PLOS. However, PLOS’ method of tracking attention looks at Facebook “likes”, whereas Altmetric only counts unique posts with links to the article. The differences in methodology and approach to tracking likely account for the discrepancies in our data for the attention surrounding MSF papers on Facebook.
Based on our little study, we concluded that MSF could use the altmetrics data to craft new strategies for promoting their research online. We also felt that having mentions in a greater diversity of communication channels could be beneficial. We suggested that one simple method for increasing public awareness of the scholarly articles could be through existing blogging activities. Since MSF blogs by patients and staff (e.g., see the “TB & Me” blog) are already very influential, the inclusion of links to relevant publications could increase the reach of the scholarly works. Finally, we pointed out that if usage of online communication channels to talk about MSF publications increases, then altmetrics could certainly act as rapid indicators of research uptake within society, and could be used to measure impact more effectively in the short-term.
Altmetrics case studies – 3 MSF non-research publications
[altmetric doi=”10.1126/science.1225702″ float=”right”]
Getting HIV Treatment to the Most People
Published on 12 July 2012 in Science
This “Policy Forum” article on HIV treatment coverage was by far the most popular MSF publication in 2011-2012, and was mentioned on Twitter and in a BMJ blog post by an MSF USA staff member (see Altmetric details). After the article was published online on the 12th of July 2012, it garnered very few mentions on Twitter. However, 12 days later, an influential tweet sent by the Science Twitter account sharply boosted the article’s profile:
Ending the AIDS epidemic is now a realistic goal through the use of antiretroviral therapy scim.ag/NNdBFz
— Science Magazine (@sciencemagazine) July 24, 2012
The combination of having a concise article title and being promoted by a uncomplicated, powerful tweet likely contributed to the online success of this particular MSF article, which was largely tweeted by members of the public (62%). Despite the article’s popularity on Twitter, its academic uptake was slower. When I checked out the citation counts back in March, this article had only been cited once in the scholarly literature, which was about on par with the average number of citations for MSF non-research articles in 2012 (0.88).
[altmetric doi=”10.1186/1744-8603-7-39″ float=”right”]
A win-win solution?: A critical analysis of tiered pricing to improve access to medicines in developing countries
Published on 12 October 2011 in Globalization and Health
This article, a critical examination of the concept of “tiered pricing” of medicines, received attention mainly from Twitter, although it was also mentioned in a blog post on Scienceline (see Altmetric details). A closer examination of the people who tweeted about the article revealed that the article appeared to reach its intended target audience. Influential organisations that tweeted about the article included the Southern Africa Regional Programme on Access to Medicines (SARPAM), the Pan American Health Organization/World Health Organization (PAHO/WHO) Equity, SciDev.Net, as well as the Globalization and Health journal (the publisher). Several professionals also shared the article, including Amanda Glassman, Director of Global Health Policy and Senior Fellow of the Centre for Global Development.
This article performed well in terms of altmetrics, but what about its academic uptake? With 6 citations in the literature since publication, academic uptake of this article was actually much higher than the average for MSF non-research articles in 2011 (2.41).
[altmetric doi=”10.5588/ijtld.11.0635″ float=”right”]
Language in tuberculosis services: can we change to patient-centred terminology and stop the paradigm of blaming the patients? (see Altmetric details)
Published on 1 June 2012 in The International Journal of Tuberculosis and Lung Disease
This article took a strong stance against the detrimental and judgemental language of tuberculosis, empathising with patients by taking aim at the use of the words “defaulter”, “suspect”, and “control”. Perhaps it was this emphasis on language, but there was a particularly high number of “unique” tweets (i.e., tweets that didn’t just involve sharing the article title).
— Tim France (@francetim) May 16, 2012
— KNCV Tuberculosis F. (@kncvtbc) May 15, 2012
As with the Globalization and Health paper, this one also seemed to reach the audiences it was intended for. Organisations such as KNCV Tuberculosis Foundation (along with its Africa regional office) and TB Alliance tweeted about the article, as did health care practitioners, patient advocates, and so on. In addition to its rapid social uptake, this article has also been cited 4 times in the scholarly literature, greatly exceeding the average number of citations for MSF non-research articles in 2012 (0.88).
Thank you to Sarah Venis and Louise Bishop for inviting us to contribute a poster to MSF Scientific Day, and for guiding us through the submission process with helpful comments and advice.