Altmetric Blog

Altmetric and Policy: Discovering how your research impacted real-world practises

Josh Clark, 15th September 2016

The following post was written by Ben McLeish, Product Sales Manager at Altmetric. 

One of the greatest challenges facing our customers is the greater understanding of how research impacts policy guidelines authored by the world’s leading advisory and Non Governmental Organisations and similar institutions, and, in a second step, the understanding of how such guidelines, once implemented, can be tied to real-world effects much further down the line. Whole organisations are dedicated to answering this question, and so I wanted to give an example of how the difficult connection between your research and practical implications of implementing guidelines can be easily discovered in Altmetric.

Once the research has been published, references in policy documents first need to be discovered. This first step is quite hard on its own, as these references occur inside PDFs which can’t be simply googled with ease. Discovering what research is used within hard-to-search policy PDFs is essentially impossible without using complex and sophisticated data mining techniques to ingest full PDF documents and pick out the citations in a centralised and systemic manner. Citations are often buried at the end or within footnotes of reports which can reach hundreds of pages in length. It is simply too difficult and unreliable to try and track where your research is used within policy papers manually.

The following example of an attempted manual workflow demonstrates how the strategy for discovering policy documents which refer to your research must instead be automated using Altmetric in order to be useful for authors and institutions wishing to demonstrate how their research has played a role beyond the academy.

Example: Using the research output “How Does the Gender of Parents Matter?” by authors Biblarz & Stacy, let’s try and first attempt a manual discovery of where this item has been mentioned in policy.

The first challenge is, we do not know in which policy source to begin looking. A general Google search for the reference of this article, using the DOI for example, does little to help; in fact this approach presents more of a hindrance. Searching the URL of the article for example delivers paltry and irrelevant results, either of the item itself or duplicated results of it:

Searches version 2

Wading through these items to find policy coverage can’t work. Even searching the title of the research itself yields a different yet still mostly useless set of results, mostly to books, or to third-party hosted copies of the article. Nowhere is there to be found the policy document references which do in fact exist.

In contrast to this approach, our details page for coverage of this article shows that the Institute for Women’s Policy Research actually authored a data-driven analysis in February 2015 which evaluated the effect of a practical policy put into practise by the US Government within the African-American community known as the “My Brother’s Keeper” initiative. This multi-faceted government policy focuses on gender roles in a heavy way, meaning that the main driver of the initiative and the main focus of this research article are in the exact same sphere. The use of this work is evidence of direct social intervention practises that draw on this research to enact change in the real world, and understand that change, as we can see from the report.

CitationsNote from the citation, by the way, that neither the URL of the article, or the identifier were actually used. Altmetric managed to find the reference anyway!

Now to see how the research was actually used. The reference to the article by Biblarz & Stacy is used here primarily to show methodological difficulties in prior studies of judging gender and orientation influences child welfare and development. This demonstrates that this item systematically informed the process by which a certain program (in this case, “My Brother’s Keeper”) could ultimately be evaluated in its real-world effects. The citation even directly follows several paragraphs of conclusion which it is only fair to include here in full. I have highlighted the points which focus on gender itself.

“The MBK Initiative represents a substantial dedication of public and private resources to the unique needs experienced by Black men and boys. The data presented here show that comparable commitments are warranted to integrate a focus on the specific challenges and concerns facing young women. The data also suggest that it is important to consider the well-being of young men and women of color in an integrated and comprehensive fashion, because so many of the challenges they face are shared. An initiative that includes both genders is more likely to ensure that the economic, community and structural factors affecting both genders are highlighted and addressed. Currently many communities where people of color are concentrated do not provide high quality education, jobs, adequate housing, food and safety, and these structural and community-wide factors affect girls as much as boys. At the same time, as the data presented here show, boys and girls of color also have distinct needs that warrant gender- informed approaches. Some popular approaches such as single sex education or marriage promotion, however, lack an evidence base regarding their effectiveness and may encourage stereotyping about men’s and women’s roles. They appear to many advocates to be popular “bandaids” aimed at restoring male privilege rather than at eradicating sexism and racism and strengthening economic opportunities for women and men of color. As an alternative, many advocates and scholars advise approaches centered on addressing institutional and economic racism in a context of gender equality. Improving the well-being of young women and men of color requires an integrated approach that considers women’s and girls’ concerns, along with those of men and boys. In addition, it requires attention to approaches that promote gender equity along with those that reduce racism and increase economic and social well-being.”


This can be finally married to the original research using the Altmetric data, in addition to showing all the other areas where the research garnered coverage. In the case of this publication, it has received significant broad attention across many other spectra of attention. The authors, and their department or the institutional research administrators can for example look at the news, blogs, social media which picked up this item prior to, and even after, its use within policy evaluation. Academic impact also exists for this piece (the research was used in Wikipedia to inform two separate pages, and garnered over 250 Mendeley Readers, who are broken down demographically by Altmetric automatically.) Conversations exist on YouTube which refer to this item. This is all vital and important data about the role of this research. All time-stamped, all linked to the original mentions online, this data provides a deep and detailed timeline of the impact of this research. The authors had to do nothing to create this. They may not even be aware of this use of their work!

Should further use of the article occur, Altmetric will automatically collate this new attention into the same output details page. We do this constantly for the over 7 million datasets, articles, code data and other research outputs that we automatically track once they are online and given an identifier. What other stories like this can you discover today? Try out our bookmarklet and send your favourite ones to us!

Leave a Reply

Your email address will not be published.