We’ve been busy improving our policy tracking system this year to bring you more mentions of research in policy documents with improved text-mining accuracy and across lots more sources.
Policy documents have always been an important attention source for us, helping you uncover mentions of research in significant policy sources — mentions that otherwise might go undiscovered. This month we’ve launched a brand new system to find more mentions of research in our growing collection of policy and guidance sources and with greater accuracy.
Altmetric have now collected over 1.1 million mentions of research outputs in policy documents. We search policy sources produced by over 50 organisations from around the world. See the aggregated graph from the Explorer below to see how policy mentions of research have increased over the last two decades:
We can see this aggregated attention data over time as we to collect policy documents historically, as long as the PDF files are made available by the policy organization.
Building the new policy tracker
Our original policy tracker was built in 2014 (check out Jean’s blog post announcing the new attention source at the time) and this year we’ve overhauled the system under the hood!
The new policy tracker works by collecting PDFs from policy sources online, and searching the files for links to research outputs, output identifiers (e.g. DOIs; PMIDs) and text-mining. We improved the policy tracker in three key ways:
1) Accuracy: We aimed to improve both precision (are the citations we find accurate?) and recall (did we collect all of the citations in the policy document?) but to prefer the former. Focus was placed on fine tuning our text-mining methods to improve precision without sacrificing too much recall, and to eliminate false positives for research outputs that could be incorrectly matched to policy texts. This means we now have more policy mentions and fewer false positives.
2) Speed: the infrastructure of the new miner allows for more policy documents to be processed and mined. It also gives visibility to every step of this pipeline in our system so we can monitor exactly how policy mentions are associated with research outputs. Great for finding new mentions quickly!
3) Simplicity and accessibility: the old policy miner was complex and debugging was often time consuming. The new application is designed to provide the maximum amount of information regarding the text-mined document, the references extracted, as well as text-mining accuracy measuring tools. We can now debug with ease!
Brand new policy sources
We’ve also added a number of new policy sources, including:
- EU Policy Documents
- U.S. Agency for International Development (USAID)
- American College of Cardiology (Guidelines and Clinical Documents)
- Netherlands Government Documents
- Economic Policy Institute
- US Preventive Services Task Force
Our policy collection is growing all the time, with new sources being added every month. We curate policy sources that are designed to change or otherwise influence guidelines, policy or practice. Tracked sources range from government guidelines, reports or white papers; independent policy institute publications; advisory committees on specific topics; research institutes; and international development organisations.
We aim to curate a broad scope of policy sources from organisations around the world and cover topics from climate change to health, transport and economics.
Analysing policy attention in the Explorer
We’ve also added a new Explorer feature that makes it easier than ever to monitor policy attention! It’s now possible to search by a specific attention source, e.g. The Publications Office of the European Union, and see the research outputs mentioned in that source. In the Explorer, go to Analyze these Results, the Mentions tab and search for the policy source title as in the example below:
Add a timeframe if you’re interested in a particular period, export the mention data as a CSV or check out the details pages for more information about a specific output. You can also search for other types of sources and channels in this new Explorer feature, e.g. by Twitter handle or News outlet.
Want to find out more?
Contact us if there’s a policy source you’d like to check we’re tracking or if you have any questions about our new tracker!