We’ve been working on our policy & grey literature datasets recently, adding an extra dozen or so new sources that we’re now mining for mentions of published research. Typically this is a three step process: we first need to identify potential sources of policy, guidelines and other relevant grey literature (this is one of the things we ask customers during on-boarding and renewal), then to figure out how / where to get the actual documents to mine, and finally to do the actual development work to process them on a regular basis.
As with our mainstream media sources we’re aiming for international coverage across a range of disciplines – though in practice this can be tough. I think we’re doing pretty well covering international NGOs, Europe, North America and Australia, but we definitely have a ways to go in South America, Africa and SE Asia.
One thing I’ve talked about in the past is how the policy data we hold is different to other sources of altmetrics in that it usually takes a while to accrue. The half-life of attention from policy documents is closer to citations (in that it is measured in years) than it is to, say, the news cycle (measured in days or weeks). That’s partly just because guidelines, reviews and policy take a long time to get commissioned, put together and released.
It’s also difficult to figure out what effect the policy actually had: did a document get read and put into action? Did it influence any decisions? Or did it end up at the bottom of a filing cabinet drawer?
Sometimes the answer to the latter question is easy. This week the UK will be voting in a referendum on whether or not to leave the EU – to “brexit” the European Union. Altmetric is based in London so the ins and outs of the campaign has been a popular lunchtime topic of conversation.
We’d coincidentally just finished mining documents from the Institute for Fiscal Studies (IFS) here in the UK when they released their report on the economic impact of leaving the EU (David Cameron promptly pointed to it in a speech. Here’s the BBC story about it).
One point made in the report is that immigrants actually have a positive effect on public finances. To support that assertion the IFS cites research done at UCL and the University of Milan by Dustmann et al. We can see from the article’s Altmetric details page that the same research is also cited by grey literature documents from Chatham House (a UK think tank) and the IMF – and was picked up back in 2014 by outlets like the Financial Times (which ran an editorial two weeks ago noting that “low-skilled EU migrants meet a demand in the economy. They raise overall output and are net contributors to the public finances”).
More globally we also started mining documents from the Centers for Disease Control and Prevention (CDC) in the US. A few documents from here demonstrate how the long “citations in grey literature” half life is only a rule of thumb…. sometimes it isn’t very long at all. For example, the interim clinical guidance for pregnancy & Zika cites articles published only a few weeks earlier (like this research from the March 2016 issue of NEJM)
Collecting data and stories like these remains a core part of what we’re trying to do with altmetrics and tools like the Explorer for Institutions. You can see some of the fruits of this work already in our tools, but there’s still lots to do.
In particular we’ve got to keep building up our corpus of data: please do email support and let us know if you’ve got suggestions for places to track (global sources are great, as I mentioned above, but we’re also keen to hear about US state level policy sources and interesting NGOs).