Altmetric Blog

What do syllabi-based altmetrics actually mean?

Stacy Konkiel, 27th September 2016

Altmetric recently partnered with the Open Syllabus Project in order to track mentions of books in syllabi.

In this post, I’d like to share some of the scientometrics research that inspired our decision.

As far back as 1998, scientometricians have recognized the value of mining online course syllabi for references to research:

“[References to research in syllabi] could, over time, provide a useful metric by which to gauge the extent to which a given professor’s (or research group’s) work…has diffused into the classroom. Syllabus-related mentions could, in theory, provide a useful complementary measure alongside, [among other things], citation data in assessing the overall impact of an individual’s contributions to the wider scholarly community.”

However, actual research on syllabi mentions as altmetrics has been surprisingly scarce, likely due to the fact that, historically, syllabi have been difficult to discover, parse, and aggregate mentions from.

A few articles published in recent years do give some insights as to the uses of this data. Most of what we know is due to two researchers from University of Wolverhampton in the UK: Mike Thelwall and Kayvan Kousha.

One study has suggested that the use of syllabi mentions as an altmetric could be especially useful in the humanities and social sciences, where monographs are often used to teach. In that way, syllabus mentions are a particularly valuable altmetric for books.

Interestingly, another study found that “books recommended by Choice reviewers for undergraduates were mentioned more often in online course syllabi than were other recommended books.”

As for journal articles mentioned in syllabi, a 2008 study showed that “the articles that were most recommended in academic syllabi tended to be reasonably highly cited but that the converse was not true”, pointing to educational impact for journal articles that is often unrelated to scholarly impact.

The same study also found that “online syllabus citations [to journal articles] were sufficiently numerous to be a useful impact indicator in some social sciences, including political science and information and library science, but not in others, nor in any sciences.” Put another way, mentions of journal articles in political science and information/library science articles are a useful impact indicator, but not so for other disciplines (as too few mentions can be found).

Older articles tend to be cited more often in syllabi, according to another study. That has an obvious effect upon using citations in syllabi as a means to measure recently published work.

Even though mentions in syllabi could tell us a lot about the educational impacts of research, Thelwall & Kousha have surmised that such data is not yet widely used in research evaluation.

We’ve also got a sociotechnical challenge to contend with: many syllabi are hidden behind logins on campus intranets and CMS’s like Blackboard. The Metric Tide has suggested that this limitation renders the use of syllabi mentions ineffectual for the systematic assessment of the educational impacts of research, at least for nationwide evaluation exercises like the REF.

However, as the use of open educational resources continues to grow, we’re confident that syllabi mentions will become an increasingly useful way to evaluate the educational impacts of research. Hopefully, by raising the profile of such data, we can help accelerate that change.

Further reading

Cronin, B., Snyder, H. W., Rosenbaum, H., Martinson, A., & Callahan, E. (1998). Invoked on the Web. Journal of the American Society for Information Science, 49(14), 1319–1328. http://doi.org/10.1002/(SICI)1097-4571(1998)49:14<1319::AID-ASI9>3.0.CO;2-W

Kousha, K., & Thelwall, M. (2015). An automatic method for assessing the teaching impact of books from online academic syllabi. Journal of the Association for Information Science and Technology. http://doi.org/10.1002/asi.23542

Kousha, K., & Thelwall, M. (2015). Alternative Metrics for Book Impact Assessment: Can Choice Reviews be a Useful Source? In International Conference on Scientometrics and Informetrics. Instanbul: ISSI. Retrieved from http://www.issi2015.org/files/downloads/all-papers/0059.pdf

Kousha, K., & Thelwall, M. (2008). Assessing the impact of disciplinary research on teaching: An automatic analysis of online syllabuses. Journal of the American Society for Information Science and Technology, 59(13), 2060–2069. Retrieved from http://doi.wiley.com/10.1002/asi.20920

Leave a Reply

Your email address will not be published. Required fields are marked *