The Numbers Game Blaise Cronin, Rudy Professor of Information Science, Indiana University Bloomington, gave the lecture “The Numbers Game: Scholarly Communication and Digital Metrics” on Monday 10 December 2012 at City University London. He was introduced by Professor David Bawden.

Professor Cronin’s extensive and influential publication record in the field of Library and Information Science covers citation analysis, informetrics, scholarly communication and strategic intelligence. In a pioneering article published in 1998, Cronin et al argued that the web fostered new modalities of scholarly communication and looked at where, how, and why scholars were invoked on line, gathering comparative data from five commercial search engines and proposing a typology to capture genres of “invocation” (what we now call “mentions”) in order to assess their potential to inform sociometric and bibliometric analyses of academic interaction.

During his lecture, Professor Cronin presented a balanced and thorough overview of the current debates around academic publishing and alt-metrics, their history and the present and future challenges and opportunities they pose to scholarly communications. He described how scholars have a great diversity of communication media and publication platforms at their disposal, and reminded us scholarly activity is routinely defined by submissions to repositories and open access journals, the creation of videos and uploading and sharing of data sets and slide presentations, the publication of blogs, the use of reference managers and the interaction with peers on social media platforms such as Twitter.

A new age of numerology? Blaise Cronin at City University, London, 10 December 2012

A new age of numerology? Blaise Cronin at City University, London, 10 December 2012

 

The lecture began setting up the scene with the “big business” of traditional academic publishing, and introduced Open Access alternatives as a mechanism to remove tolls and filters and favouring access and transparency, open peer review and the wisdom of the crowds. Cronin called Open Access a “liberation technology” that flipped the dominant model of scholarly publishing, shifting costs through the Article Processing Charge, redefining peer review and enabling the transparency and visibility of the level of impact of scholarly records.

The lecture introduced students and academics in the audience to the major platforms and tools redefining the scholarly communication landscape of today. He took us on a journey that started with a description of the state of Open Access Publishing (Laasko et al 2011), covering the Finch Report’s “commitment to support the costs of an innovative and sustainable research communications system” and showing Plos One as a peer-reviewed, open access “publishing phenomenon,” referred to a scholarly “worker’s revolt” against pay-walled publication models and described the paradigmatic changes in scholarly production that set the scene for alt-metrics.

Professor Cronin compared a previous model of scholarly activity which was done individually and in isolation, through fixed scripts and on printed media to a newer model which is connected, collaborative, mutable, through mobile scripts and on multiple media. He referred to two different sets of scripts:

From 1970:

  • Working paper
  • Occasional paper
  • Preprint
  • Conference paper
  • Journal article
  • Letters
  • Research notes
  • Monograph
  • Book review
  • Lit. review
  • Encyclopedia

From 2010. Same as 1970 plus…

  • Journal article and supplementary materials
  • Hyperlinked articles
  • Tweets
  • Blogs
  • Videos (YouTube)
  • E-preprints
  • Webpages
  • Webinars/podcasts

He also spoke of scholarly microscripts (such as tweets) and nanopublication, referred to the Force 11 Manifesto, which proposes a future where “every significant element of the discourse can be explicitly represented, along with supporting data, software, workflows, multimedia, external commentary and information about provenance”, and addressed the Research Excellence Framework’s definition of impact beyond academia and the need to go beyond bibliometrics to assess said impact.

The lecture set the rise of alt-metrics in a historical context, inserting them in a genealogy including  the past (c1955): citation indexes (SCI, SSCI A&HCI), the present: WoS, Scopus, Google Scholar, etc. and the future  social media monitoring and analytics.

Professor Blaise Cronin at City University London, 10 December 2012

Professor Blaise Cronin at City University London, 10 December 2012

 

Showing examples from Plos article-level metrics, F1000, Altmetric on Scopus, ImpactStory and Science Direct amongst others, Professor Cronin explained how these platforms provide new methods to assess scholarly engagement with articles which have bee invoked on the web; tracking digital footprints in real time.

He explained that since traditional citation counts are not obtained in real time and are not fully reflective of a scholar’s “true contributions,” a “battery of alternative metrics” can now supplement bibliometric indicators. He explained that alt-metrics provide indicators of the times “a scholar’s work is downloaded, acknowledged, included in syllabi, quoted in the press, cited in policy documents, recommended by others, praised by opinion leaders, mentioned in social media.”

Citing Lozano, Lariviere and Gingras, JASIST (in press), Professor Cronin argued alt-metrics go beyond the Impact Factor (IF):

“… high IF journals are losing their stronghold as the sole repositories of high-quality papers, so there is no legitimate basis for extending the IF of a journal to its papers, much less to individual researchers.”

The lecture proposed a future in which alt-metrics could provide academics and academic managers with “Academe’s Equivalents of Q- and Klout Scores” through an “all-source, multi-metric, comparative performance indicator dashboard” that could provide insight into complements or correlatives between positive mentions on social media and reference managers and citations and downloads. As examples, he cited evidence that

• Citations in Wikipedia correlate with JCR data (Nielsen, 2007)
• Articles in the top quartile of tweets were 11 times more likely to be in top quartile of citations 2 years later (Eysenbach, 2012)
• High positive correlation between F1000 score and JIF (Nature Neuroscience, 2005)
• Positive correlations between inclusion in reference managers and citation
(Bar-Ilan, 2012; Li etal. 2011; Priem et al., 2012)
• Downloads predict /correlate with subsequent citation (Brody et  al., 2006).

The lecture also referred to the perceived dangers of “cybernating the academy” and falling into “an Orwellian surveillance net” (Sosteric 1999), and explored issues relating to validity, reliability, utility, comparability and ethicality in the metrics themselves, and of transparency, usability, persistence and cost in the alt-metrics platforms.

He suggested alt-metrics could be seen as “the holy grail of holism”, offering multidimensional indicators of impact, influence, worth and trends through a a matrix of established & alternative metrics, aiming perhaps in the near future towards a unified measure or composite score.

Nevertheless Professor Cronin warned of the possibility of “a new age of numerology” where there would be atomization of efforts, outputs and impacts, fetishization of metrics and a triumph of triviality over transparency.

What should be captured and counted, by whom and for what purposes? Will reputation management and astrategic gaming of the system become unedifying staples of scholarly life in an age of digital analytics? Will triviality trump transparency?  These were some of the questions the lecture left open as food for thought and urgent work to be done by all of us.

With thanks to Dr Lyn Robinson for the invitation to attend this lecture.