Altmetric Blog

When Metrics and Politics Collide: Reflections on Peer Review, the JIF and Our Current Political Moment

Guest Author, 18th September 2017

Editor’s Note:  Today’s post is from Sara Rouhi. Sara manages business development for Altmetric in the US and Canada. She is an active member of the Society for Scholarly Publishing’s Education Committee and the 2015 recipient of their SSP Emerging Leader Award. We’re cross-posting today with the Scholarly Kitchen blog.

This past week, three distinct moments came together for me prompting me to write this blog post: a peer review conference, a debate on the journal impact factor (JIF), and an art exhibit.

I’m just coming back the 8th International Peer Review Congress (PRC) in Chicago. This once-every-four-years meeting brings together journal editors, authors and anyone else working in the journal editorial or production space to discuss the “big questions” facing the peer-review process.

While in Chicago, I visited the Museum of Contemporary Art (MCA) – one of my favorite US art museums – to explore their Takashi Murakami retrospective and stumbled upon the “Politics and Power Plays” exhibit featuring artists examining the intersection of politics and power.

Several weeks prior to this trip, I had the privilege to participate in the Charleston Library Conference’s famous “Hyde Park Debate.” This Oxford-style debate (complete with prepared opening statements and first rebuttals) posits a statement and asks experts to argue for and against it. This year the debate statement for support/rebuttal was “Resolved: The Impact Factor does more harm than good.” (Which do you think I argued?)

What do a journal conference, art exhibit, and webcast debate have in common?

imbalanced scale

The “Politics and Power Plays” exhibit and the PRC meeting/Charleston debate examined similar themes at the macro and micro levels respectively.

The MCA exhibit used art as form to highlight how structural inequities in power, access, and opportunity are foundational to liberal democracies and may ultimately render them unstable. The PRC and Charleston debate – with their examination of the inherent biases in the western research paradigm – emphasized how the barriers to entry in this space are silencing voices whose research is vital to the research endeavor.

Charleston Hyde Park Debate: Beyond the Usual Arguments

When I initially accepted this exciting challenge, I planned on taking the standard approach and arguing the “go-to” list of Impact Factor “against” positions everyone is familiar with. When I really stopped to reflect on the topic, my inner political scientist rose up, and I realized that despite being one of the few employees at Altmetric/Digital Science with a social science background, my expertise in sociological and political theory might finally be brought to bear on the thorny question of metrics in the research paradigm.

My undergraduate and graduate degrees were both in political theory and the study of how culture impacts the formation of definitions of self and community (and by extension, politics). Evaluating this year’s Hyde Park debate statement through the lens of critical and post-colonial theory, I realized there was a lot more to probe about the journal metrics than just their pros and cons as evaluative tools.

I decided on more a radical approach toward my argument and focused on the structural inequities perpetuated by journal based metrics, rather than focusing on the same arguments everyone has heard. My hypothesis/argument was simple: the group in power makes the rules and the rules typically favor the group in power.

In this case, I coined the term “western research industrial complex” (because who doesn’t love adding ‘industrial complex’ to anything they’re arguing against?) to describe the dominant power paradigm of the global research landscape: white, western, English-speaking men and the prestige institutions they represent.

These folks created the paradigm, its rules of engagement (Study with this professor! Postdoc at that institution! Publish or perish!), its costs for entry (JIF! Citations!), and its prizes (Tenure! Grant money! Status!). Everyone else – the global south and non-white, non-western countries – starts at a disadvantage by not belonging to the dominant group and constantly confronting the structural inequities of that exclusion.

The Peers Who Review Me: White, Western, Men

The PRC meeting further examined these questions with almost half the presentations addressing bias in the peer-review process: a bias that generally excludes those outside the “research industrial complex.”

The big question facing authors, editors, and publishers at this meeting was the question of whether and how to tackle the inherent biases built into the peer review paradigm. This paradigm is dominated by white, western, English-speaking men who have come of age in the “research industrial complex” and benefit greatly from their association with it (see my comments on Eugene Garfield and his development and monetization of journal metrics in the Hyde Park Debate opening statement here.)

While this group likely does not intend to allow bias into the peer review process, the data examining the outcomes of this process tell a different story. Research going as far back as 2013, confirms that for various structural reasons, women receive fewer citations than men. The journal selection, peer review, and acceptance process is a critical part of this. (See Alice Meadow’s report on PRC from last week’s Scholarly Kitchen.)

When speaking to the powers-that-be in western publishing, the chorus is often the same. “We’re scientists. We’re objective. We’re not biased in our assessments. If the research is good, it will get published.”

Alas, as much of the research at PRC proved, that is not the case. While these individuals genuinely believe they’re objective, the idea of a truly “objective” human being is laughable. Max Weber, Emile Durkheim – indeed any student of sociology would snort in incredulity at the idea that a lifetime of experiences does notcolor judgements about the world.

It’s the definition of privilege to make this naïve assertion. Ask any woman who has mulled over what to wear to a job interview or any black person who has contemplated expressing an unpopular opinion in the workplace. Can’t identify with those examples?

Consider the Latina who honestly admitted that her background as a “NewYorkrican” would impact her perspective as a judge on the Supreme Court, Sonia Sotomayor. Essayist and public intellectual, James Baldwin, writing during Jim Crow put it another way:

For it means something to be a Negro, after all, as it means something to have been born in Ireland or in China, to live where one sees space and sky or to live where one sees nothing but rubble or nothing but high buildings, We cannot escape our origins, however hard we try, those origins which contain the key – could we but find it – to all that we later become. [emphasis mine]

We bring our experiences, exposures, and education (or lack thereof) to everything we do and the presumption of objectivity does the greatest disservice to those most effected by bias.

And none of this is new.

Playing Politics: The Quest to Maintain Traditional Hierarchies

Indeed, theorists of power (most famously Friedrich Nietzsche, Michel Foucault, and Hannah Arendt among others) have been top of mind for me lately as they all highlight the same underlying truth: Systems of power insinuate themselves so subtly that their captives not only accept but embrace the norms and conventions of the system even if those norms disadvantage them.

The “Politics and Power Plays” exhibit speaks directly to this insinuation. The opening statement of the exhibit reads:

Power profoundly affects our lives yet often operates invisibly; consequently it’s easy to take it for granted. This selection of art drawn from the MCA Collection considers the ways that artist give power a form in order to name it and work with or against it… All of the works look critically at power structures from the position of the artist, who stands outside traditional hierarchies. [emphasis mine]

The challenges seen in the US – whether it’s a question of taking down Confederate statues or allowing transgender citizens to serve in the military – are more apparent than they have been since the civil rights movement of the 1960s.

This past election revealed that many of our neighbors, friends, work colleagues, and family members value traditional hierarchies and maintaining them. While the American political experiment was built on the promise of “equality under the law” and “life, liberty, and the pursuit of happiness,” it’s easy to forget that those inspirational documents were forged under an economic system reliant on chattel slavery, where each black person counted as only 3/5 a person.

This past election did us the painful service of momentarily revealing the power matrices that ceaselessly entangle the many while benefiting the few. Can I fly back to the US from a visit to Iranian relatives in Canada while traveling on a green card? Will my cleaner, an immigrant with no papers, be deported despite having American children, owning her own business, and paying taxes for decades in the US? Will my recently married gay friends still be recognized as married by the state? Will my elderly black neighbor still be able to vote when her drivers license has expired and she has no access to the internet?

Before we can address the impact of traditional hierarchies on those at the bottom and top of the food chain, we must observe that they exist and unflinchingly acknowledge where we sit within that hierarchy – journal editors and engaged citizens alike.

As a woman, an immigrant, the child of a Muslim, and a person with brown skin – how long will being a US citizen protect me?

These are real questions I have asked myself in the last 7 months since my friends, colleagues, family, and fellow citizens elected an ethno-nationalist president.

So all this to say…

Before we can address the impact of traditional hierarchies on those at the bottom and top of the food chain, we must observe that they exist and unflinchingly acknowledge where we sit within that hierarchy – journal editors and engaged citizens alike.

Academic publishing and the research industrial complex particularly need this kind of self-reflection to acknowledge their role in vetting research and passing judgement on what is included vs. excluded. We don’t need to read the latest literature on peer review bias or to (re)read Discipline and Punish to see the effects “politics and power plays.”

It’s incumbent upon all of us – trapped within the myriad hierarchies that we inhabit – to shed light on them and call out inequities as we confront them, whether from a privileged position or not. (My argument for the Charleston Hyde Park debate was my small attempt at doing this.)

As I fly back from Chicago relieved that my “Premier Access” status allows me to skip the line and check a bag with no fee, I find I’m running into (and benefiting from) structural inequities at every turn. 90% of the battle is recognizing them and the last critical 10% is deciding which side you’re on.

So whether you’re pro- or anti-JIF, or pro- or anti-Brexit/Trump, remember that these inequities are all around us. Notice them, check your privilege, and do your homework.

This blog post does not reflect the position of Altmetric or its employees and is solely my position as a member of the resistance.

Leave a Reply

Your email address will not be published. Required fields are marked *