Interviewed by Larry Au on November 4, 2022 for the Fall 2022 SKATOLOGY Newsletter

Juan Pablo Pardo-Guerra is an Associate Professor of Sociology at the University of California, San Diego, where he is a founding faculty member of the Halicioğlu Data Science Institute and a co-founder of the Computational Social Science program, as well as the Associate Director of the Latin American Studies Program. He is the author of the recent book, The Quantified Scholar: How Research Evaluations Transformed the British Social Sciences (2022), published by Columbia University Press. He is also the author of Automating Finance: Infrastructures, Engineers, and the Making of Electronic Markets (2019).

Q: One of the core arguments in The Quantified Scholar is that over time, British social science is becoming less innovative and becoming more homogenous due to the process of “epistemic sorting”. What effect does this have on subfields that are smaller like SKAT and STS, as well as particularly interdisciplinary fields that might take critical orientations like queer studies and disability studies?

A: One of the effects of this process of homogenization is that these fields become very standardized themselves. For example, STS, which of course has origins in the UK, has matured to become a very standard approach within British academia. Finding an STS scholar in a particular department isn’t a peculiar thing today as it might have been 30 years ago. And this means that the canon itself of what the discipline is, what the subfield is, has become much more paradigmatic in a way. This has downstream consequences for innovations within the field. The type of STS that we’re doing is much more “normal sciencey” than what it was 30 years ago. The way we hire folks into departments is also reflecting those new canons and those new expectations about what the field is and what should go into it. And this can have repercussions on dialogue between subfields and innovations within those subfields. Very critical fields, for example, critical cultural studies, have become untenable in many settings. We can see this in recent decisions by certain institutions to disband groups that name themselves as critical cultural studies, or that do the type of research that we would associate with critical cultural studies or heterodox approaches, more generally. This has negative repercussions on the type of knowledge that is being produced overall, because it becomes, again, a very “normal sciencey” kind of knowledge, which is fine at one level, but at another level creates spaces where innovation is impossible.

Q: Another core finding is that quantification is mediated by organizations. In the book, you write: “No matter how many similar themes emerged across my interviews with academics, I never found a single archetypical experience of quantification… It was all and always local, contextual, and variegated” (p.153). But as you write, through reflexive knowledge, “we can bend the bars of our cage into slightly less unwieldy structures” (p. 181). What are some practices that universities and departments can take up to resist these broader trends?

A: That’s something that will vary a lot across institutions. In some institutions, academics have more power: they have a greater role in the governance of the department, school, etc., than in other places that are much more corporate and have stronger management. But even in those settings, there are things that academics can do to make each other’s lives much more livable, humane, and inviting. That goes from decisions on how to train new scholars when we’re training graduate students: emphasizing certain types of expectations around productivity and around what is an ideal career. These are sites where we can also create alternatives: Alternative ways of being an academic, centering not necessarily prestige, but care and mutual recognition of the problems of others. Also, when we’re evaluating each other’s work, the simple act of peer reviewing can become very constructive as much as it can take the form of a disciplining activity. We choose. We have lots of liberty on the type of person that we want to be in those settings. 

There’s also professional settings where we can be very vocal about what we expect others in the field to do. I won’t name names, but some very prominent British sociologists who have moved into management have been responsible for creating very hostile places, and I think that it’s part of our responsibility to call them out for what they’re doing and not celebrate their scholarship without separating that scholarship from the type of administrators that they’ve become, and being very clear what they’re doing is contradictory, problematic, and not necessarily reflecting the ethos that we want in the profession. Trying to dismantle a little bit of this prestige by saying, “Hey, you know what? You’re a terrible boss”, is also part of the things we can do. I recall this wonderful essay by Tressie McMillan Cottom on shame and how shame has an important role to play in human relations. And the fact that we don’t shame people who create these hostile places is problematic.

Q: On that note, one of the things that has been on the mind of a lot of folks, especially on the SKAT Communications Committee is the loss—or potential loss—of academic Twitter. That’s one of the spaces that has created this alternate model of knowledge dissemination and celebration of people’s work—no matter where it is or what it is—creating forms of solidarity, even though at times performative. What are your thoughts on academia, social media, and how visibility is structured differently?

That’s a terrible loss because it was a site—or it still is a site—where we could connect in ways when it was difficult either institutionally or physically. For example, I ended up discovering tons of really amazing scholars in higher ed, not necessarily through journals, but through Twitter debates that were occurring amongst the younger generations of higher ed scholars. The loss of that space is critical because it was a—it’s a very non-hierarchical space. There are, of course, superstars and Twitter has its own politics of visibility and reputation, but at the same time, it’s a space that is flatter and much more connected globally than, for example, our professional associations. You could also see what was happening in STS, sociology, and the social sciences throughout the world, connect with journals, people, and institutions that, again, were very difficult to recognize from the US perspective or from a UK perspective, and also to amplify some of the issues that were happening or relevant to the community as a whole. For example, the labor issues that are central to UK politics right now are very visible in the Twitter sphere, which creates opportunities for demonstrating and articulating solidarity with our colleagues in Britain right now. The loss of that is huge because even though it’s a very dodgy platform for a number of reasons, we could create these spaces of solidarity that, even if performative as you mentioned, could generate true forms of solidarity through other means. I hope that as a community we find a space where we can do that. I don’t know if it’s going to be the migration to Mastadon—I have no clue. We did it before also with blogging, which has very different barriers to entry and involves a different level of investment on the part of individuals. We’re a bunch of bright people. We should find a solution to this, even if the solution is using Twitter, however problematic it may remain.

Q: I really enjoyed your use of different methods, primarily bibliometric analysis, computational text analyses, and also wonderful interviews with your respondents. I read the methodological appendix and know you’ve also written about this elsewhere on your extended computational case study approach. Could you say a few things about your approach to multi-methods research?

One of the things that inspired this project was the rise of all these really interesting quantitative studies on science, scientometrics, and the science of science, which triggered me to think about potential methodological approaches to study the impact of REF. But at the same time, having this STS training, I recognize the limitations of quantitative methods and counting things, and it’s a book about quantification—so quantifying things in order to write about the effects of quantification is a little bit strange. I found from the very beginning that the combination of methods would provide additional sources of evidence and would not only provide more legs to the table—which is like the standard metaphor of multi-method approaches—but would reveal things that are invisible using just one approach, either quantitative or qualitative. For example, going into this project, I confronted a very widespread discourse about research evaluations in the UK, which is they’re imposed by the government and administrators on academics. That is the standard narrative that you find in a bunch of work in the UK and a bunch of work on audit cultures, too. And it is confirmed by the quantitative data, the computational analysis, etc.: these evaluations seem to send shocks into the labor market that have negative effects and that resemble top-down interventions. But what happened through the interviews is that I would realize, when I presented the data or the results from the quantitative analysis to my informants, that they would confirm only some of that standard story. In particular, it provided opportunities to find “negative cases” that were not obvious under the standard account. Contrasting the quantitative evidence with the experiences of people in the system made me rethink a lot of the logics and mechanisms of quantification and led me down this path of “what matters is the organization not as much as the rules set from high above”. It also revealed these really interesting complex situations where scholars themselves feel like they needed to be quantified because it’s a way of getting public recognition, of getting funding at the end of the day, etc. So I think that one of the interesting things about using mixed methods is not only that they create more sources of evidence to bolster arguments, that they provide more scaffolds that make the structure of the argument sturdier by showing the contradictions between two bits of evidence that are interesting in and of themselves, and exposing more of the processes that are at work within a particular phenomenon. It’s something that is completely inspired by Burawoy’s extended case method, this idea of looking at different sites and then building theory out of that.

Q: Your work focuses on social sciences, but I can also imagine how quantification and the REF exercises have perhaps a stronger impact on the natural sciences and the life sciences, where maybe reflexive knowledge about their discipline is a little bit more scant and where a lot of these hierarchies and metrics are taken a little bit more on face value. What is your hypothesis or your understanding of how your finding can be generalized to the STEM fields, the natural sciences, and the life sciences?

I think what we would see in the natural sciences is a much more rapid convergence, which also has to do with the fact that natural sciences are much more paradigmatic in a very specific way. You can’t have like several theories of relativity going on for long periods of time. There’s a shared canon, so people converge much faster either because there’s methodological approaches that become common to the field or there’s like a common dogmatic theory, etc., that is quite central to the way people do work in the labs every day. Of course, there are paradigms in sociology, anthropology, politics, but they’re a little bit looser. You can still be a Marxist nowadays without being penalized. And there are more varieties of explanations about the same phenomena than in the natural sciences. 

The other thing is that in the natural sciences, moving is incredibly difficult. So there is less effect on labor markets themselves, because of the complexity of moving groups and labs. That changes a little bit the dynamic. There’s other evidence that shows that there are similar processes of homogenization in the natural sciences. The work by Pierre Azoulay and colleagues around star researchers who have early demises followed by a bout of creativity in their fields because they were effectively serving as gatekeepers is a great example. There is also of course the fact that metrics have a very different meaning and weight in the natural sciences (they believe, much more, in the impact factor than we do). The natural sciences are an interesting case. Someone should investigate! 

Q: One of the other things that struck me when I was reading your book was the role of US academic standards in influencing UK social sciences. It’s one of the axes of differentiation on Figure 5.2 on the different hierarchies of prestige. To what extent can we understand the REF or similar quantification metrics as pushing out research that’s highly local and nationally relevant in favor of more American-style economics, political science, sociology, etc.?

A: So definitely, that’s the key trend. One of the objectives of these evaluations is to identify leading research that is “internationally excellent”. The word “international” is actually a stand-in for “more American”. That’s very clear in economics. Economics has converged rapidly to a very standard way of being an economist, which tends to be the schools of economics that developed in the United States after the 1950s that are more math-heavy, financialized, econometric/microeconomic. When the evaluations introduce alleged global standards of excellence, they are actually centering the US as the site of knowledge production and creating incentives to be more Americanized. One of the interesting cases is anthropology, which was structurally different a couple of decades ago. anthropologists would sometimes say that, “back in the days, having an Oxford or a Cambridge book was perfectly fine because those are perfectly good presses. But nowadays, if you have a Chicago or a Duke that’s better for your file for getting a job”. That shifts the gatekeepers in a completely different direction. Indeed, British anthropology and US anthropology were very distinct up until 15-20 years ago. And they converged ever since, not to some mid-Atlantic position, but towards US models of scholarship. You see it in career structures too: it’s much more common to see the title of assistant professor, associate, and full professor being adopted in the UK nowadays, versus the traditional lecture, senior lecturer, reader, and full. 

Q: You write in the book that the UK is a neat empirical case in looking at the effects of quantification because it’s somewhat neatly bounded. As someone who was recently on the job market last year, one of the things that struck me was the sheer range of higher ed institutions here in the US: from public teaching universities, public research universities, elite privates, SLACS, community colleges, etc. It seems like there’s a wider range of universities and institution types here in the US. What kind of lessons we can draw from The Quantified Scholar for the landscape of higher ed in the US? How might quantification unfold differently in a SLAC versus a R1?

A: The comparison with the US is interesting because, as you say, the US is a very complex ecosystem. It has all these different types of institutions which serve different audiences and have slightly different trajectories and forms evaluating their academics. At the end of the day, the key lesson with the REF is that what was at play in that system isn’t really the financial incentives. The amount of money that is disbursed through the  funds associated to the evaluations isn’t huge. It’s significant, but it’s not what makes or breaks institutions. What ends up mattering on the ground is the prestige associated to standings in the evaluations and how it’s interpreted. That story of prestige is readily transportable throughout higher education systems, including the US system, where there is are also clear prestige hierarchies across institutions. Much of what we see in the UK, we would expect to see also in the US, not through formalized quantitative measures, but rather through the way we collectively inhabit and imagine shared, tacit hierarchies of prestige. We see this in the patterns of hiring in labor markets, in including classical studies of hiring in sociology (in particular, Val Burris’ work). And there are similar patterns of hiring in biology, computer science, and across a number of disciplines where the status of one’s institution becomes a marker that follows you throughout your career. That is interesting because, again, it’s very similar to the logics of REF. 

Q: What’s are you working on next after this book? Anything you can preview for the SKATOLOGY audience? 

A: I have two projects right now. One has to do with this spat I had on Twitter, on the politics of knowledge in Mexico, particularly in connection to the political change that has happened in the last four or five years. I’m interested in exploring what that political change has done to the status of scientists in the country. It’s really a story of scientists, intellectuals, and the state, and how political upheavals reconfigure the relationship between scientists, public intellectuals, and the state bureaucracies. In the case I am studying, I track the degradations of science’s position the country, both discursively and institutionally. That’s one project that I hope to finish relatively soon. 

The other one is a longer-term book project on budget models in US higher education. It’s very similar to The Quantified Scholar, but has to do with how these devices, which populate our organizations and are often invisible to most of us, are central to configuring the worth of knowledge in academia. For example, I investigate how budget models encode such things disciplinary values (the “value” and “cost” of, for instance, an ethnic studies scholar versus a computer scientist, both explicitly represented as numbers in a spreadsheet). By controlling the flow of resources, the allocation of positions, and encoding the ideals of what a campus should be like through financial means, these budget models end up becoming central to the governance of higher education and academic knowledge in the US. The fact that they are  often quite invisible and not contested makes them also quite interesting from an organizational perspective. So the next project studies the history and use of these budget models, to make sense of how they interact on and reproduce the racial, gender, and disciplinary politics of knowledge in US higher education.

Q: A final question—and partly selfish. Now that you finished two books and you’re a book-writing veteran. Do you have any advice for aspiring book writers in SKAT?

The first one is always the most difficult because you expect every word to be perfect. And I think that at some point you have to step back and let the book be what the book is. As your first project, it takes a lot of love, care, and anxiety. But in this process of love, care and anxiety, it is important to trust yourself and the book you’ve written to have the life it will have. This allows you to move on to the next project with less sense of loss. For example, my first book took forever to write. Every single word would be revised, I spent hours and hours working on every paragraph, and yet it’s imperfect. But that is fine. Books aren’t necessarily perfect and maybe shouldn’t attempt to be perfect. Great books have spaces of ambiguity that allow people to have conversations about the contents. I think it’s better to have those spaces of ambiguity than to try to achieve perfection because perfection closes the possibility for conversation. Something that helps you know if “you are there” is having a good editor. Find someone that you trust to work with and who’s comments you will welcome (not necessarily agree with, but certainly welcome). That’s really critical. And don’t be worried about approaching people. People actually want to help you. If you have an idea, pitch it and start developing it with a great supportive editor.