"Reforming academic assessment is a major challenge in Latin America"

---

Transcript of Karina Batthyány's column
in InfoCLACSO – September 20, 2023

The principles and values ​​of open science seek to develop alternative practices that improve both individual and collaborative research processes. They also aim to improve the dissemination of research findings and their reproducibility in order to accelerate the production and use of new knowledge by society. In other words, they seek to ensure that this knowledge, resulting from research, can be rapidly adopted by society.

We are at a point where there are recommendations, guidelines, and regulations at various levels to promote and implement open science practices. Therefore, different paths are emerging for scientific collaboration, which is based on principles such as openness to the growing potential for involving diverse citizens in the production and dissemination of knowledge. This, in turn, poses significant challenges to evaluation systems.

Ultimately, we are talking about open science, the idea of ​​knowledge as a public good, and how these two elements are articulated within scientific evaluation systems. One of the main criticisms of dominant evaluation practices points to the incorrect and widespread use of quantitative indicators based primarily on the impact factors of scientific journals and on the main publication circuits (which are often closely linked to commercial interests).

This is how academic and scientific performance is evaluated. For those of us dedicated to research, it matters little how much we may have accomplished in different fields if we do not meet these quantitative requirements for publication and indexing in those journals.

Indeed, a more widespread practice is occurring that we want to change. We must move away from these standardized rankings produced by providers, primarily commercial ones, and from data analysis. Furthermore, there is a strong correlation between publishing in high-impact journals and rising in the rankings of universities and knowledge-producing centers.

There is a need for quantitative evaluation to assess knowledge, but it must also support qualitative assessment. That is, the evaluation of knowledge and the people and systems that produce it should be qualitative, conducted by specialists, and in accordance with different institutional parameters and missions.

It makes little sense to evaluate a person based on their scientific, academic, and research career without relating or aligning it with the context of knowledge production, both the national context and the institutional mission of the place where that person carries out their activities.

At CLACSO, we want to discuss these topics at the seminar “Present and Future of Academic Evaluation and Open Science: Debates and Perspectives from Peru.” Since 2019, we have been promoting the transformation of current academic evaluation systems in our region, aligning them with the principles and values ​​of open access and open science, which are being strongly promoted regionally. That year, we created the Latin American Forum on Scientific Evaluation (FOLEC), a decision made by the CLACSO Executive Directorate, which quickly spread throughout the region, bringing together more and more national science and technology organizations and agencies that now comprise it and discuss these issues together.

We have already held three international seminars: the first in 2019 in Mexico City, organized jointly by CLACSO and the National Council for Humanities, Sciences and Technologies (CONACYT) of Mexico, where FOLEC was created. The second, in 2021 in Buenos Aires, in parallel with the Latin American and Caribbean Open Science Forum (CILAC). And the third, at our Ninth CLACSO Conference in 2022, again in Mexico. These three seminars have facilitated the mobilization and achievement of some consensus among various international, regional, and governmental actors and organizations, representing the main discussion on this topic in our region today.

FOLEC has also produced a series of documents, but it is primarily focused on proposing alternatives to these dominant evaluation indicators. I would like to highlight one of these proposals: the Declaration of Principles towards a New Academic and Scientific Evaluation for Socially Relevant Science in Latin America and the Caribbean.

The Declaration was approved by the CLACSO Ordinary General Assembly in Mexico in 2022. It establishes 14 principles, proposes an evolutionary, participatory and transparent evaluation, which effectively guarantees a science with social relevance in which open science, qualitative evaluation and principles that favorably weigh teamwork and that the academic community actively participates in the definition of the processes and indicators are valued.

Reforming academic assessment is a significant challenge. We need to take decisive steps toward this transformation, driven by our Latin American and Caribbean region. At CLACSO, we are already engaging in this discussion, exchanging ideas with the Global South on these proposals.

And finally, CLACSO's daily work on its different themes is indeed at the center of our work, because it is with this type of knowledge production and academic evaluation that we can guarantee the production of knowledge with social relevance, articulated with social organizations and movements and with an impact on the field of public policy.

– Why does academic evaluation become more complex because research lacks social impact?

– For two reasons: first, because publishing in these high-impact journals necessarily requires going through the commercial circuit of scientific publishing. In other words, if you look at these high-impact journals, most of them are not open access, because they charge for reading or publishing.

The second issue lies in defining the topics. For knowledge to have social relevance, the topics cannot be defined by international agreements or by what appears to be the mainstream in knowledge production. Furthermore, the possibility of defining research agendas in conjunction with social organizations and movements is often not considered a positive indicator of knowledge generation and the impact of that research.

– There could be a social scientist working on a topic of enormous impact on the community where they are conducting their research, but they need the money to present and publish their research in a scientific journal; and when it is published in that journal, a subscription must be paid to read its content. Therefore, there is a complex issue of social relevance…

If someone works on a topic relevant to the national context or to a specific local community, those topics are unlikely to be included in the research and publication priorities of those scientific publishing circuits. So, the research might have a significant impact locally or even nationally, but when it comes time to be evaluated by a university or other reputable organization, it will be very poorly rated because it didn't meet certain publication requirements in certain journals.

We want to emphasize that the unique nature of our sciences also requires indicators tailored to the topics addressed in the social sciences and humanities. Ultimately, this is the discussion we have been raising since 2019, and which will be revisited in Lima, Peru, at our seminar. We will revisit and reach a consensus on the purpose of evaluation and its instruments. Furthermore, we must understand that we are not only talking about technical procedures, but also about decisions that fall under the umbrella of science and academic policy.

– How do you see Our America when it comes to thinking about open science and (re)thinking academic evaluation?

– We've made enormous progress, because it's a topic that's now on the agenda. In that sense, FOLEC has contributed greatly to making this discussion visible, considering that Our America is very heterogeneous and diverse and difficult to standardize.

For example, the publication of books, which at times had lost its absolute value because the tyranny of scientific journals had become absolute, was revalued. In that sense, today we are beginning to observe the revaluation of books as mechanisms for the dissemination of knowledge, for the appropriation of that scientific knowledge by societies.

I believe we are witnessing a major step forward. I often find myself on the other side of the table, working as an evaluator in national systems, both in Uruguay and in other countries, and I must say that this transformation is noticeable. At CLACSO, we have repeatedly emphasized the importance of books, and today we are on the path to their (re)valuation. We must not forget the commercial interests and the industry behind scientific publications.

---

---