Helsingin yliopisto
In recent years, the Research Council of Finland has been a pioneer in developing research assessment towards responsible and qualitative evaluation. For researchers, this cultural change in evaluation has been reflected in, among other things, a reduction in the emphasis on citation metrics and a greater emphasis on qualitative evaluation of research outputs and career paths. What has led to the introduction of responsible and qualitative evaluation, how is it implemented in practice and how will it be further developed? Authors from the Research Council, Kata-Riina Valosaari, Counsellor of Science, and Riitta Maijala, Vice President for Research, address these questions in their article.
(Tämä artikkeli on saatavissa myös suomeksi.)
Text: Kata-Riina Valosaari and Riitta Maijala (Research Council of Finland)
New forms of sharing research results, evolving research processes, multidisciplinarity, phenomenon-based research methods, digitalisation and researchers’ career paths are changing. This should be taken into account in the development of research evaluation. Research careers also increasingly involve working across different sectors and in a multidisciplinary and diverse community.
A researcher’s job already involves much more than just research and publishing the research results. It is obvious, therefore, that the evaluation of researchers and research can no longer be based, even in its simplest form, on a single research output, i.e. publications, or on the journal impact factor (JIF), which is problematic for evaluation in general. For more on this topic, see for example Howy Jacobs’ blog post Taking back control (Professoriliitto.fi 2.11.2023).
The need to change evaluation criteria and methods has emerged strongly from the research community. Over the last few years, significant steps have been taken to raise awareness and promote this issue in both national and international contexts.
Evaluation has a powerful impact on a researcher’s work and career because it enables, shapes and guides research. Change must therefore be controlled and based on broad, participatory and collaborative processes. As research is conducted, published and evaluated internationally, it is important that the Finnish evaluation method and system does not lag behind – they should enable and support up-to-date evaluation of research and researchers.
When designing an evaluation, we need to identify the current scientific community and the work of the researcher. It is also necessary to identify the type of competences required. The evaluation of research and researchers is linked to many aspects of a research career, such as recruitment, applying for funding and publishing research results. Evaluation should more clearly recognise and value the diversity of research, researcher competences and skills, and researcher career paths. To support this, the European Commission has published guidance on ways to make career diversity visible (Knowledge ecosystems in the new ERA – Using a competence-based approach for career development in academia and beyond).
The abandonment of old evaluation methods can be perceived as a threat if they have defined one’s career merit through criteria such as publication and citation metrics.
While the reform of researcher evaluation has emerged strongly from the research community, its widespread recognition and acceptance will take time and active measures – as with any major cultural change. Right now we are in a period of transition. The abandonment of old evaluation methods can be perceived as a threat if they have defined one’s career merit through criteria such as publication and citation metrics. Young researchers may also find it challenging to plan their own career path in a period of changing criteria. The development of assessment needs to take these aspects into account and accept that such a major change will take place slowly.
The Research Council of Finland’s new strategy emphasises responsibility
Responsibility has been a key principle in the Research Council of Finland’s work for several years, and now it has also been put at the heart of this year’s updated strategy. The aim is to make the Finnish scientific community a pioneer of responsible science. The Research Council’s mission is to open up new avenues for high-quality, responsible and impactful research.
Research Council’s research funding is competitive, and the high level of international peer review is the main tool for identifying the best and most promising projects.
Evaluation is therefore at the heart of the Research Council’s activities, and the Research Council has invested and continues to invest in its development. When an evaluation is planned or previous practices are updated, it always starts with the objectives of each funding application that the evaluation aims to meet.
The high level of international peer review is the main tool for identifying the best and most promising projects.
The values guiding the evaluation are transparency, integrity, fairness, competence and diversity. In practice, these values take many forms. The objectives and decision-making principles of the research funding schemes are described openly on the Research Council’s website for each call for applications. The evaluation criteria, guidelines for evaluators and evaluation forms are also published in the call for applications.
The Research Council invites mainly foreign experts as evaluators. International evaluation is a prerequisite for recognising the quality of research in international comparison. The evaluators are recognised experts in their field or otherwise peers in relation to the applications being evaluated – this is of course in line with the objectives of the calls. International evaluation also helps to avoid the most common situations where evaluators would be biased against the applicant. Typically, the evaluation takes place in multidisciplinary panels. The applicant will receive a statement with the names of the experts who evaluated his/her application. Following funding decisions, the Research Council will also publish on its website the names of the reviewers who have evaluated the applications.
The evaluation by the research funder differs from recruitment situations, for example, in that the evaluation is mainly focused on assessing the research plan, not just on the merits or achievements of the applicant. However, the evaluation of the applicant is central for example in the case of applications for Academy Research Fellow.
Research Council develops evaluation as part of the scientific community
The launch of the national preparation of the concept and objectives of responsible researcher evaluation, coordinated by the Federation of Finnish Learned Societies (TSV) in 2018, gave the Research Council of Finland an opportunity to examine its own principles more critically. In 2018, the Research Council signed the Plan S commitment to open access to scientific publications, in 2019 the San Francisco Declaration on Research Assessment (DORA), and in 2020 the recommendation for the responsible evaluation of a researcher in Finland and the Declaration for Open Science and Research 2020–2025. The most recent commitment is the signed international Agreement on Reforming Research Assessment (ARRA) – to promote the implementation of this commitment, the Coalition for Advancing Research Assessment (CoARA) and its working groups were established in 2022.
Many of the principles of responsible peer review were already commonplace before these commitments, but with the commitments, the Research Council’s application review process began to evolve in a more qualitative direction. The first changes to the application and evaluation guidelines came into force in 2021. As before, the application will be evaluated as a whole, with the focus on the research plan, but with a greater emphasis on the qualitative assessment of the applicant’s competence in terms of research outputs and career paths. The researcher’s achievements will be assessed through a wide range of research outputs and career tasks, not through the number of publications or citations.
The Research Council of Finland had already adopted a template for researcher’s CV in line with the recommendations of the Finnish National Board on Research Integrity TENK. The template supports the goal of responsible researcher evaluation and guides the evaluation broadly towards the outputs of a research career and multiple career paths. The CV template also highlights the promotion of open access to research, as well as societal impact and interaction.
The use of citation metrics – if metrics are used at all to support qualitative evaluation – should be responsible and take into account the differences between disciplines, the multidisciplinary nature of the research and the stage of the researcher’s career.
The Research Council of Finland guidelines for evaluators stress that the use of citation metrics – if metrics are used at all to support qualitative evaluation – should be responsible and take into account the differences between disciplines, the multidisciplinary nature of the research and the stage of the researcher’s career. The use of journal impact factor (JIF) or Jufo classifications is explicitly prohibited in both the evaluation and decision making process, and applicants are not permitted to include JIF or Jufo information in the application. Other specific changes for applicants have been the description of the main research outputs and publications and their importance or relevance either for their research career or for the implementation of the research plan applied for.
During the reform of the funding scheme for Academy Research Fellow in 2022, a narrative section on ”merits and increased competencies” was also added to the applicant’s CV. In this section, the applicant can describe in his/her own words the merits as a researcher to date, the renewal of the research idea proposed in the application after the doctoral thesis, targeted qualification as a researcher during the funding period and the career plan after the funding period.
As part of the responsible evaluation process, the Research Council of Finland also takes into account the balancing of work and family life, and career breaks are taken into account in the evaluation of the researcher’s achievements, in line with established practice. For example, breaks in a research career related to family leave should not negatively affect the assessment of the applicant’s merits.
Working together and learning from each other for cultural change
In Finland, the need to change the principles of responsible researcher evaluation was identified early on, by international standards. A recommendation for the responsible evaluation of a researcher was completed in 2020 under the leadership of the TSV, which also coordinated the implementation of the measures in the national recommendation and the establishment of the national branch of the CoARA coalition. The majority of Finnish research organisations, 28 in total, including all universities and a large number of universities of applied sciences, have joined the CoARA coalition and committed to promoting its objectives in their own activities.
Research and research evaluation are international activities. A shared understanding in the scientific community of what quality and impact in science are and how they should be assessed is essential. Therefore, international harmonisation of evaluation methods and principles is important. Harmonisation of criteria facilitates international cooperation between researchers and the practical organisation of international peer review. In addition, appropriate harmonisation of methods and materials will help to ensure that the workload of applicants and reviewers does not increase unduly.
Several projects and working groups have been launched at international level, particularly in Europe, to help change the culture of evaluation. One of the initiatives of the European Research Area (ERA) policy agenda (no 3) is harnessed to promote a cultural change in evaluation. The search for a common vision and approach to change among European research funders is being pursued, for example through the Science Europe’s Research Culture group. In addition, one of the ten working groups set up under the CoARA umbrella focuses specifically on improving the evaluation of research proposals by research funders. The Research Council of Finland participates in the activities of the afore mentioned groups.
CoARA also requires its members to make an action plan commitment to implement its recommendations.
Training, monitoring and follow-up
The move to qualitative evaluation requires the adoption of new methodologies and also targeted guidance and training for researchers, whether they are applicants or evaluators. Decision-makers also need guidance and training.
Each year, the Research Council of Finland organises a series of Ask & Apply application advice sessions for applicants in conjunction with its main funding call, the winter call. The winter call 2024 sessions, in particular those on Academy Research Fellow qualifications and evaluation principles, will also provide an introduction to the practices of responsible researcher evaluation.
In addition to the responsible evaluation guidelines (Review principles, pdf), there are also introductory webinars for evaluators, where the Research Council’s policies are discussed with them before the evaluation work. A one-size-fits-all approach to evaluation is not appropriate for every situation, and evaluation must be adapted to the objectives of each situation. However, different situations are guided by a common set of principles for responsible evaluation. The Research Council of Finland is constantly developing its evaluation practices on the basis of feedback from applicants, evaluators and other stakeholders.
It is important that the Finnish scientific community actively monitors and develops the evaluation of research and researchers as part of the international scientific community. The Research Council of Finland also wants to play a strong role in this joint effort.
Kata-Riina Valosaari, Counsellor of Science, leads the team responsible for review and decision-making at the Research Council of Finland. Riitta Maijala is Vice President for Research at the Research Council of Finland.