Reviewing Reviewers: Advanced Evaluation and Rewarding Reviewers of SCIndeks Assistant journals
Centre for Evaluation in Education and Science, Belgrade
Abstract: The assessment of reviewers by authors and editors is enabled by the software tool Reviewing Reviewers (RevRev) built in SCIndeks Assistant. By responding to the RevRev questionnaire editorial boards evaluate usefulness of peer reviews for making their editorial decisions, while the authors evaluate usefulness of reviews for improving the quality of various parts/aspects of their manuscripts. Consequently, two versions of the questionnaires are largely different. For editorial boards, the procedure is single-, while for the authors double-blind. The results are available only to the editors to be used in deciding on the possible re-engagement of reviewers, as well as to the CEON/CEES for the awarding of annual prize and recognition to reviewers. As an innovative procedure whose justification has yet to be shown, the evaluation of reviews by the authors of manuscripts was received with scepticism and even resistance in some journals.
Identifying, training, evaluating, and rewarding reviewers is an important part of the editorial work (WAME, 2001). At the same time, the editorial boards of the journals face difficulties in providing impartial, responsible and competent reviewers on a daily basis. Finding engaged, dedicated and competent experts in a particular field who are willing to invest their time and effort, most often as volunteers, is a problem in world scientific publishing. Rejecting a request for a review, as well as breaking deadlines are just some of the many difficulties faced by editors (Zaharie & Osoian, 2016). The problem is more pronounced in the so-called small, low-impacted journals, such as the majority published in Serbia. Therefore, the greater importance to peer reviewing and reviewers is given in the scientific periphery. What is also characteristic of such environments is the emphasis on the reviewers obligation to help the authors of the manuscript with their useful suggestions to refine and improve the final version of their works. Thus, a domestic author, otherwise distinguished in the field of scientific publishing, entitles her activistic article with "Reviewer - the greatest friend of the author" (Vučković-Dekić, 2013). The specific role assigned to reviewers on the scientific periphery is a natural expression of the need to raise the modest quality of papers in local journals with the assistance of reviewers as experts.
Precisely such a motive drew CEON/CEES to upgrade the existing reviewer evaluation system and introduce special recognitions for the best reviewers. For technical reasons, a tool supporting the evaluation, called Reviewing Reviewers (RevRev) could only be applied in those SCIndeks journals that are edited online, through the SCIndeks Assistant as a platform for electronic editing.
Evaluation of reviewers by editors in RevRev is mandatory, while rating by authors is optional, although editors are expected to certainly provide author ratings. The editorial boards use the Report for decisions on the future engagement of reviewers, while CEON/CEES uses them to award a monetary price "Reviewer of the Year" and the recognitions "Outstanding Reviewer" (CEON/CEES, 2019).
Editors evaluate reviewers by using a questionnaire, which becomes available to them in RevRev after the review has been submitted. On a three-point scale, four features of each review are evaluated: clarity, exhaustiveness, argumentation, and timeliness (Figure 1.; the form exists only in Serbian).
Figure 1. Assessment form for editors
Individual scale values are combined into an average grade and accumulated for all reviews of each reviewer engaged in the journal. Reviewers' actual ratings are visible to the editor in the part of the application that he/she uses when selecting reviewers.
The evaluation form used by the authors becomes available to the author immediately after the editorial decision on accepting/rejecting a paper. The authors also evaluate four, but significantly different features of the peer review than those used by the editors (Figure 2.). It is understood that the authors have no insight into the identity of the reviewers. If the submitted paper is multi-authored, the corresponding author is expected to consult co-authors during the evaluation.
Figure 2. Assessment form for authors
RevRev is designed to make it easier for editors to select reviewers, giving them a clear insight into their past performance. It is our intention to encourage reviewers to respond more easily to editorial calls and put in more effort, especially in assisting authors of the manuscript with helpful recommendations. At the same time, we intend to encourage the competent regulatory bodies in Serbia to start respecting the reviewer's contribution in normative acts for evaluation and promotion of researchers. Results of some research show that most reviewers believe that their work is inadequately recognized (Warne, 2016). In that way, a long-term effect of introducing this innovation could be not only higher quality of papers in SCIndeks journals, but also the improvement of local science as a whole.
References:
- WAME. (2001). Report of the World Association of Medical Editors: Agenda for the Future. Croat Med J, 42, 121-126.
- Warne, V. (2016). Rewarding reviewers – sense or sensibility? A Wiley study explained. Learned Publishing, 29, 41-50. doi:10.1002/leap.1002.
- CEON/CEES. (2019). Award and acknowledgments for the reviewers of SCIndeks Assistant journals established. Retrieved on 14. 03. 2019. from https://www.ceon.rs/index.php?option=com_content&view=article&id=615:award-and-acknowledgments-for-the-reviewers-of-scindeks-journals&catid=170&lang=en&layout=post&Itemid=574.
- Vučković-Dekić, Lj. (2013). Recenzent – najveći prijatelj autora. Biomedicinska istraživanja, 4(1), 75-79. doi:10.7251/BII1301075V.
- Zaharie, M. A., & Osoian, C. (2016). Peer review motivation frames: A qualitative approach. European Management Journal, 34(1), 69-79. doi:10.1016/j.emj.2015.12.004.
