Published as: Šipka P. (2018). Nepotpuna, netačna i nestručna, ali zauzvrat skupa: Bibliometrijska analiza MI SANU 2016, CEON/CEES Occasional Paper Series, No. 2018-03-1. ISBN 978-86-89475-04-3. http://doi.org/10.5937/cees-ops-2018-03-1
Incomplete, inaccurate and incompetent, but, in turn, expensive: Bibliometric analysis of MI SASA for 2016
Centre for Evaluation in Education and Science, Belgrade
Abstract: In this paper, the Bibliometric Analysis of MI SASA (BA MI SANU) for 2016 is critically examined. It describes the previous history of conducting this kind of analyses in Serbia and the circumstances under which the analysis for 2016 was entrusted to the present contractor. Furthermore, the applied methodology, the obtained results, and their main consequences for the assessed journals are evaluated.
The circumstances are described on the basis of the CEON/CEES documentation and the available documentation of the ministry responsible for science in the Serbian Government. As the description of the methodology used in the analysis has not been published, it has been reconstructed based on the results and critically reviewed. The results are checked by cross-tabulating the findings of the BA MI SANU internally and by comparing them to the results of the Journal Bibliometric Report (JBR) produced by CEON/CEES for the same year.
The report is rated extremely unfavourably. BA MI SANU is not usable for the purposes it is intended for. For such a judgment, extensive evidence and argumentation are offered. The explanation for the failure is found in the crude simplification of the previously used (and still effective in JBR) journal evaluation methodology verified by the Ministry. The oversimplification is evidently aimed at minimizing the production costs at the expense of the overall quality of the analysis and the legitimacy of its results.
It is demonstrated that the consequences of the applied methodology for journal categorization are numerous and serious. Especially affected are the journals assigned to category M24 – National journal of international importance. Many journals are not awarded this category, although they fully meet the requirements of the Ministry's rulebook regulating this matter. Even more journals are awarded this category unjustifiably, since they do not meet the basic requirements under the Act on Journal Editing of the Ministry, or the elementary standards that journals of an international character are required to meet. The overall quality of some of these journals is totally incompatible with the critical article of the Rulebook, making M24 equivalent to prominent international (WoS) journals. As the number of journals classified into various categories is determined by the Rulebook, the mistakes in awarding M24 category produced a domino effect, inevitably and automatically resulting in an unfounded categorization of other journals. In this way, the BA MI SASA is compromised in its entirety, both in the expert-methodological and in the formal-legal level.
Keeping in mind the unjustifiably high price paid for the service to the contractor by the Ministry, the issue of responsibility for investing in such a major professional and scientific failure is raised. The available evidence indicates that the responsibility for this failure on the part of the Ministry, as its procuring entity, must be attributed to the responsible Assistant Minister as the true decision-maker in selecting the contractor and the model for implementing the analysis.