The effect of research evaluation excercises on research output: fifteen years of evidence from Italy

Abstract

In this paper, we investigate the potential benefits of implementing performance- based university research funding systems (PRFSs) in terms of both the quantity and «quality» (i.e. citation impact) of research outputs (measured through the Web of Science database). The Italian experience of the first two Research Evaluation Exercises is examined. We use a difference-in-differences in reverse (DDR) strategy, in which a country where a PRFS was always in place during the period of analysis – the «always treated» country, the UK – is compared with a country that switched from not having to implementing a PRFS – the «switched» country, Italy. Our analysis highlights that PRFSs are associated with an improvement in the average «quality» of research output (percentage of documents cited in the Web of Science database), although they do not positively impact output quantity (number of articles), which actually decreases, or «excellent» outputs (measured as the percentage of top-cited articles). An analysis of heterogeneous effects by scientific discipline shows that the effect on research «quality» concerns all fields except the humanities, with some positive effects on excellent research outputs in most hard sciences. Effects on research quality were not generalised but were observed only for the second research evaluation exercise. Other innovations introduced in the university system (the National Scientific Habilitation) may have partly contributed to this result.