This report builds on the working paper “Jonkers, K. & Zacharewicz, T., Performance based funding: a comparative assessment of their use and nature in EU Member States – a working paper”. This publication is a Science for Policy report by the Joint Research Centre, the European Commission’s in-house science service. It aims to provide evidence-based scientific support to the European policy-making process. The scientific output expressed does not imply a policy position of the European Commission. Neither the European Commission nor any person acting on behalf of the Commission is responsible for the use which might be made of this publication.

JRC Science Hub

https://ec.europa.eu/jrc

Abstract

Performance based funding is used by most EU Member States to increase the performance of their public research system. This report analyses the different nature of systems in EU Member States, selected associated and third countries. It aims to inform Member States which are in a process of mutual learning to improve the design of their allocation systems.

Title Research Performance Based Funding Systems: a Comparative Assessment

• Research Performance based funding systems provide incentives to increase scientific performance and concentrates resources in well performing organisations

• Novel quantitative data on project vs institutional funding by Member State

• The nature of systems in place differs widely

• The specific features of RPBF assessment designs can generate unintended consequences

• The assessment suggests RPBF as a potential avenue for several Member States

• The choice for specific designs, taking into accounts costs and potential benefits, should take into account the national context

JRC Mission

As the Commission’s in-house science service, the Joint Research Centre’s mission is to provide EU policies with independent, evidence-based scientific and technical support throughout the whole policy cycle.

Working in close cooperation with policy Directorates-General, the JRC addresses key societal challenges while stimulating innovation through developing new methods, tools and standards, and sharing its know-how with the Member States, the scientific community and international partners.

Serving society

Stimulating innovation

Supporting legislation

Europe Direct is a service to help you find answers to your questions about the European Union

Free phone number (*): 00 800 6 7 8 9 10 11

(*) Certain mobile telephone operators do not allow access to 00 800 numbers or these calls may be billed.

A great deal of additional information on the European Union is available on the Internet.

Our publications are available from EU Bookshop (http://bookshop.europa.eu), where you can place an order with the sales agent of your choice.

The Publications Office has a worldwide network of sales agents.

You can obtain their contact details by sending a fax to (352) 29 29-42758.

JRC101043

EUR 27837 EN

    PDF            ISBN 978-92-79-68715-0            ISSN 1831-9424            doi:10.2760/70120            KJ-1A-27-837-EN-N

    EPUB         ISBN 978-92-79-68837-9            ISSN 1831-9424            doi:10.2760/096620              KJ-1A-27-837-EN-E

© European Union, 2016

Reproduction is authorised provided the source is acknowledged.

How to cite: Jonkers, K. & Zacharewicz, T., Research Performance Based Funding Systems: a Comparative Assessment; EUR 27837 EN; doi:10.2760/70120

All images © European Union 2016

Research Performance Based Funding Systems: a Comparative Assessment

Acknowledgements

This report draws among other sources on the RIO Country Reports 2015 which were drafted by a network of experts following a template prepared by DG JRC J6. The country specific information has been reviewed and supplemented in close coordination with the country and statistical analysts of DG JRC J6: Robert Gampfer, Hristo Hristov, Jessica Mitchell, Giovanni La Placa, Milena Slavcheva, Maren Sprutacz, Blagoy Stamenov, Katarzyna Szkuta, Jana Zifciakova, Peter Fako and Athina Karvounaraki. The quantitative data on public funding allocation is derived from the results of the PREF study coordinated by Emanuela Reale. She and Benedetto Lepori (also part of the PREF consortium) have provided comments on the report at different stages. The report benefitted from further comments from Vera Calenbuhr and Gemma Derrick. The following experts provided input to the annexes and table 2: Susan Alexander, Liviu Andreescu, Vladimir Balaz, Pierre Bitard, Paul Cunningham, Tibor Dory, Ana Fernández Zubieta, Radu Cristian Gheorghiu, Christoph Grimpe, Kimmo Halme, Pim Den Hertog, Can Huang, Merle Jacob, Matthijs Janssen, Stijn Kelchtermans, Gundars Kulikovskis, Kzrysztow Klincewicz, Tom Martin, Manuel Mira Godinho, Leopoldo Nascia, Agne Paliokaite, Domagoj Račić, Ruuta Ruttas-Kuttim, Veli-Pekka Saarnivara, Klaus Schuch, Vitor Simoes, Hulda Skogland, Wolfgagng Sofka, Espen Solberg, Martin Srholec, Angelina Todorova, Lena Tsipouri, and Brian Warrington. Finally we would like to thank Xabier Goenaga Beldarrain for asking us to write this report.

Executive summary

Policy context

The introduction of performance based funding systems is one of the central mechanisms through which many EU Member States have tried to increase the effectiveness and performance of their Public Sector Research systems (ERA priority 1). In the EC communication on “Supporting Growth and Jobs – An Agenda for the Modernisation of Europe’s Higher Education Systems” the Commission recommends the introduction of funding mechanisms linked to performance which introduce an element of competition. The Council Conclusions of November 2011 also recommend the introduction of mechanisms linked to performance and competition to improve the governance of the higher education system. The initial aim of this report is to provide an input to a DG RTD managed Mutual Learning Exercise on performance based funding in the framework of the Horizon 2020 Policy Support Facility¹. To do so, it presents an analysis of the different approaches taken by EU Member States to use performance based allocation of organisational level public R&D funding. In addition this paper identifies a number of issues which Member States should take into account when evaluating/implementing these type of funding allocation mechanisms.

Main Findings

Most EU Member States have implemented Research Performance Based Funding systems. The rationales for the implementation of RPBF systems include 1) providing incentives for improving research performance, and 2) the concentration of resources in the best performing organisations. Research performance based funding systems differ widely in nature and in terms of the type of assessments they use. Many countries use a funding formula partially based on the quantitative assessment of research outputs. Another set of countries base their funding formulae instead on evaluations of research output through peer review. A subset of the latter uses quantitative assessments of research outputs to inform their peer review process.

The report contains a brief overview of the ongoing debate regarding the benefits and disadvantages of peer review and of different types of bibliometric assessment approaches. While RPBF aimed to increase the quality of research outputs, they have also the potential to generate perverse incentives. The costs involved in setting up different types of assessment, is also a factor to consider.

The available evidence on the effect of this funding mechanism is mixed. Some systems without a clear RPBF system perform very well, probably in part because they have an alternative way of concentrating resources in top performing organisations: e.g. the binary university systems in place in The Netherlands and Switzerland. All the EU Member States which did not experience a consistent improvement in impact scores over the decade studied, did not have a RPBF system in place. This may explain why the European Commission and the World Bank recommended these countries to implement RPBF. It is important to acknowledge that there are other factors that can explain the differential increase in performance. Some systems suffer from relative underfunding and/or the outbound mobility of their scientists.

RPBF systems vary considerably between Member States. Differences include the share of organisational level funding which is allocated through RPBF, the speed within which the system is introduced, the degree of stakeholder involvement, the impact different systems have on the autonomy of research performers, the criteria on which research outputs are assessed as well as the other missions and behaviours which the government wants to promote in these organisations.

In view of supporting mutual learning between the Member States this report presents an analysis of the different assessment and allocation systems in use in the Member States. The analysis and the annexes provide examples of relevant practices such as the way RPBF was gradually introduced in Denmark, the sophisticated bibliometric approach in use in Flanders and the successful, though relatively costly, peer review approach in place in the UK. Much can be learned from the experiences of countries which have had RPBF systems in place for a longer time and especially from the rationale for the changes in RPBF design which they have implemented over the course of the past decade. This is true both for the fine-tuning of policies which took place in e.g. Belgium and Denmark as well as the rationales for more radical system overhauls planned in Sweden and the Czech Republic.

Related and Future JRC work

It is the intention to build upon this report in future publications which will complement the qualitative analysis, with newly collected quantitative data for the EU member states. These future analyses will include assessments of the impact of differences in funding allocation systems on the performance of research systems.

1. Introduction

The introduction of performance based funding systems is one of the central mechanisms through which many EU member states have tried to increase the effectiveness and performance of their Public Sector Research systems (ERA priority 1). Already in 2010, OECD Member States indicated their interest to engage in mutual learning on this topic. Given its prominence among ERA priorities and the fact that different EU Member States have implemented RPBFs with different characteristics, it is an important potential topic for Mutual Learning among EU member states as well. The main aim of this report is to provide an input to an RTD PSF Mutual Learning Exercise on performance based funding. To do so, it presents an overview of the different approaches taken by EU member states to use performance based allocation of organisational level public R&D funding. In addition it aims to identify a number of issues which should be taken into account when implementing and evaluating these funding allocation mechanisms.

With direct public funding constituting around 35% of total R&D spending in European countries, public spending to research and development is widely considered as one of the main mechanisms for orienting science, technology and innovation policies (OECD, 2002; Cruz Castro et al, 2011). For more than thirty years, EU Members States have been seeking a consistent policy strategy to ensure an efficient allocation of public funding (Lepori, 2007a; 2007b). The focus on efficiency is central to the new public management approaches with its stress on monitoring and evaluation of policies and funding, which has influenced policy makers in many member states. Austerity induced public budget restrictions have further increased the emphasis on efficiency also in the public research system. A widely held assumption is that enhancing competitive allocation mechanisms can lead to improvements in research performance by facilitating a more efficient use of the funding resources, by selecting the best research groups, promoting cooperation and competition among them, promoting research themes and supporting structural changes in the modes of knowledge production with the aim to increase societal impact (Geuna, 2001; Braun, 2003). This approach received support from the European Commission, which invites MS to “introduce or enhance competitive funding through calls for proposals and organisational assessments as the main modes of allocating public funds to research and innovation, introducing legislative reforms if necessary” (EC, 2012). Yet, national policy frameworks are still fragmented and the empirical evidence supporting competitive allocation mechanisms as a tool to enhance an efficient allocation of funding is mixed (Abramo, Cicero, & D’Angelo, 2012; Auranen & Nieminen, 2010; Sandström, Heyman, and Van den Besselaar, 2014; Geuna and Pozziolato, 2015, Sivertsen & Schneider, 2012 and Larsen & Ingwesen, 2014).

The efficiency and performance effects generated by competitive funding are not homogenous between countries due to the differences in the quality levels of public research systems and of national research performers. Countries with different administrative traditions also tend to implement competitive funding instruments in different ways (Bleiklie & Michelsen, 2015). Universities are the prime target for performance based funding systems in most countries. In some systems research performance based funding allocation extends to other important research performing actors, such as Public Research Centres.² Where this is the case this will be flagged in this report. Another important distinction between higher education systems is whether countries have binary systems of research universities and universities of applied science, such as the Netherlands, Germany, Belgium, Norway, Denmark, Finland and Ireland or systems in which all universities are in theory treated alike as is the case in most large European research systems except from Germany (Jongbloed et al., 2015; Kyvik, 2004).

Performance based funding mechanisms are among the central ways to include competitive elements in the allocation of organisational level funding. In the EC communication on “Supporting Growth and Jobs – An Agenda for the Modernisation of Europe’s Higher Education Systems” (EC, 2011) the Commission recommends the increase of “funding mechanisms linked to performance” which “introduce an element of competition”. The Council Conclusions of November 2011 recommend the introduction of mechanisms linked to performance and competition to improve the governance of the higher education system (EU Council, 2011). As a further indication of the importance EU Commission R&D policy makers attach to the introduction of Performance Based Funding it is illustrative to read the very specific Country Specific Recommendations which the European Commission gave to the Czech Republic in the European Semester Reports:

(2011) “Establish a transparent system of quality evaluation of academic institutions and link it to its funding in order to improve the performance of tertiary education”

(2012) “Adopt the necessary legislation to establish a transparent and clearly defined system for quality evaluation of higher education and research institutions. Ensure that the funding is sustainable and linked to the outcome of the quality assessment”

(2013) “Adopt measures to enhance accreditation and funding of higher education. Increase the share of performance- based funding of research institutions”

(2014) “Accelerate the development and introduction of a new methodology for evaluating research and allocating funding in view of increasing the share of performance-based funding of research institutions.”

While many EU member states have adopted performance based funding mechanisms albeit to different extents, the nature and methodologies adopted by MS differ widely. It is the aim of this report to provide a comparative overview and analysis of the different types of research performance based funding mechanisms in the EU Member States. This is accompanied by a reflection on the advantages and drawbacks of the different methodological approaches to performance based funding.

2. Brief review of the literature

One of the main rationales for introducing performance based funding is to increase the accountability of universities over their spending of public R&D funding (Frølich, 2008; Hicks, 2012). Among the main motivations for moving from a block funding system towards the implementation of performance based funding mechanisms is to give governments, as a “principal”, greater leverage over their “agents”: research performing organisations in their public research systems. Research performance based funding can be used to stimulate research organisations to increase the volume or quality of their output, to prioritise certain fields of research, develop greater interaction with industry and other ways to increase their socio-economic impact, internationalisation etc (EC, 2010; Geuna & Martin, 2003; Hicks, 2012; Marginson, 1997; Tapper and Salter, 2003). Competition for resources between research performing organisations, in itself, is furthermore believed to increase efficiency and performance. Not only is it thought to stimulate organisations to perform better on specific tasks of their missions, but also from a systemic perspective, it allows for the allocation of resources to the actors that make most productive use of them. In doing so, the efficiency and performance of the system as a whole may improve (see also Herbst, 2007; Hicks 2012; OECD, 2011; Moed, 2008).

Even in cases where the share of funding involved is small, organisations can be sensitive to the gains/loss of reputation to which their degree of success in research performance based assessments gives rise. The susceptibility of universities to factors that influence their relative degree of prestige has been observed over the past decades in relation to the importance given by university administrators to the performance of their organisations in the various university rankings (Hazelkorn, 2010). Among the underlying reasons why universities are motivated by these effects, is that their reputation has an influence on their future access to resources (attractiveness to students, staff, collaborators, the acquisition of project funding etc) and the influence they can project in the scientific and societal realm. The reputational effect can even have a stronger impact on the strategic behaviour of universities than the effects of changes in the funding allocation (Hicks, 2012). An assessment of the weight of this effect in EU Member States requires further empirical analysis.

Analyses of the impact of RPBF systems indicate that the incentive system should be carefully thought through as the effect may not be the one desired. Butler (2003) for example found how an earlier iteration of the Australian performance based funding system which stressed the volume of publications had led to an increase in the number of Web of Science (WoS) indexed publications. However she observed that this came at the expense of the average impact of these papers. Sivertsen & Schneider (2012) and Larsen & Ingwesen (2014) argued that the design of the Norwegian and Danish system have had both a positive impact on the output and a positive or neutral influence on the impact of the Norwegian and Danish system respectively. Most of the countries which have implemented a RPBF in the past decades include incentives which promote an increase in output and an increase in impact, either directly or by promoting the publication of articles in higher impact journals.

The incentives which research performance based funding places on public research performing organisations are frequently “translated” by these organisations to incentives at the level of departments, research groups and the individual researchers that make up these professional work organisations (Whitley, 2000). Incentives for behaviour which results in an improved outcome in performance based evaluations, such as the publications of high impact publications or successful collaboration with industry, can include promotion opportunities, recruitment, salary increases, bonuses or access to resources to do research. For example, some Danish universities provide direct financial incentives to researchers who engage in high impact publishing (Andersen and Pallesen, 2008) and UK universities are known to recruit staff which can improve their evaluation output prior to the research evaluation exercises (Cruz-Castro et al, 2011).

Over the years, there have also been detractors of RPBF mechanisms arguing for example that due to their frequently imperfect design and implementation they create perverse incentives and end up stimulating “gaming” behaviour such as salami style publishing strategies³ as well other undesirable behaviour (scientific fraud, self-plagiarism etc) (Hazelkorn, 2010; Talib and Stelee, 2000). The prioritisation of certain fields or disciplines by policy makers also means that others get a smaller share of the “public funding pie”. By design not all universities and/or research organisations can compete equally on the basis of the performance based measures favoured. Both dynamics therefore often raise a degree of institutional resistance from actors/groups who lose out in this redistribution – or from actors who feel forced to change their behaviour to conform to the indicators on which their behaviour will be assessed (e.g. Good et al, 2015; Sorlin, 2007; Tsipouri, 2015). Other potential adverse effects of performance based funding regimes include a negative impact on staff morale and a perceived inducement of gender bias (against female researchers) in staff selection (McNay, 1997 and Baty, 2004 in Geuna and Piolatto, 2016).

Depending on its implementation, RPBF and the increase in competitive funding in general, is also sometimes considered to increase short-termism by reducing the certainties required by organisations and individual researchers to engage in long term, risky research and investment in research infrastructures. Others argue that performance based funding approaches risk disadvantaging non-orthodox or multi-disciplinary research in the face of dominant mainstream approaches – an argument which could be made for both peer review and bibliometrics based approaches (see also Hicks 2012; Rafols et al, 2012; Wilsdon et al, 2016). Unless countervailing incentives are in place, performance based funding systems may also reduce university efforts in applied research or e.g. in social science and humanities research to grapple with local or regional issues in order to focus on topics that can be more easily published in international journals (Hazelkorn, 2010). Finally there are the not unsubstantial costs involved in setting up and running the assessments on which performance based approaches to the allocation of organisational funding are based. Several experts and analysts raise the question whether these costs yield sufficient benefits (e.g. Boer et al, 2015; Hicks, 2012; Martin 2011; OECD 2010;), though Geuna and Piolatto (2015) indicate that at least in the short term the benefits outweight the costs in the case of the UK and Italy

3. Methods, definitions and sources:

This section defines how performance based funding is understood in this report by building upon the definitions developed by the OECD NESTI for “project funding” and “institutional funding” as well as on Hick’s (2012) work on performance based funding. While we base ourselves largely on the van Steen (2012) definitions, which are also employed by Eurostat (see next paragraph), we make an adaptation in the label of organisational level funding which is generally referred to as institutional funding. The reason for doing so is that “institutions” refer to “rules, norms, habits and ways of doing things” as distinct from organisations (See also Edquist, 1997; North, 1990). We therefore refer to organisational level instead of institutional funding⁴. In order to analyse the relationship between RPBF systems and university autonomy we consider the strategic actor-hood of universities and make a distinction between “internal autonomy” and “external autonomy” (Whitley, 2007; Cruz-Castro, Jonkers, & Sanz Menendez, 2016;). The analysis of organisational actorhood is more compatible with the understanding of universities as organisations rather than institutions.

Public research funding is generally allocated in two main ways, through project funding and through organisational level funding. Project funding is defined as “the total of national budgets in a given country, attributed to a group or an individual to perform an R&D activity limited in scope, budget and time, normally on the basis of the submission of a project proposal describing the research activities to be done.” (Van Steen, 2012). Since the 1980s the importance of project funding has increased in many European Member States. The other traditional way of providing public support to research is through organisational level funding of universities and public research organisations. Organisational level funding is defined as “the total of national budgets in a given country, attributed to a research performing organisation (university or PRO), with no direct selection of R&D project or programmes and for which money the organisation has more or less freedom to define the research activities to be performed.” (Van Steen, 2012). Organisational level funding can be allocated in the form of non-competitive Block funding. To a large extent this block funding may be earmarked for particular expenditures such as infrastructure or researcher’s salaries, especially in research systems where permanent researchers are civil servants. The university may, however, also have its own discretion in allocating a non-earmarked part of this block funding to research activities. Organisational funding may also be allocated in a variable/competitive manner tied e.g. to ex post assessments of the output and performance of universities (Cruz Castro et al, 2010). It is this latter type of competitively allocated organisational level funding which we consider as performance based funding. As indicated in the introduction, many EU member states have implemented some form of performance based funding over the past decade(s) and the share of organisational level funding which is allocated competitively on the basis of performance assessments has increased in many MS.

Analysts and policy makers have used different understandings of Performance Based Funding. It is therefore useful to further define what we mean in this paper by this funding allocation system by building on the definition developed by Diana Hicks (2012). Doing so will allow us to assess whether Member States have implemented a research performance based funding system for the allocation of organisational research funding. In order to be considered as RPBF, Member States funding allocation systems must have the following characteristics:

“- Research must be assessed. Evaluations are a necessary but not a sufficient criterion to qualify as an RPBF system. Evaluations of the output of degree programmes alone will not be considered as RPBF. This excludes many of the RPBF systems in the US states as well as a number of EU member states funding systems. These will be touched upon but not considered as RPBF.

- Research evaluation must be ex post. Evaluations of project proposals or the evaluations of organisational proposals for excellence initiatives are ex ante evaluations and are therefore not considered as RPBF.

- Research output and/or impact must be evaluated. Research systems which distribute funding on the basis of PhD numbers alone are not considered to be performance based by Hicks (2012). In this report they will be considered as such since PhD theses/Defences are considered research outputs.

- Part of the governmental allocation of university research funding must depend on the outcome of the evaluation. Ex post evaluations which are solely intended to provide feedback to universities or governmental actors will not be considered Performance Based Funding Systems.

- The performance based funding system must be a national or regional system: intra-organisational funding allocation rules are not considered RPBF, (Adapted from Hicks, 2012).

In this report, countries will be classified according to the type of Performance Based Funding System they have in place, distinguishing between:

1) countries who have no performance based elements in their university funding allocation system and countries which allocate funding solely on the basis of education related metrics or assessments (without research output considerations),

2) countries which based their funding allocation formula on quantitative metrics highlighting those who use different types of bibliometric approaches and

3) countries which allocate funding on the basis of peer review based assessment exercises. The latter category can be separated into metrics based peer review and “pure peer review”.

Formulae are generally used in the allocation of organizational funding and can be applied to the total amount of public funding transferred to the organizations or only to a part of it. An alternative approach is the signing of performance contracts between universities and ministries in order to agree ex ante on a set of targets that the universities need to achieve in order to be eligible for part of the organisational level funding. Considering its close link to RPBF, some discussion of such performance contracts will be provided in this report. Figure 1 below gives a graphical representation of this definition of performance based funding.

Figure 1. Research Performance-Based Funding Systems

This report includes an analysis of trend data of the research output in the systems concerned. Performance of a country on the basis of output and impact measures is influenced by other factors than the absolute public investments in publicly performed R&D such as the scientific human capital base, the stock of tacit scientific knowledge, the institutional set up of the research system and their pre-existing scientific research infrastructure. What is important in assessing the effect of the introduction of an RPBF on output and impact measures is to take into account the evolution of public R&D investments in publicly performed R&D in the period under study. A large increase in output (and impact) may also be partially due to an expansion of the system associated with a substantial funding increase over a sustained period, rather than being the effect of institutional changes such as the introduction of an RPBF. Likewise relative drops in output and impact or a failure to improve on these indicators may be related to sudden funding shocks or sustained underfunding of the public R&D system. An analysis of the impact of different funding systems is planned for a future paper. In order to assess the relative performance of EU Member States, we present the evolution in the number of publications and the share of (field weighted) 10% highly cited publications which is a measure of the extent to which research systems succeed in producing high impact scientific results. Considering differences in institutional set up and public R&D funding levels there are limits to the extent that meaningful comparisons can be made between North Western European Countries, Southern European and Central and Eastern European Member States. The institutional and organisational structure, including e.g. the funding system, was probably relatively similar in the EU13 countries in the early 1990s. Since then they have evolved in different directions and it may therefore no longer be possible to refer to a single type of EU13 funding model even if they do still share similar feature such as the important role of the Academies of Sciences (Radosevic & Lepori, 2009). Nonetheless comparisons between e.g. the EU 13 Member States may be meaningful to assess potential differences in output and impact which may be partially related to differences in funding models. Comparisons between some countries, such as Czech Republic, Poland, Hungary and Slovakia may be especially useful, considering similarities in their public R&D intensity and institutional structures. Likewise comparisons between North Western European Countries or between South European countries can be the most appropriate.

Sources:

This report is based on a comparative qualitative analysis of recent trends in and current use of performance based funding instruments in the EU Member States. An extensive knowledge of 35 national R&I systems is required which may be too much for an individual analyst to possess. This analysis is therefore based on two types of information. The first are the RIO Country Reports 2014/2015 which have been drafted by JRC J.6 and a network of national experts in national R&I systems. These reports have been reviewed by National Contact Points appointed by ERAC. In order to cross check and supplement these sources we have also built on collected national policy documents and the results of analyses and assessments published in the academic literature and further contacts with research policy experts in several of the countries under consideration. The resulting annexes and the classification in table 2 have been reviewed by the individual national experts. For the national level bibliometric data, use is made of a data collection that was carried out by ScienceMetrix for DG RTD (Campbell et al, 2013). For quantitative data on project vs organisational level funding use is made of data collected in the framework of the Public Funding of Research (PREF) project carried out by a consortium led by CNR CERIS on behalf of DG JRC of the European Commission. This project aims at providing a comprehensive analysis of public research funding with a focus on the allocation mechanisms adopted in different countries. This analysis is broadly consistent with EUROSTAT statistics on Global Budgetary Allocations for R&D (GBARD), but provides a more fine-grained disaggregation by mode of allocation, managing organizations and funding flows to performers. Definitions and categories correspond to those adopted in the 2015 edition of the Frascati Manual. For a more detailed description the reader is referred to the PREF handbook.

4. Public funding allocation

This section provides a synthetic analysis of the performance based funding systems for competitively allocated organisational funding which are adopted in the different European Member states. It is based on brief national case studies which are described in annex 1 where the reader can find more detailed information on the systems in place in each of EU member states.

In order to gain an understanding of prevailing funding systems in the Member States, table one first provides an overview of the relative important of project versus organisational level funding. Project funding is by its nature competitive funding. However, it is based on ex ante evaluations of research proposals, rather than on ex post assessments of outcome. The table does not make a distinction between organisational block funding and performance based organisational level funding, but when countries implement Research Performance Based Funding Systems it implies that part of the organisational level funding is allocated in a competitive manner. The share of organisational level funding thus provides a certain indication of the potential for performance based funding in a system if such a change is to be budget neutral (ERAC Peer Review Spain, 2014).⁵

Table 1 is based on data collection through the PREF study. On the basis of the available data we can make a provisional distinction between a few different groups of Member States according to the relative importance of project versus organisational level funding. For the sake of comparison, available data from Eurostat are also presented. For most countries, the data only show a slight deviation with the PREF study. For a small set of countries (e.g. France, Portugal, Czech Republic), more significant discrepancies appear which need to be explored further in consultation with the PREF team, national authorities and/or Eurostat. A first group of countries is characterised by a high level of organisational funding. A second group of countries shows a greater balance between organisational level and project funding. The qualitative analysis of the evolution of funding mechanisms in the different member states presented in the next section will provide greater insight in the extent to which this is allocated competitively through RPBF. There is a third group of countries which is characterised by relatively high shares of project funding in comparison to block funding. It is relevant to know that some of the countries in this third group also make intensive use of performance based funding such as the UK (Cunningham 2015), thus increasing the share of “competitively allocated funding”. For a few remaining countries quantitative data is not yet available. The PREF project will in the future also provide data on the share of competitively allocated organisational level funding of which RPBF is an important subset (others include e.g. centres of excellence and performance contracts) From the ordering of the table it emerges that it is not easy to make a direct link between the relative balance between project and organisational funding and the level of development of the respective system.

Table 1. The share of project vs organisational level funding

Country | Year | Organisational | Project | Organisational | Project
Source | EUROSTAT | PREF
Malta | 2013 | - | - | 99% | 1%f
Italy | 2013 | - | - | 95% | 5%
France | 2014 | 93% | 7% | 79% | 21%e
Estonia | 2012 | - | - | 75% | 25%c
Austria | 2013 | 73% | 27% | 72% | 27%
Switzerland | 2014 | 72% | 28% | 72% | 28%b
Luxembourg | 2013 | 82% | 18% | 70% | 30%
The Netherlands | 2013 | 71% | 29% | 70% | 30%
Sweden | 2013 | - | - | 69% | 31%
Slovenia | 2013 | - | - | 69% | 31%
Lithuania | 2013 | - | - | 67% | 33%a
Germany | 2013 | 64% | 36% | 64% | 36%
Spain | 2013 | - | - | 56% | 16%d
Finland | 2013 | - | - | 56% | 44%
Bulgaria | 2013 | - | - | 54% | 46%a
Denmark | 2013 | - | - | 54% | 46%
Portugal | 2013 | 78% | 22% | 53% | 45%
Norway | 2013 | 54% | 46% | 51% | 49%
Greece | 2014 | 50% | 50% | 50% | 50%
United Kingdom | 2013 | - | - | 47% | 53%
Belgium | 2013 | - | - | 45% | 55%
Poland | 2013 | - | - | 40% | 60%
Ireland | 2013 | 32% | 68% | 32% | 68%
Latvia | 2013 | - | - | 25% | 75%f
Czech Republic | 2013 | 49% | 51% | 21% | 79%
Croatia | 2014 | - | - | 8% | 92%
Romania |  | - | - | - | -
Slovakia* |  | - | - | - | -
Cyprus |  | - | - | - | -
Hungary |  | - | - | - | -

Methodological notes

a) Deviation from GBARD still to be checked

b) Year 2014- R&D questionnaire only in pair years

c) Only 2012 available

d) Shares do not yet add up to 100% due to regional funding which can’t be classified as project or institutional funding

e) Only 2014 available

f) Excluding structural funds

Highlighted in orange: countries above 75% of organisational funding; Highlighted in pink: countries between 50% and 75% of organisational funding; Highlighted in green: countries below 50% of organisational funding; No highlight: NA. Source: ongoing PREF study.

4.1 Performance based funding models

As shown in table 2 many European countries have implemented some form of performance based funding. The exceptions are Bulgaria, Cyprus, Greece, Hungary, Ireland, Luxembourg, Latvia⁶, Malta, Romania, Slovenia and Spain⁷ (Alexander, 2015; Tsipouri, 2015; Warrington, 2015). In the cases of Malta, Cyprus and Luxembourg, this is probably a partial consequence of the relatively small size of the system (Malta and Luxembourg only count one single university). In other instances, i.e. Greece, the absence of performance based funding may be due to resistance from the academic community which has protested heavily against suggestions to introduce performance based evaluation and allocation models (Tsipouri, 2015). This group also includes countries that allocate funding on the basis of education/training based metrics (student inputs, number of graduates etc) and related metrics only.⁸ For example most US states⁹ allocate their organisational funding for universities on the basis of a formula which takes into account the output of the university in terms of the number of graduated students alongside other indicators. Most of these states do not consider research output performance based metrics. We refer to this category in this report to explain the differences in funding systems and to highlight the fact that many other approaches also include education based metrics in complement to the research output based metrics in their research funding allocation systems. Another traditional metric is to base funding allocation decisions on the number of research staff in the organisation. This criterion is not performance based and while most countries base part of their funding allocation on the basis of this indicator it is considered part of non-competitively allocated block funding rather than performance based funding – since some countries include this criterion in more complex funding models it will nonetheless be mentioned in some of the sections below.

Countries like Spain, Slovenia, Hungary and Germany have pursued some of the goals of performance based funding, such as international excellence, by awarding centres or units of excellence to universities based on the assessment of proposals (CHEPS Germany 2006 in Hicks, 2012; MINECO, 2015; Dory, 2015; Fernandez Zubieta, 2015). Since these initiatives are based on voluntary ex ante selections of proposals they are not considered as performance based funding systems in this report. Many of the countries that did implement Performance Based Funding have also engaged in the set up of centres of excellence type funding programmes, including Norway with its Centres of Excellence, Poland (KNOW), Finland (Centres of Excellence in Research), France (IDEX), Denmark, Sweden¹⁰ and Austria (Institute of Science and Technology) (Jacob, 2015; Klincewicz, 2015; Bitard, 2015; Cuntz, 2015; Saarnivaara, 2015).

Table 2 provides a classification of the organisational level funding allocation systems in place in the Member States according to the model described in the methodological section. A distinction is made between three different systems: those that are considered to have no research performance based system in place, those who have implemented a limited RPBF and those which are considered to have implemented a performance based system. In the latter case a distinction is made between those systems which rely primarily on the peer review of research units/organisations and those systems in which the assessment of research performance is primarily based on quantitative assessments of research output. As indicated previously the funding formulae used to allocate organisational level research funding, in addition to assessment of research also tend to take into account a number of other variables, including for example education or diversity related metrics. An overview of the main criteria on which such assessments are based is provided below.

The first criteria which many systems use to allocated not only education but also research funding are “education metrics”.¹¹ These metrics can involve both input (students enrolled) and output (BSc and MSc graduates). A related metric refer to PhD degrees awarded. The latter is considered a research output measure as a PhD thesis is considered a substantive research output. “Historical” refers to the practice in many systems to base the allocation of organisational level funding (partially) on the funding allocation distribution that was used in preceding years.

Bibliometric indicators refer to indicators for scientific publications in academic journals and monographs from leading publishers. In this table, bibliometric indicators are separated into three categories. The first refers to counts of publications. Often the mere counting of publications contained in bibliometric databases is considered insufficient and a measure of scientific impact is taken into account as well. The first approach to tackle this is to weight the publications on the basis of the impact of the journal in which it was published. This may be based on various forms of the Journal Impact Factor. Another approach is to construct custom made lists in which journals are classified by national expert commissions according to quality profiles. Apart from journal weighting, another indicator for the impact of publications is to engage in publication level citation analysis – which, as in the case of journal impact measures, tends to involve field normalisation.

Apart from the number of PhDs awarded, other (input or output) indicators frequently used in research performance assessments are classified here as “other formula elements”. They include the number of patents, which may be restricted to e.g. PCT and EPO patents or may also involve patent filings at national patent offices. One indicator which could be considered an input rather than an output metric is the amount of funding which is generated by participation in national or international research projects. External funding generated by contract research for companies or public administrations, income from Knowledge Transfer activities and spin off companies generated is another variable that is considered in many systems. Two other indicators included in table 2 are diversity measures, which refer e.g. to the gender composition of staff and internationalisation indicators e.g. referring to the degree of international experience of staff, the attraction of foreign PhD researchers and/or the engagement in other international activities. Since participation in international projects was already considered in the preceding category it is not counted here.

In the systems which are considered to be based on a peer review assessment variables such as the impact of the research output, number of PhDs, patenting behaviour etc can also be considered. Since the dominant assessment mode is qualitative we group them together under this heading however.

Several systems are experimenting with including assessments of societal impact in their funding allocation formulae (e.g. UK, France and Australia). This is not captured in the table but considering that it is one of the major novel developments in performance based funding we do discuss it in the text and the accompanying annexes.

Apart from the indicator based or peer review assessments, some information is also provided on the systems which engage in the signing of performance contracts. The latter can involve a number of issues on which research performers agree with their principal (ministry), e.g. research or education reform to attain critical mass or quality thresholds, objectives set in the strategic plan, the extent to which university plans contribute to other policy objectives etc. Not included in the table are a number of criteria which are used in only some systems, including e.g. community outreach and the quality of research staff.

The (limited or large) number of indicators considered should not necessarily be taken as an indicator of the level of sophistication of the system. For example the UK REF system is one of the oldest and most developed RPBF systems, however in the table it is not considered to include many indicators in its funding formula.

Table 2. Dominant assessment approach for the allocation of Performance Based funding in the EU28

¹ Bulgaria did have a RPBF system in its education law for several years, but this appears not to have been implemented. In the new law however, a RPBF system has been introduced again (see Annex).

² Hungary The introduction of a RPBF system in Hungary is foreseen (see annex)

³ Luxembourg has only a single university, it was not considered to have a RPBF system to allocate organisational level funding between universities. However its performance contract has many of the trappings of RPBF systems.

⁴ Latvia has used assessments to change the allocation of its funding of PROs. For universities it has recently implemented a RPBF system, this is not yet reflected in this overview table but annex 1 provides more detail.

⁵ In Romania some steps have been taken to design an institutional funding system linked to performance assessment, in practice this is not yet implemented. (see also country annex).

⁶ In Germany the organisational level funding of universities is mainly provided at the regional (Laender) level. These have increasingly included quantitative assessment criteria and/or performance contracts, but the nature of their funding allocation procedures differ from state to state (Van Daalen, 2014)

⁷ Belgium (Wallonia): for the Special Research Funds, the education-metrics based system will still be partially linked to the historical distribution among universities during a transition period (2014-2016). For the concerted research actions, there is also a transition period with performance criteria being gradually introduced into the funding allocation formula from 2016 onwards.

⁸ In 2009, Czech Republic introduced a research performance-based methodology to allocate research funding. This methodology, based exclusively on quantitative indicators, was modified for the period 2013-2015 and introduced some peer review assessment elements.. After 2016, bibliometric informed peer review should become the main mode of assessment (Shrolec &Szkuta, forthcoming)

⁹Estonia does have performance contracts for universities for which universities are assessed on a number of education metrics. This does not concern the allocation of research funding.

¹⁰Poland used to base its funding formula partially on historical considerations. Since 2015 however, this variable has been removed (Klincewicz, forthcoming). See annex for more detail.

In addition to the countries without any RPBF system, three categories have been identified. The first one is composed of Austria, Germany and the Netherlands. Hicks (2012) did not include the Netherlands and Austria among the countries which have implemented RPBF systems because they do not base their funding allocations on assessments of research outputs. However since successfully defended PhD theses could be considered a research output, this report includes the countries that include this indicator as an element in their funding allocation systems as the first category of research performance based systems. As will be highlighted in the results and discussion section, the incentives which these systems generate are expected to be different from the systems that include publication output assessments.

The second RPBF category bases organisational funding allocation mechanisms on assessments of research outputs as assessed through quantitative bibliometrics. This group comprises of Belgium, Czech Republic, Denmark, Estonia, Finland, Croatia, Poland, Sweden and Slovakia. The third group, consisting of the France, Italy, Lithuania, Portugal and UK takes a different approach. These countries have opted for a funding allocation system which is based on peer assessment of universities and/or university groups: in most cases these peer review assessment are heavily based on bibliometrics and other metrics and thus approach the second category in nature. Czech Republic had introduced in 2009 a research assessment system exclusively based on quantitative bibliometrics. This system is currently evolving towards a bibliometric informed peer review that should be fully implemented after 2016 (Srholec & Szkuta, forthcoming). There are also exceptions such as the UK in which bibliometrics play a lesser role. There are strong variations in the nature of the performance based funding systems between these countries: especially the United Kingdom stands out as a country with a long tradition of research performance based funding allocation on the basis of an elaborate peer review system. The nature of the funding formulae and the criteria used differ considerably between these countries. Governments are still changing and fine-tuning their approaches to formula based funding as will be described in the analysis that follows. Table 3 provides an overview of the timing at which RPBF systems were implemented and the agencies responsible (adapted from Hicks, 2012).

Table 3 Overview of Performance Based Funding Systems in several EU member states

EU MS | Year implementation/revision RPBF | Agency Responsible
Austria | 2002; 2013 | BMWFW
Belgium (FL) | 2003; 2006, 2012; 2014 | EWI (BOF Key)
Belgium (Wa) |  | ARC/FSR
Croatia | 2013 | MSES
Czech Republic | 2013 (potential new change in 2015/2016) | CRDI
Denmark | 2009 | Ministry of Education
Estonia | 2012 | Ministry of Education; Estonian Research Council
Finland | 1998/2010 / 2014 | Ministry of Education
France | 2008 / 2013 | HCERES
Italy | 2010, 2014 | ANVUR (VQR)
Lithuania | 2009, 2011 and 2012. | Minister of Education and Research
The Netherlands | 2012 | MoESC
Poland | 2008/2013 | KEJN
Portugal | 2015 | FCT
Slovakia | 2013 | MESRS
Sweden | 2009 | Swedish Research Council
UK | 1986 (RAE), 2014(REF) | HEFCE (REF)
Germany | Various | Lander

4.1.1 Funding allocation formula based on quantitative indicators

4.1.1.1 Education and PhD awards based formula

The Netherlands bases the allocation of part of its organisational funding to universities on a set of education and research related indicators. The education part mainly revolves around student and graduation numbers. The research part however also includes a component on PhD defences. A remaining substantial share of the research part is allocated on the basis of historical/political considerations (Janssen and Den Hertog, 2015). Austria also takes into account a number of education and research related metrics. As in the case of the Netherlands, publication based output assessments/indicators are not included in the funding allocation model. However, as in the case of some other systems, input based metrics such as the volume of project funding are included in the formulae (Cuntz, 2015). In Germany, in addition to the Excellenz Initiativ described earlier, many of the states (Länder) have implemented a form of performance based funding on the basis of both teaching and research performance. The weighting differs by type of university: for the universities of applied science (hochschule) teaching indicators carry a greater weight than for research universities. The indicators used differ by state, with all Länder using measures of third party funding and number of completed doctorates, and individual Länder incorporating further indicators as deemed appropriate. At least in 2010, none of the Länder studied included assessments of publication output (Gläser et al, 2002; OECD, 2010; Van Dalen et al, 2014). While in contrast to Hicks (2012) we include the Netherlands, Austria and Germany as performance based funding systems it is clear that they are border-line cases distinct from the formula based allocation models which are based on a (bibliometric or peer review based) assessment of publication output. The next paragraphs will mainly discuss the use of Performance Based Funding allocation which is (partially) based on assessments of the research output.

Bulgaria states in legal documents that it considers the R&D output in the formula based allocation of its university funding (see annex 1). However, some analysts (Todorova, 2015) argue that in practice Bulgaria does not have a performance based system as the relative funding levels of its universities have remained (completely) stable in the past years (see annex 1). Recently, analysts and international organisations argued that a performance based funding system (or alternative approaches to increase competitively allocated funding) needs to be implemented in Bulgaria to improve the quality and relevance of its research output (Todoreva, 2015; Worldbank, 2013; Soete et al, 2015). Latvia has used some performance based elements in the restructuring of its system of Public Research Organisations. For universities, however, no RPBF system is in place, though it is being considered for the near future (see annex 1). Bulgaria and Latvia are therefore grouped in table 2 among the countries which at present have not yet implemented a Research Performance Based System.

4.1.1.2 RPBF based on Journal based impact assessments

A number of countries (e.g. BE, CZ, DK, EE, FI, HR, PL, SE, SK), have introduced bibliometric based formulae. The simplest approach used in these countries consists of counting publications in international journals as was done in some assessment exercises in the past. However, this practice was shown to encourage publication in lower impact journals to boost output (Butler, 2003).

In the Norwegian system (see annex 1) (Sivertsen, 2009; Fosse Hansen 2009) journals are therefore ranked to weight publications of higher impact/quality. The Danish system drew inspiration from the Norwegian system (Fosse Hansen, 2009). It is journal based: publications are counted in around 20,000 peer reviewed journals and publishing houses. The journals and publishers have been selected by sixty eight panels of researchers. The journals and book publishers are divided into two groups, where publications in the top 20% receive a higher weight than publications in the other outlets (Fosse Hansen, 2011; Grimpe, 2015; DASTI, 2009, 2014). Australia came back from its own approach of using ranked journal lists in 2012 (see annex 1). Poland has a RPBF tradition that goes back into the 1990s. It underwent an evaluation of its RPBF system in 2008 and revised the evaluation process for its 2013 evaluation, for which it uses the journal impact factor (Klincewicz, 2015). Here the KEJN conducts detailed assessments on the basis of the following criteria: counts of publications taking into account impact factors of specific academic journals, patents, revenues from industry co-operation and external R&D funding normalized by numbers of R&D employees of an organization, scientific awards of researchers, patents, and financial results of commercialization of research results. In 2013, the evaluation criteria were substantially modified to further promote organizations conducting world-class research. Journal impact factor based assessments are also used in the Czech Republic, though the government has implemented measures to attempt to address the problems in assessing social science and humanity research bibliometrically (Malek et al., 2014). In Finland around 90% of organisational level university funding is allocated on the basis of completed qualifications and credits as well as scientific publications and success in attaining competitive project funding (Saarnivaara, 2015)..¹²

In Slovakia a considerable share of the organisational level research funding allocation is based on evaluation carried out by the Accreditation commission and the share of a HEI in the national research output (see annex). This Commission approved new, more stringent, rules on 15 April 2013. The new rules put more emphasis on (simple) bibliometric assessments of research excellence in terms of scientific papers published in journals listed in international bibliometric databases (WOS, SCOPUS), research monographs published with high-quality publishers and international patents. The last evaluation took place in 2009-2010 (Balaz, 2015). In the Czech Republic initial reform plans were to allocate almost the full amount of organisational level funding using performance-based formulae. However, a medium term modification of the evaluation methodology, called Metodika 2013, has been introduced for the period 2013-2015, (CRDI, 2013). On the base of this methodology organisational level funding is going to be allocated until 2016 (Srholec, 2015). The first part of this assessment is based on an assessment of research output of publications in indexed journals.¹³ A second part of the evaluation is based on high impact results/inputs, including ERC grants. Each organisation selects a number of high quality results in each scientific field under consideration. These results are peer reviewed and compared for every organisation. A third part of the evaluation concerns non-scientific outputs of R&D, including patents and plant varieties. The remaining points in this part of the assessment exercise are allocated on the basis of an assessment of income generated through applied projects and collaboration with industry. The whole assessment relies on panels of experts to make the assessment for each field. The funding allocation decisions are based for 75% of the assessment of the scientific output, for 10% on the evaluation of high impact research outputs and for 15% on the assessment of applied research (Malek, 2014). According to Good et al (2015) the result is a system that is primarily quantitative in nature.

4.1.1.3 RPBF based on citation based impact assessments

Whereas the previous countries based their bibliometric assessment of counts of publications, potentially weighted on the basis of the quality of journals, Belgium and Sweden have explicitly introduced a combination of output and publication level citation based impact metrics to take into account both the volume as well as the actual impact of the research output. In Belgium, (Flanders) the share of the competitively allocated organisational research funding (the BOF- Key) which is based on bibliometric assessments of the research output has increased from 10% when it was first introduced in 2003 (DeBackere and Glanzel, 2004) to 36% in 2008. In the latest revision in 2012, the weight of the social sciences and humanities have increased on the basis of a specially developed publication database made for this purpose (DeBackere and Veugelers 2015). In addition to citation based impact assessments Flanders also considers a journal based indicators in its formulae. Sweden has implemented a sophisticated field weighted impact measurement system to assess university output (Hicks, 2012; Jacob, 2015; Sivertsen, 2015).

As discussed in more detail in the country based Annex 1 the formulae used in bibliometrics based funding allocation tend to take into account many other factors than publication and citation metrics alone. These can include employment of graduates; external research funding; faculty characteristics and qualifications; faculty size; student enrolment, graduate students graduated; PhD defences, impact of research; participation in international research projects, collaboration with industry, private sector funding etc (elaborated on Hicks, 2012). In many systems (see e.g. Denmark, Poland and Czech Republic) also the variable part of organisational research funding retains an element that is based on previous funding decisions (historical factors), which are used in part to avoid too large swings in research funding that could cause instability (Hicks, 2012; Srholec, 2015). In Poland the weight of this historical element has been decreasing since 2011-2012 and was removed in 2015. Nowadays only the outcomes of the most recent evaluation are taken into account.

4.1.2 RPBF based on peer review assessments

The UK Research Assessment Exercises which started in the 1992, were among the first central research assessment approaches used for performance based funding allocation decisions. Apart from the UK a number of other countries including PT, FR, IT and LT use peer review based assessments of research outputs. In the UK, the HEFCE allocated performance based funding on the basis of a mechanism known as the Research Excellence Framework (formerly the Research Assessment Exercise – RAE), a peer review process which produces ‘quality profiles’ for each submission of research activity made by universities. Once funding levels for institutions have been set, these are used for the annual allocation of funding until the next round of assessment (Cunningham, 2015). After a series of extensive consultations and reviews, the Higher Education Funding Councils replaced the RAE with the new REF. For the REF, the government had initially proposed a metrics only exercise (Martin and Whitley, 2010). However after strong resistance from the academic community, and partially on the basis of considerations regarding the limitations of bibliometrics, the HEFCE has so far shied away from a full switch to a bibliometrics based assessment of its universities in its evaluation exercises. Disciplinary panels were given the possibility to use the results of centrally provided bibliometric analysis. They were not given clear top-down instructions on how to use these metrics – though they were cautioned against e.g. the use of impact factors as a quality proxy. Wilsdon et al. (2016) provided a review of the potential for responsible use of metrics in the REF arguing against a full switch. Currently Lord Stern’s commission is assessing how the new REF can be improved in 2018. The latest (2014) REF includes societal impact assessment on the basis of cases provided by the evaluated departments (Martin, 2011). The HCERES¹⁴ evaluates all French research teams and research Centres. The evaluations provided are based on a peer-review assessment which is based, though not exclusively on, bibliometric indicators. Research groups are requested to select a number of high quality outputs on which they will be evaluated. They have to argue as well how their research has had a societal impact, can be considered as frontier research or helps addressing societal challenges. Other factors that are taken into consideration include the integration of research teams in international networks.

In Italy, the ANVUR published its latest metrics informed peer review assessment in 2013. This is a large scale exercise in which all universities and PROs are evaluated: it is done at the level of universities rather than research units as in the UK (ResearchItaly, 2015). The share of organisational level funding allocated on the basis of performance criteria has risen from 7 % in 2009 to 13.5 % in 2013 and is set to increase further in the coming years. Currently, existing rules are said to limit the ability of the funder to fully allocate this funding across universities on the basis of performance (European Commission, 2015a). The ANVUR based its assessment on the best research outcomes obtained by each organisations (universities and research institutes) in the seven years from 2004 to 2010. The approximately 195,000 publications by 130 organisations are evaluated partly by submitting them to international experts who appraise their scientific quality, and partly by analysis of the citations received from third parties and an examination of the impact they have made in their respective field of research. Secondly it considers the ability of the evaluated organisation to attract funding, the number of international collaborations, patents registered, spinoffs, museums and archaeological sites, third-party activities etc.¹⁵

4.2 Taking into account disciplinary differences

Peer review systems tend to bring together field-based committees which have some discretion for developing appropriate standards of judgment for their field (e.g. PL, IT, CZ, UK, FR). These committees may, for example, decide on how to evaluate different types of publications, books or other forms of intellectual output: be it Open Access databases, films, patents, plant varieties etc which may be more important in some fields than in others. In France, for example, the HCERES committees take into account PhD supervision in some fields (Mathematics) but not in others. In Lithuania the weight given to the four categories differ between fields. For example, results of the evaluation of research production are given the highest weight in social sciences and humanities (80%) as well as physical and biomedical sciences (55%). Assessment of R&D activities in other fields of science mostly depends on institutions’ capacities to attract funding from privately and internationally funded R&D projects (Paliokaite, 2015). In the Czech Republic, peer review is used in assessing the publication output in the social sciences and humanities. In the natural science fields the assessments are mainly based on journal (impact factor) based bibliometrics (Malek, 2014). Field differences can be taken into account, either by applying field normalisations in publication level analyses (e.g. SE, BE), by taking into account field journal rankings in the journal weighted output assessments (e.g. BE, DK), or by developing complete custom made bibliometric databases for the Social Sciences and Humanities as was done in Belgium (Flanders – see annex).

4.3 Current developments in assessment models

The Czech Republic is one of a number of countries which is currently revising its performance based funding approach – a new design is being developed by Technopolis and will be considered for introduction in 2015/2016 (Good et al, 2015; Srholec, 2015). Another country in which the government is currently considering a change in the system is Sweden. Sweden currently has a system in place which is based on a sophisticated (field weighted publication based) bibliometric assessment (Hicks, 2012; Jacob, 2015; Kalpazidou & Schmidt, 2012). The previous government launched a study into the extent to which this could be replaced by a system in which peer review played a larger role (Jacob, 2015; Sivertsen 2015). As highlighted in previous sections and the annexes Hungary, Latvia and Bulgaria are also considering the introduction of a Performance Based Funding System or are implementing such systems at present (see annex 1)

One of the mechanisms through which RPBF systems aim to improve the efficiency of research systems is through the concentration of resources (Adams and Gurney, 2010). In addition to basing funding allocation decisions on ex post assessments of research outputs, a number of Member States have included variables in their funding formulae which are tied to the success of universities and/or PROs in the attraction of national or EU level research funding. These measures further strengthen the concentration of resources in top performing research organisations. However, they may also have unintended side effects especially for the social sciences and humanities. By including project funding as one of the variables under consideration, institutional funding becomes partially dependent on project funding. To the extent that SSH researchers are less able / prone to bid for project funding, this may affect the weight of these disciplines in universities. It may also stimulate researchers in these fields to bid for (larger) projects if funding is available.

4.4 Research Information Systems

Both Peer Review based assessment exercises and Formula based approaches require research organisations, research funders or evaluation agencies to systematically keep track of research outputs and a number of other data and metrics (staff numbers and characteristics, research project funding contracts, etc). In order to address this challenge, universities and national governments have been establishing Current Research Information Systems (CRIS). Norway was probably the first European country in which universities have set up a nation-wide Research Information System (FRIDA). FRIDA is used to supply information and reports to the ministry as well as internal information and knowledge management functions. In Finland, universities are obliged to upload the requested data in a centrally management information system KOTA, which is used to inform university assessments (De Boer et al, 2015). Also in Poland the assessment process which underpins the RPBF is managed by a central IT system (Pol-on) to eliminate the risks of human error or duplication of records for researchers working at more than one scientific organization (Klincewicz, 2015). In Czech Republic research organisations are also required to register their research outputs in a centralized R&D information system (RIV) (Good et al, 2015). Sweden is developing a nation-wide CRIS to support the new version of the assessment framework in case this is implemented (Sivertsen, 2015). In most other countries, CRIS are developed at the university level. Attempts are made to come to standardisation of the CRIS at a national and European level to facilitate compatibility. The development of such system and the various models that are being considered suggest that: a considerable investment in resources, time and expertise is required to come to well-functioning national CRIS. At least in some systems, it requires the voluntary buy-in of individual universities to share data with other universities and the governmental evaluators. There are also “real” privacy issues which need to be addressed before rolling out a national or European wide system (Sivertsen, 2015). Apart from potentially facilitating the development of improved versions of the current Performance based funding systems, CRIS may help increase the accessibility of information to university administrators and in doing so increase their potential for strategic action. They also offer potential for analyses and evaluations of university funding and performance beyond those that are required for the assessments on which funding allocations can be made in Performance Based Funding Systems.

4.5 Performance contracts and the steering of research

The use of performance based funding in public research systems is often considered to be a manifestation of new public management. Hicks (2012) argued that one feature of New Public Management was yet not common in performance based public funding systems, namely the use of performance contracts. A recent study form the EUA (2015) indicates, however, that 15 of 22 countries studied had implemented a form of performance contracts including: AT; BE (Wallonia); DE; DK; EE; ES; FI; FR; IS; IT; LV; NL; PT; SE. Not all these countries are included in table 2, because the definition used by the EUA of performance contracts means it is not necessarily tied to research performance. In the case of Austria and The Netherlands the performance contracts are tied to funding allocation: in the Netherlands 7 % of organisational funding is allocated to universities on the basis of performance contracts which include commitments in the profiling of teaching portfolio’s to achieve critical mass and avoid duplication (MinOC&W, 2012). Also in the other countries which implemented such contracts, they can have an influence on the strategies and directions in which universities develop.

Governments may also use Performance Based funding systems to steer their universities on other grounds than on output and excellence based criteria. In Norway, for example, government research funding is targeted at research within distinct priority areas (Sivertsen 2009, EC 2010). In other countries, including the Czech Republic and Poland, the RPBFs include indicators that gauge interaction with industry. In the UK REF exercise and in the French RPBF, universities are partially evaluated for their broader societal impact. Belgium (Flanders) and Sweden include gender diversity measures among the indicators in their funding formula.¹⁶

5 Trends in publication output

The UK quality related funding based on the RAE/REF aimed to improve the quality of the system by stimulating selectivity and concentration while ensuring moderation: not too large swings and especially drops in funding. Over the past two decades the rewards tied to the outcome of this exercise have increased. Initially units which performed well relative to the national average would receive additional funding. After 2002 it was only units which were considered nationally excellent. In the latest exercise only units that were considered world class or world leading received additional funding. As a consequence the RPBF funding is being concentrated in fewer universities. In the last exercise, 10 out of 130 (English) universities received 50% of the funding. The top four actors received 30%. 9 universities received no and 49 received only a small share of the RPBF funding (Boer et al, 2015). Boer et al argue that “The process has increased England’s overall publication levels by embedding a culture of research management across all HEIs, with every [“research active”]¹⁷ academic producing at least four peer reviewed articles per review period. It is thus fair to conclude that, judged on the basis of the scores and the absolute outputs, England’s research assessment approach has been highly effective in raising research performance across the sector as a whole¹⁸.” This chimes with the public debate in the British system and it is clear that the British performance both in terms of output and impact is impressive, also considering the relatively low and decreasing levels of public R&D funding (Cunningham, 2015). Similar claims could be made for the formula based Belgium BOF key (though without the funding argument) and other long established systems of Research Performance Based Funding Systems.

However in a comparative perspective the relative increase in performance of these systems could be seen in a different light. It is not immediately clear whether the increase in the performance of these systems is better than that of systems which were late to implement RRPBF systems or did not include them at all (e.g. the Netherlands and Switzerland).

Most Performance based funding systems aim to provide incentives for increasing the output and scientific impact of university research activities. Figure 3 and 4 present an overview of trends on several bibliometric indicators.

Table 4 shows the evolution of public funding to publicly performed R&D (the performer category includes both PROs and Universities). The reason for taking Purchasing Power Standards (PPS) at 2005 levels as the unit of measurement is that it allows for a comparison between countries with different currencies and different levels of inflation. One observes large differences in the absolute funding levels between countries. In this report we do not analyse these in depth. However, in the following figures we have ranked the countries according to their respective funding levels. What is relevant in this table is especially the evolution of public funding. One observes that most countries experienced an expansion of the public budget as measured for these years in time. Exceptions are Bulgaria, Croatia, Italy, Latvia, Hungary and the UK which showed a decrease in the public funding of the public R&D system in the last period.

Table 4. Funding evolution in the MS (PPS 2005 prices) (Eurostat 2015)

| PPS 2005 prices | % Change
| 2000 | 2006 | 2011 | 2000- 2006 | 2006- 2011
Belgium | 956.24 | 989.17 | 1283.48 | 3.44 | 29.75
Bulgaria | 168.40 | 190.64 | 159.88 | 13.21 | -16.13
Czech Republic | 615.90 | 874.34 | 1137.08 | 41.96 | 30.05
Denmark | 876.08 | 1039.95 | 1207.01 | 18.70 | 16.06
Germany) | 13674.2 | 13770.1 | 18415.1 | 0.70 | 33.73
Estonia | 44.96 | 93.96 | 135.56 | 108.99 | 44.27
Ireland | 249.19 | 509.33 | 537.85 | 104.40 | 5.60
Greece | 517.38 | 606.87 | 650.77 | 17.30 | 7.23
Spain | 2657.94 | 4269.82 | 5189.97 | 60.64 | 21.55
France | 9963.87 | 10472.93 | 11008.5 | 5.11 | 5.11
Croatia | 217.98 | 245.80 | 206.92 | 12.76 | -15.82
Italy |  | 6704.35 | 6435.73 |  | -4.01
Cyprus | 20.72 | 38.83 | 47.94 | 87.38 | 23.46
Latvia | 29.21 | 71.55 | 37.56 | 144.97 | -47.50
Luxembourg | 22.89 | 69.58 |  | 203.98 |
Hungary | 387.72 | 578.38 | 479.34 | 49.18 | -17.12
Malta | 9.92 | 11.79 | 16.61 | 18.86 | 40.92
Netherlands | 3199.73 | 3313.52 | 3496.46 | 3.56 | 5.52
Austria | 1378.94 | 1489.43 | 1866.16 | 8.01 | 25.29
Poland | 1338.90 | 1388.67 | 2299.68 | 3.72 | 65.60
Portugal | 732.08 | 748.81 | 992.50 | 2.28 | 32.54
Romania | 80.93 | 342.85 | 419.42 | 323.64 | 22.33
Slovenia | 161.15 | 200.56 | 217.11 | 24.45 | 8.26
Slovakia | 107.98 | 178.95 | 293.17 | 65.72 | 63.83
Finland | 931.64 | 1085.67 | 1247.28 | 16.53 | 14.88
Sweden | 1604.34 | 1934.16 | 2244.50 | 20.56 | 16.04
United Kingdom | 6213.82 | 7802.63 | 7371.84 | 25.57 | -5.52

Figure 3 provides an overview of the relative share of Scopus publications made by researchers in the different 28 EU member states as share of the total EU publications. These are relative figures denoting the share of EU publications. The number of EU publications has increased markedly over this period and the figure therefore does not (necessarily) show that the UK, DE and FR made less publications in 2006 or 2013 than they did in 2000. The figure only shows that their share of the total EU output has decreased. Countries like BE and NL which maintained a relatively stable share may thus in reality have seen their absolute output grow.

Figure 2. Evolution of MS share of the EU publication output 2000-2006-2013

The MS in figure 3 and 4 are ranked according to their public funding of publicly performed (universities + PROs) R&D in euros in 2011 (ESTAT, 2015). The * sign denotes the MS which did not implement a RPBF. The # sign denotes the countries which do not base the underlying assessments on publication output.

Figure 4 shows the increase in the share of highly cited publications (Tijssen et al, 2002): Most Member States show a consistent improvement in the share of their publications which are among the top 10% most cited worldwide. This is an indication that they have succeeded in raising the impact of their research output. In contrast to the other Member States, Latvia, Bulgaria and Romania do not realise a consistent increase in the quality profile of their publications.

Figure 3. Share of publications among top 10% most cited (field weighted)

Figure 3 provided an overview of the relative share of Scopus publications made by researchers in the different 28 EU member states as share of the total EU publications. It is clear that the publication share of the UK, Germany, Sweden and France is decreasing.

France and Germany have only recently adopted RPBF systems. The UK and France do not focus (much) on increasing output but more on high quality output in its peer review based RPBF. It is interesting to see that there is no one-to-one relationship between the public budget spend on publicly performed R&D and the output (the UK producing more publications than both Germany and France), especially coupled with the impact measures in which the UK also performs better than the two preceding countries.

Sweden introduced its new system with output/impact measures in 2009, but nonetheless saw its share of publications decrease further albeit slightly. The share of Belgium, the Netherlands and Denmark remained stable and as shown in Figure 2, they are among the countries with the highest share of highly cited publications as well. While Belgium and Denmark have formal incentive systems in place to promote scientific output, this is not the case for the Netherlands. In the Dutch case, however, alternative approaches to promote high impact publishing are in place, including the consideration of this parameter in evaluation exercises.¹⁹ The Netherlands, like Belgium and Denmark, also has a binary university system which results in the concentration of research funding in a limited number of high performing research universities. By contrast some of the Southern European (Italy, Spain, Greece, Portugal) and Central and Eastern European Member States (especially Poland, Czech Republic and Romania) saw their share of EU publications grow. While Italy, Poland and Czech Republic have implemented Performance Based Funding systems, this is not the case for Greece and Spain. Cuts in the public R&D budgets since the financial crisis may take some time to have a (negative) effect on the performance of the latter two systems.

On the basis of this analysis, it is not possible to draw firm conclusions on whether or not the implementation of RPBF has had an effect on impact indicators in Western European countries. There appears to be little difference in the dimension of change between countries which have (BE, DK, EE*, FI, FA, GE, IT, NO) and countries which have not put in place such incentives as part of an RPBF (ES, IE, NL). In most of the countries in which such a system has been in place for a considerable amount of time (e.g. UK or Belgium) the degree of change is relatively high in both periods, though comparable with the strongest performers which have not implemented publication incentives in their RPBF. The Central and Eastern European countries that have not put in place (or fully implemented) a performance based funding system such as Latvia, Romania, Bulgaria do not show consistent increases in their system wide impact indicators as their 2011 values are below their 2006 values. CEE countries which did implement RPBF systems (CZ, HR, SK, PL, EE), did see a steady, consistent increase in the impact of the output of their system. On the basis of this data alone, one cannot argue that this improved performance is a necessary consequence of the implementation of RPBF systems. It does however provide some context for the recommendations made by the EC to countries to implement RPBF to increase competition for organisational level funding.

When assessing the effect of performance based funding regimes on the output of research systems it is important to consider the institutional structure of countries as a variable as well. One could observe that funding in the UK is increasingly concentrated in research universities, while the “new” universities (new since the 1992 change in the higher education system) receive little of the performance based funding. To a certain extent this situation is not dissimilar in countries with a binary system (Kyvik, 2004) such as The Netherlands, Belgium and Switzerland where a very large share of research funding is allocated to the research universities and where the universities of applied science receive much less research funding. Some of the countries with binary systems also tend to perform well on performance indicators while making no or limited use of RPBF systems. In comparison to large unitary systems such as the UK, Spain and France, there may be less of a need for increasing competition between universities in such countries because concentration is part of the institutional design of the system. Nonetheless also in several such binary systems (Norway, Belgium (Flanders) and Denmark) RPBF systems have been implemented to foster competition between research universities.

In the case of Bulgaria and Latvia the real government funding of publicly performed R&D decreased. While funding reduction may not always have direct short term effects on publication output and impact, funding shortages might explain part of the observed decreases in the impact indicator. Both countries also suffer from an outflow of their scientists to Western Europe. While funding levels decreased also in the UK, one does not observe a decrease in impact indicators. It is clear that the UK has a declining share in EU publication, but the same is true for Germany and France which did not reduce their public funding (as measured in PPS – 2005 values).

6. Discussion

Butler (2003) analysed the impact of an earlier iteration of the Australian performance based funding system. She found an increase in the number of WoS indexed publications but highlighted that this had come at the expense of the average impact of these papers. The incentive system of the RPBF, should therefore be carefully thought through as it may also have unintended consequences. Sivertsen & Schneider (2012) and Larsen & Ingwesen (2014) showed how RPBF has had a positive impact on the output and a positive or neutral impact of the Norwegian and Danish system.

There are many factors which can explain the differences between e.g. Bulgaria, Latvia and Romania and the Central and Eastern European Countries that did introduce RPBF systems. Research Performance Based Funding systems can be a strong instrument to foster institutional and behavioural changes conducive to strengthening the impact of the output of research performing organisations. The implementation of RPBF (or other measures with similar targets) offers one potential approach to help these countries achieve similar improvements as witnessed for most other EU member states. However this should probably be coupled with a sustained increase in public funding of publicly performed R&D since chronic underfunding may play as big a part in their relative performance as a suboptimal institutional framework. Introducing RPBF as additional to core funding would furthermore reduce institutional resistance against its introduction.

The choice of assessment and allocation methods (quantitative/bibliometric research assessments, UK style peer review, bibliometrics informed peer review) to assess research performing organisations tends to be related to the level of analysis chosen (Hicks, 2012). From an evaluation point of view the research unit is often considered the most appropriate considering the differences in field dynamics and intra-organisational variations in the university. Peer review based evaluation tends to be used for evaluations at the departmental or research group level (Hicks, 2012; Cunningham, 2015; Boer et al, 2015). Bibliometric approaches tend to be used for organisation wide evaluations. In practice most countries opt for university or departmental level evaluations in order to reduce the scope and resource demands of the assessment exercise. In the Polish RPBF the unit of assessment is the faculty/department rather than the university as a whole. By taking into account field differences other countries may also approach this type of assessment even if they evaluate at the level of universities.

6.1 Strengths and drawbacks of peer review

Advantages of peer review include first and foremost that it is grounded in specialised knowledge of the scientific field, its methods and literature. Another strength is that it can help assess elements of research which are difficult to quantify, such as novelty. Finally it can help to come to nuanced understandings of research in context (Wilsdon et al, 2016). While peer review tends to be held in high esteem in the academic communities and therefore has a relatively high degree of acceptance, it is not without its weaknesses. These weaknesses are partially practical. Peer review is difficult to implement in small countries in which the pool of experts is insufficiently large. Relying on international experts, as is done in Estonia, can be a solution to this problem though it precludes the assessment of some types of research outputs made in the national language. Nepotism and a lack of transparency can hamper the openness and fairness that should be basic principles of the peer review process. Peer review is often also considered to be conservative, favour mainstream research and disadvantage interdisciplinary or heterodox approaches. This is partially due to the organised scepticism that is inherent in the scientific enterprise which can pose barriers to the acceptance of ideas or findings that contradict established ideas (Hicks, 2012). Struggles over resources and influence between competing schools of thought/research for influence and resources can also play a role. As indicated, e.g. the French assessments include contributions to “frontier research” in their assessment. Whether this favours interdisciplinarity research and/or ground breaking research within a disciplinary area is a potential topic for further empirical analysis.

Peer review is subjective by nature. To mitigate this, for example, the UK system requests two reviewers to assess each submitted publication. Frequently reviewers are partially guided by the reception of the submission within the academic world. Even in pure peer review systems, for example, journal impact factors are a predictor of the outcome of the exercise (Boer et al, 2015; Taylor, 2011 in Rafols et al 2012).²⁰ One of the most important drawbacks of peer review systems is the potentially large investments required in terms of resources and scientific manpower. For example for the REF, tens of thousands of submissions are reviewed by 2 reviewers in addition to the time invested in the disciplinary coordination committees for each of the fields considered (see Boer et al, 2015; Geuna and Piollato, 2016; Hicks, 2012).

Considering the costs in resources and time involved in a peer review based assessment, peer review exercises tend to be implemented irregularly or with a considerable time interval between assessments. For example a number of years pass between the RAE/REF and the Italian evaluation exercises. As a result, funding decisions may be based on out-of-date information. The quantitative assessment approaches (e.g. BE (FL), DK, and CZ²¹) by contrast can often be implemented on a year by year basis.

6.2 Advantages and disadvantages of bibliometric approaches

The use of bibliometric indicators has various advantages over classical peer based assessment mechanisms in which individual experts are requested to evaluate organisations. These advantages include their 1) relatively low costs both in terms of resources and time: bibliometrics is often used to “short-cut” experts/expertise, for by allowing for an assessment of credit awarded by the scientific community as an indicator of scientific impact, non-expert evaluators can make an assessment of these publications without having knowledge of their content – though the extent to which they can do so in a meaningful way is contested by detractors; 2) their non-intrusiveness: the researchers and organisations evaluations do in theory not need to engage in administratively heavy procedures to provide proof of their research output. The latter does not always work in practice though as in a number of systems, universities and individual researchers do have to provide the material on the basis of which they should be evaluated rather than relying on bibliometric assessments by outside experts alone; and 3) their perceived objectivity as they do not rely as clearly on the subjective assessments of individual reviewers, but instead are an indication of the use which is made of the publication by the whole scientific community.

In contrast with this potentially relatively low-cost, there are also a number of disadvantages of the sole use of bibliometric methods in assessments. The first is that while university level analyses are methodologically not too challenging, carrying out analyses at the departmental or research unit level can involve a considerable amount of investment in data collection and cleaning to ensure the data are assigned correctly (Debackere & Glanzel, 2004). When funding allocations are based on these assessments it is crucially important that stakeholders have faith in this assignment as they may otherwise challenge the outcomes of the assessments.

Other drawbacks include that some research outputs, especially in the Social Science and Humanities, are not fully covered in the bibliometric databases. Bibliometric impact measures are considered to disadvantage certain fields, such as humanities (Hicks and Wang, 2009). The use of journal indicators, including the impact factor for research evaluation is the subject of debate and criticism from within the bibliometric and wider scientific community (e.g. DORA, 2012; Hicks et al., 2015). As argued by Rafols et al (2012) and others the use of journal based indicators disadvantages a.o. interdisciplinary research. Findings on whether more sophisticated bibliometric indicators also have this problem are mixed (Rinia et al., 2001 HEFCE, 2015). The outcomes of bibliometric assessment are sensitive to the methodology adopted and the choice of indicators which in the absence of clear standards may remain a subjective decision. A consensus appears to be emerging among Bibliometricians on the more and less suitable forms of indicators (Hicks et al, 2015). However it remains important to consider that the sole use and at times misuse of bibliometric indicators can create perverse incentives (e.g. gaming behaviors). Many leading bibliometricians argue that bibliometrics should be an input to rather than a replacement of peer review in evaluations of research organisations.

6.3 Bibliometrics informed peer review

Partially for cost consideration and partially because of a recognition of the potential of bibliometrics to contribute to quality assessments, many governments have implemented a mixed approach in which the peer review process is to a greater or larger extent based on or informed by bibliometric analyses (e.g. Italy). In the latest UK REF, peer review panels were given the option to receive the results of bibliometric analysis of the analysed departments – though departments are only asked to select a limited number of publications (per researcher). Some of the panels decided to make use of this, but were not given top down instructions on how to use these metrics, apart from the indication that journal impact factors should not be used as a proxy for quality. Hicks (2012) argues that arguably “departmental or field level [performance based funding systems] using peer judgment based on indicators represents the current state of the art”. It is such systems which have been implemented in many countries in recent years. On the other hand the Wilsdon review (2015) carried out for the HEFCE does not agree that a systematic integration of bibliometric indicators in the REF would constitute an improvement in the UK context at present. In the UK the debate concerning the potential of metrics to replace or inform the peer review system has thus continued and will continue to do so in the future as well.²² It is also interesting to note that e.g. Sweden may introduce a greater peer review component in new iterations of its assessment exercises. Performance Based Funding Systems thus do not clearly evolve in a specific direction.

6.4 University autonomy

The degree of “University autonomy” is affected by Performance Based Funding Systems in several ways (Herbst, 2007; Hazelkorn, 2010; Hicks, 2012). A certain degree of university autonomy is considered by many actors (e.g. EUA, 2013) to be essential for the functioning of research universities as it allows these organisations to engage strategically with their environments and build long term strategies for the improvement of their performance (see also Aghion et al, 2010). When reflecting on university autonomy it is useful to consider the notion of “organisational actorhood” and make a distinction between “internal autonomy”, or the extent to which university leadership has control over the researchers working in the professional work organisation which a university is, and “external autonomy”, which relates to the autonomy of university leadership in the context of environmental pressures including the autonomy that the organisation has vis a vis its principal (national or regional governments) (Cruz-Castro et al, 2015, Krücken & Meier 2006; Whitley 2008). The degree of internal autonomy affects the extent to which universities can promote/reward/recruit staff for helping them to attain good assessments. In many countries performance based systems were implemented to increase internal university autonomy which was virtually non-existent in the not so distant past: policy organisations distributed funds directly to individual research groups or researchers without much interference or control by university leadership (Hicks, 2012, Herbst, 2007). Also, because the central assessment of research requires improved data collection, it can be beneficial for strategic planning and management at the organisational level and in doing so increase both internal and external autonomy (Hazelkorn, 2010). On the other hand, as suggested by the principal agent discussion above, Performance Based Funding also constrains the potential for strategic action by university leadership by giving greater control to policy makers over university behaviour by setting targets/indicators, assessing performance and rewarding “good” or punishing “bad” performance in the medium term. The introduction of Performance Based Systems in MS which currently did not have related traditions can introduce considerable instability and unpredictability, which hampers the ability of research organisations to plan strategically. In e.g. the Czech Republic, Poland and Denmark, research funders therefore decided to base a substantial part of the funding on previous funding distributions, though the historical element was removed from the funding formula in Poland in 2015 (Good et al, 2015; Klincewicz et al, 2015; Kalpazidou & Schmidt, 2012). Performance contracts have on the one hand a constraining function. On the other, governments also commit themselves to e.g. the supply of certain levels of funding if agreements are met which can provide the stability that allows university leadership to engage in strategic planning to a greater extent.

6.5 Costs of RPBF

Several analysts recommend the introduction of Performance Based Funding systems in MS which do not yet have these (ES, BG), since they are considered a (relatively) budget neutral way to improve efficiency (ERAC, 2014; Todorova & Perez, 2015; WorldBank 2014). As indicated in various places in this report, there are (considerable) costs, in money, time and resources, involved in doing the assessments on which the funding allocation decisions are made. The costs involved in the different types of exercises differ. Probably at the higher end of the scale are the large scale assessment exercises carried out in the UK and Italy. But as argued by Geuna and Piolatto (2016), between these systems there are differences too. For the UK less than 1% of total research funding is consumed by the evaluations exercise whereas in Italy it is estimated to be above 2.5%. This in contrast to the amount of funding which is allocated through the respective exercises which is higher (approximately 20% of organisational level funding) in the UK than it is in Italy (13%). Apart from direct costs related to the running of the exercise (the set up and running of an evaluation agency; the contracting of peer reviewers; the data collection and analysis as well as the set-up of an information system for this purpose; etc), consideration should also be given to the substantial costs borne by the research performing organisations in coordinating their research submissions (Hicks, 2012; Geuna and Piolatto, 2016). The metrics based exercises as carried out in e.g. Poland, Czech Republic and Finland are expected to be less costly and can partially for that reason also be carried out on a more regular basis. However, also in these countries universities are often responsible for submitting the material on which they are evaluated. Exercises such as the Belgian BOF-key allocations do not rely on the universities for the submission of their research output. This type of formula based funding allocation systems, while difficult to design and implement, is also expected to have relatively low running costs as it does not involve (large teams of) peer reviewers. As discussed in the preceding sections they can also have unintended consequences. These types of systems therefore have to be very carefully designed and be open to fine tuning. The extent to which they are suitable to systems in different stages of development should also be considered.

7. Conclusions

The implementation of Research Performance Based Funding (RPBF) systems aims to improve research cultures and facilitate institutional changes that can help increase research performance. Many EU countries have introduced, are introducing or are considering to introduce such systems. There are alternatives to Research Performance Based Funding. This includes performance contracts. Binary systems such as Switzerland and the Netherlands can offer another alternative approach to the concentration of researchers in strong performing organisations. The logic that placing incentives on certain types of behaviour (such as publishing in high impact international journals) and the concentration of resources in well-performing units to attain a certain critical mass can lead to sustained improvements in output and impact at the systemic level, seems hard to refute.

On the indicators considered in this report none of the systems which have implemented a RPBF have experienced strong negative effects. Though many other factors are thought to have influenced the observed improvements in performance, the effects are likely to have been positive. These indicators, however, do not necessarily tell the whole story and in a mutual learning exercise it is crucial to get a clear understanding of potential downsides and perverse incentives generated by specific designs and implementations of Performance Based Funding Systems: including and beyond the issue of research excellence as measured, necessarily imperfectly, by these metrics.

The role of Performance Based Funding is likely to differ in relation to the institutional design of the system. Systems with large Public Research Centres such as the Academies of Sciences in Central and Eastern European countries need to decide whether to include these units. The rationale for and likely impact of performance based university funding is likely to differ between unitary systems such as the UK and Spain and binary systems in which the concentration of research funding is implied in the separation between research universities and universities of applied science.

Other issues in which Member States differ include the share of organisational level funding which is allocated through RPBF, the speed in which the system is introduced, the degree of stakeholder involvement, the impact different systems have on the autonomy of research performers, the criteria on which they evaluate, their likely impact on research excellence indicators as well as the other missions and behaviours which the government wants to promote in these organisations. An extended analysis of the impact of funding allocation regimes on research system performance will be carried out in a follow up to this report.

Many countries inside and outside Europe have learned from the evolution (design and improvements) of the RAE and REF in the UK which was the first country to introduce a funding allocation system based on peer review assessments in Europe. As suggested by the ongoing redesigns of the systems in for example the Czech Republic and Sweden, lessons can be learned from past experiences. Especially for those Central and Eastern European countries which at present have not implemented any such system and which are considering to do so, an overview of the pro’s and con’s of different designs is expected to be beneficial. Also systems in which the RPBF are a recent or even a long established feature can benefit from further adapting or fine tuning their approaches.

Decisions to implement a RPBF should include assessments of the (expected) costs of different systems. In these assessments the considerable costs which some systems bring to the research performing organisations should not be overlooked. These costs should be weighted, in so far as possible, with the potential benefits the introduction of such systems can bring. Of crucial importance is a consideration of the potential unintended consequences which different incentives, indicators and methodologies can generate. A strategy to monitor the effects and impacts of the system should be in place and the administrating agency should be open/flexible to fine-tune the approach. It is important to involve stakeholders in deciding on the indicators considered. It should be realised, however, that in the absence of additional funding, there is likely to be a certain degree of institutional resistance from parts of the academic community and research organisations. This resistance in itself should not be an argument to forego on the decision to implement such systems. However it is clear, as is argued by the EUA (2015) and Geuna and Piolatto (2016), that it will be easier to implement Performance Based Funding systems in a period in which the funding of public research performers is expanding.

Considering the relative differences in costs of time, resources and timeliness, especially the allocation mechanisms based on quantitative assessments appear of immediate interest to countries which have not yet established RPBF. However, as observed in e.g. Sweden, the UK and the Czech Republic, governments are re-evaluating the extent to which such approaches are most beneficial to their system. These reconsiderations are possibly in part due to institutional resistance from research performing actors in these systems, but possibly also due to a realisation of their potential unintended negative consequences or the perception that alternative incentives can help to improve the performance of their system more.

An important factor influencing the impact of a Performance Based Research System is the amount of funding that is allocated on the basis of assessment of the research output. The levels of performance based funding and the rate at which it is introduced or at which allocations can change is important as it affects the ability of research organisations to engage in longer term planning. In several countries, such as Poland, Czech Republic and Denmark the introduction of the system has been gradual in order to ensure the system remains in balance. Also in long established systems such as the UK, historical considerations play a role in funding allocation decisions in order to avoid large swings (especially falls) in the amount of funding granted to specific universities (De Boer et al, 2015). An example which deserves further reflection is the Danish model in which each year a small share (2 percent) is re-allocated on performance based criteria while the rest is mainly allocated on the basis of the previous’ years funding. Over time this resulted in a gradual increase in the “accumulated” performance based component of organisational level funding.

In the discussion this report elaborated on several of the advantages and disadvantages of both peer review and bibliometric approaches. This debate is too big to fully capture in this report, but some of the references provided can present relevant further reading. What is clear from past experience is that if bibliometric indicators are used to underpin funding allocation mechanisms, it is important to take into account field differences, paying special attention in this light to the Social Sciences and Humanities (see also the Belgian (Flanders) and Czech experience in Annex 1). Simple publication counts alone are not considered to provide the most appropriate incentives for an upgrade of research systems. Some of the debate and criticism towards journal based impact measures were provided, but research funders in a number of countries still consider these or alternative journal based measures to generate desirable incentives. Publication level citation based measures are generally considered superior to the journal based approaches though they are more difficult to implement. When introducing such measures they can be accompanied by other output and impact measures to provide additional incentives and potentially mitigate some of the adverse side effects by a sole use of bibliometric indicators. In many countries which rely on quantitative assessments experts still play a role to interpret the data in the light of a.o. field specificities. Monitoring is also necessary to guard against excessive “gaming” by research organisations in the light of the indicators considered.

The involvement of peers/experts remains the preferred approach to research assessment for many analysts who sometimes doubt the ability of bibliometrics to provide a good and complete assessment of the output of research systems. As was shown, metrics informed peer review exists in different forms in the member states that adopted them. While peer review has limitations of its own, these may be partially addressed by the input from these publication analyses. Whether the benefits from full-fledged exercises outweigh their costs is an issue that needs to be decided on a country by country basis. Some argue that the relative costs may be too high for smaller research systems (Geuna and Piolatto, 2016).

It is important to start reflecting on the potential of alternative metrics and indicators and to pay close attention to the various mechanisms which are currently being developed to gauge societal impact (see e.g. the examples of UK, France and The Netherlands). Striking the right balance between scientific excellence and societal impact is among the main challenges in the reform of public sector R&D systems in Central and Eastern European, and indeed all, EU Member States. Whether the assessment exercises that underpin RPBF allocation decisions are the most appropriate avenue for evaluating and incentivizing the societal relevance of public research remains an open question.

The impact of RPBF and the research funding allocation system in general will be analysed in more depth in follow up work related to this report.

Annex 1 Country descriptions of RPBF funding systems²³

1. Austria²⁴

Project vs institutional funding

According to the data collection in the PREF study Austria’s public allocations for publicly performed research is allocated for 72 % and 27% in the form of organisational level (institutional) funding and project funding respectively. These data are in line with Eurostat figures. Its share of organisational level (institutional) funding is thus comparatively high.

Timing and method

In Austria the financing structure of the higher education system changed considerably with the University Act of 2002 which bases competitive organisational and block funding to universities on three-year performance contracts between each university and the BMWFW. The 2002 regulations of this financing system have been revised in 2013, 2014 and 2015 and now increasingly account for research, teaching and infrastructure demands at Austrian universities. It aims to make financing more transparent and flexible, e.g. easier and faster budget access when buying equipment or hiring professors. Revisions will become fully effective by 2019, at the latest. The “Gesamtösterreichischen Universitätsentwicklungsplan” headed and drafted by the BMWFW in cooperation with the Assembly of Universities and the Austrian Science Council, will become effective for the performance contract periods 2016-2018 and 2019-2021.

Modality of the assessment

Competitive funding to HEIs outlined in these performance contracts considers separate budget areas, namely 1) budgets for teaching, 2) budgets for research and advancement of the arts, and 3) budgets for large scale infrastructures. HEI performance assessments in all three budget areas are based on a variety of indicators. These indicators include 1) number of active students per scientific discipline, number of graduates per scientific discipline, 2) the number of PhDs, project volume financed by the Austrian Science Fund, project volume financed by industry or other sources, and 3) societal indicators such as the share of women in PhDs programs, participation in outgoing mobility programs, number of foreign graduate students.

Since the research part is either based on PhD completion rates which are not assessed or on input factors such as the volume of project funding received, Austria is not included among the countries that have adopted performance based funding according to Diana Hicks’ definition (2012), because it does not include an assessment of the research output apart from PhD degrees awarded. Currently, more than 20% of total organisational funds to HEIs are competitively allocated according to this organisational funding model. The latter model will be fully implemented until the performance contract period 2019-2021. For the period 2016-2018 up to 60% of organisational funding to HEIs will be based upon this model. The share of organisational funds in Austria that is allocated competitively has been increasing in the past years and will continue to do so in the next few years (Cuntz, 2015). For further information on the Austrian system, also see De Boer et al (2015).

2. Belgium

Project vs organisational level (institutional) funding

According to the data collection in the PREF study Belgium’s public allocations for publicly performed research is allocated for 45 % and 55 %in the form of organisational level (institutional) funding and project funding respectively. Its share of project funding is thus comparatively high.

In 2003, Flanders pre-established ‘pre-allocation keys’ which are recalculated every year and published in the Belgian State’s official legislative documents (Moniteur Belge – Belgisch Staatsblad). The Department of Education and Training provides organisational level block funding for universities and the Department of Economics, Science and Innovation (EWI) provides performance based funding through the Special Research Funds (Bijzondere OnderzoeksFondsen: BOF) and the Industrial Research Funds (Industrieel OnderzoeksFonds – IOF). Project funding is allocated by the FWO. Finally there is the so-called ‘Hercules-key’ through which funding is allocated to support research infrastructures at universities and PRO’s (Duchene, 2015)²⁵.

In Wallonia, the two main sources of university institutional funding are the Special Research Fund (Fonds Spécial pour la Recherche, FSR) and the Concerted Research Actions (Actions de Recherche Concertées, ARC). As is the case for Flanders, allocation keys are used, but they are generally less tied to competitive output parameters (Decree 30.01.2014) (Kerchtelmans & Zacharewicz, forthcoming).²⁶

Modality of the assessment

BOF funding is formula based according to the so-called “BOF-Key”. The system was last adapted in 2013. The bibliometric analysis on which the BOF key is based is performed by the inter-university centre ECOOM. The data on degrees is collected in a central database on Higher Education in Flanders²⁷.

Since 2012 (Decision Flemish Government dd. 21.12.2012), the BOF allocation formula takes into account five key indicators: master degrees, defended doctorates, gender diversity, publications and citations (Spruyt & Engels, 2013).

Table A1. Parameters included in the calculation of BOF allocation formula²⁸

Year t | 2013 | 2014 | 2015 | From 2016
Structural part |  |  |  |
Weight Master and Bachelor Degrees | 25,00 % | 25,00 % | 24,00 % | 23,00 %
Weight PhDs awarded | 35,00 % | 35,00 % | 35,00 % | 35,00 %
Diversity parameter (gender distribution) | 3,00 % | 2,00 % | 2,00 % | 2,00 %
Sum | 63,00 % | 62,00 % | 61,00 % | 60,00 %
Bibliometric part |  |  |  |
Weight publications in WoS | 15,36 % | 15,77 % | 16,19 % | 16,60 %
Weight SSH publications in VVAB | 6,28 % | 6,46 % | 6,62 % | 6,80 %
Weight citations | 15,36 % | 15,77 % | 16,19 % | 16,60 %
Sum | 37,00 % | 38,00 % | 39,00 % | 40,00 %

Publications which are indexed in the WoS are grouped into categories based among others on whether or not an impact factor was assigned to them. The journals are assigned for different disciplines to 20 equal sized categories on the basis of their average impact factor in a ten year period. Publications are then weighted on the basis of the category in which the journal in which they are published is placed in the manner indicated in table A2:

Table A2.

20 equal sized categories of journals based on their impact factor | 1 | 2 | 3 | 4 | 5-14 | 15-16 | 17 | 18 | 19 | 20
Weight | 10 | 6 | 3 | 2 | 1 | 0,5 | 0,1 | 0,1 | 0,1 | 0,1

Source: Ministry of Education Flanders (2015)

In addition to publication data, citation data is also considered. For each broad discipline, citation data to the publications made by an organisation is analysed for a ten year time frame.

To account for the underrepresentation of SSH journals in the WoS, a new database was created with relevant SSH journals: the Flemish Academic Database for the Social Sciences and Humanities (VVAB). Publications in this database are considered since 2010 and their relative weight has increased since 2012.

When bibliometric assessment was first introduced in the allocation of the BOF key, 10% of the funding was based upon it (DeBackere & Glanzel, 2004). This was increased to 36% in 2008 (Hicks, 2012; DeBackere and Veugelers, 2015). Between 2013 and 2016, the bibliometric share increased from 37% to 40%. Quantitative measures of research output and scientific impact therefore determine a considerable share of the variable organisational funding provided to Flemish universities (DeBackere and Veugelers, 2015. Spruyt & Engels, 2013).

In Wallonia, the FSR provides funding on the basis of the number of awarded degrees i.e. not a research parameter. For the ARCs, 80% is allocated based on awarded degrees while the remaining 20% is allocated based on the university share in terms of the following (equally-weighted) criteria:

1. European FP funding;

2. Post-doctoral researchers;

3. Academic staff who obtained their PhD from another university;

4. Publications;

5. Citations.

3. Bulgaria²⁹

Project vs organisational level (institutional) funding

According to the data collection in the PREF study Bulgaria’s public allocations for publicly performed research is allocated for 54 % and 46 % in the form of organisational level (institutional) funding and project funding respectively. Its share of project funding is thus comparatively high.

In Bulgaria according to previous versions of “The Higher Education Act” the state envisaged a subsidy for research, innovation and creative activities to HEIs not less than 10% of the education grant (Todeva, 2015). It should be noted that the version of the Law for Higher Education last amended 24 July 2015 does not specify a portion of the state grant to be allocated for the research activities of HEIs.

On 26 February 2015 the Parliament adopted a Strategy for the Development of Higher Education in Bulgaria 2014-2020. The Strategy foresees to strengthen the role of the quality of the education and the realization of the graduates in the allocation of funds to the HEIs.

Modality of the assessment

For 2014 all public universities and HEIs received in total €4.1m, or 2.77% organisational funding for research, distributed on equal basis to all state HEIs in addition to their education grant (Todeva, 2015). The organisational funding for HEIs is subject to a ministerial decision by MES.

The Ordinance № 9/8.08.2003 stipulates that the 10% share for research activities can be increased or decreased depending on the submission of yearly/ half-yearly reports by the respective HEI.

The education grant itself also has a performance based component, as it is constituted on the basis of the number of students, and a complex evaluation using numerous indicators for measurement of the quality of education based on the evaluation from the accreditation and teaching staff (30%), the evaluation of the volume and impact of research and publication outputs (25%). This includes citation impact by scientific field, citation impact without self citations by field, the number of publications cited at least once, the number of articles in scientific journals and the availability of PhD programmes / graduate schools. And finally, employment of graduates and links to the labour market (45%). (Law for Higher Education; Decree of the Council of Ministers № 15 of 30 January 2015).

There thus appears to have already been a performance based funding element in the allocation of organisational funding to Bulgarian Universities. However, when one considers the evolution of the budgets of universities over the past years one observes no, or very limited change, over time. This suggests that the performance based funding regime has not yet been fully implemented or has limited effect on the redistribution of resources. Plans are being developed to increase the role of RPBF.

The recent Peer Review on Bulgaria states that: “In several reports on Bulgaria, there are recommendations to change and introduce legislation to control the development of the Higher Education sector. One of the topics high on the agenda is performance-based funding. A World Bank report from 2013 recommends that ‘funding would be allocated on the basis of regular, independent monitoring and evaluation of each PRO’s performance’. (Worldbank 2013) As a first step towards a performance-based system, the Bulgarian government has drafted regulations for monitoring and evaluating the research performance of higher education institutions and research institutes (and as referred to above on the NSF). According to these regulations, there will be an annual evaluation of all institutions. According to the plan, a commission of 13 independent experts appointed by the Minister of Education and Science will be responsible for evaluating the research performance of all institutions, based on fixed metrics.” (Soete et al, 2015).

4. Croatia³⁰

According to the data collection in the PREF study Croatia’s public allocations for publicly performed research is allocated for 8 % and 92 %in the form of organisational level (institutional) funding and project funding respectively. Its share of organisational level (institutional) funding is thus comparatively (very) low.

Up to 2013, 80 to 85% of academic (i.e. universities and research institutes) research activities were block-funded. The funding was allocated by the Ministry of Science, Education and Sport (MSES) and decomposed as such: 70% for salaries, 10% of direct organisational funding (overheads, phone, energy, etc.), 10% of research grants (material and operational costs) and remaining 10% other research-supporting activities (conferences, publishing, etc.).

The share of project based funding in GBAORD was rather low until 2013, and amounted to 6.2% in 2011 and 5.6% in 2012, while institutional funding amounted to 93.8% and 94.4%, respectively. The reforms started in 2013 will significantly increase the share of project-based funding in total public funding of R&D activities in Croatia, which already increased to 8.5% in 2013.

Amendments to the Act on Science and Higher Education, adopted in 2013, introduced significant changes to the overall Croatian R&D system. A new model for public R&D funding has been developed, focused on awarding multi-annual block grants for HEIs and PROs from the State budget by MSES. Funding is now awarded at a three-year basis, with funding amounts determined on the basis of institutional performance indicators. For the period 2013-2015, indicators have been evaluated during the first half of 2013, after which the exact funding amounts have been agreed between MSES and HEI/PROs. A decision on multi-annual institutional funding of research activities in public research institutes and universities for the period 2013-2015 has been adopted in June 2013.

Another new form of organisational level funding has been introduced in 2014. This refers to the Centres of Research Excellence, whose establishment will be funded by MSES from the State Budget. For the period 2014-2016, €0.330m annually has been reserved within the State Budget, which should be sufficient for the initial phase of establishing 7 centres. Additional financial resources for Centres are planned to be ensured from the EU Structural Funds.

Modality of assessment

The amount awarded depends on institutional performance indicators of scientific activity in the previous five-year period. These include four types of criteria weighted differently depending on their importance for research evaluation, as follows:

• Scientific productivity (with the largest weight of 60%): this criterion includes the number of scientific papers published in journals covered by the Web of Science and SCOPUS, number of citations and the number of employees who attained a doctoral degree in a given period. The number and type of publications are further specified by the 6 scientific areas of natural, technical, biotechnical, biomedical, social sciences and humanities;

• National and international competitive research projects and mobility (25% weight) includes the number of national competitive research projects (CSF and UKF) and the number of projects funded by foreign sources (FP6, FP7, ESF, HERA, COST);

• Popularization of science (5% weight) includes participations in actions like Festival of Science;

• Commercialisation of science (5 % weight) includes the number of contracted projects with businesses, government bodies and local authorities, civil sector and non-governmental organizations.

The allocation of total funds of € 6.6m per year to each institution depends on the institutional coefficient, calculated by a complex formula which includes the criteria defined above and a number of other elements combined with the specific coefficients and ratios such as: an average and basic value of scientific area, the coefficient for a particular area of science (natural sciences = 2.7; engineering = 2.5; biomedical Sciences = 2.7; biotechnology = 2.6; social sciences = 1.2; humanities and arts = 1.2), number of researchers in full-time equivalent (FTE) by a field of science, etc.

Each institution has to deliver an annual report on its scientific activities in standardized format. This report is evaluated following the criteria previously exposed and used for the calculation of the organisational funding.

5. Cyprus³¹

In Cyprus, the government plans organisational funding annually through the state budget. It is exclusively based on education metric criteria (number of students and researchers) and is not associated to research performance indicators. It is directed to Higher Education Institutions and Public Research Organisations.

6. Czech Republic

Project vs institutional funding

The public R&D funding has been traditionally dominated by institutional support. However, this is changing in the context of the Reform of the RDI System that has been launched in 2008. According to the data collection in the PREF study the Czech Republic’s public allocations for publicly performed research is allocated for 21 % and 79 % in the form of organisational level (institutional) funding and project funding respectively. Its share of organisational level (institutional) funding is thus comparatively low. The PREF analysis of the share of project vs institutional funding appears incompatible with national level data (CRDI 2014), which indicates that project funding will be between 50 and 52 % in the period 2015-2017 up from 44% in 2009. This apparent discrepancy needs to be assessed in the continuation of the PREF project.

Timing and method

In the Czech Republic initial reform plans considered to allocate almost the full amount of organizational funding using performance-based formulae. A medium term modification of the evaluation methodology has been introduced for the period 2013-2015, so-called Metodika 2013 (CRDI, 2013). Organizational level funding will be allocated following this new evaluation methodology at least until 2017. After 2016 a new system of RDI evaluation and distribution of organizational funding developed by the Technopolis Group is scheduled to be gradually implemented (Srholec, 2015; Good et al, 2015). The expectation is that this new system will be based on informed peer review, hence relying to a lesser extent on quantitative output indicators (Good et al, 2015).

Methodology

In the current system, the dominant part of the assessment is based on journal (impact factor) based bibliometrics (Malek et al., 2014; Srholec 2015). A second part of the evaluation is based on high impact results, including ERC grants. Each organisation selects a number of high quality results in each scientific field under consideration. The number of results selected is related to the funding which the organisation received for that field in the preceding year. These results are peer reviewed and the best outputs are selected, for which organizations obtain funding bonuses. A third part of the evaluation concerns non-scientific outputs of R&D, including patents and plant varieties. The different categories considered come with a set number of points per item. The remaining points in this part of the assessment exercise are allocated on the basis of an assessment of income generated through applied projects and collaboration with industry.

Funding levels

The funding allocation decisions are based for 75% of the assessment of the scientific output, for 10% on the evaluation of high impact research outputs and for 15% on the assessment of applied research (Malek et al, 2014). In order to stabilize the funding flows, a consensus has been reached that at present only 20% of the money is allocated using the evaluation results, hence performance based, while 80% of the money is proposed to be divided in the same proportion as in the previous year, hence in the form of block funding based on historical considerations (Srholec, 2015).

7. Denmark

According to the data collection in the PREF study Denmark’s public allocations for publicly performed research is allocated for 54 % and 46 % in the form of organisational level (institutional) funding and project funding respectively.

Up until 2009, the Danish government funded universities on the basis of a model that covered indicators for education, external funding and PhD graduates.

The education part of public funding is strongly based on education metrics and particularly on the number of students who passed an exam. The amount of funding per exam varies between different fields of study (Van Daalen et al, 2014).

The amount of organisational level research funding is mainly based on historical grounds (98 % based on the previous year’s budget). Each year, 2% of the research funding is retained in a “restructuring fund” and redistributed among the universities. Over the years the importance of this cumulative performance based element has become substantial.

The European University Association’s DEFINE project estimated that the cumulative share of organisational level funding, in 2012 was for 54% based on historical principles, for 27% on research performance, and for 15% on a PhD performance model and for 4% on earmarked research funding. The Danish Ministry of Higher Education and Science estimated that in 2013 the historical principle based part was 50%, while 30 % was based on (accumulated) research performance and 20 % came from investments in PhD programmes (De Boer et al, 2015).

Modality of the assessment

Until 2010, the redistribution was based on a 50-40-10 ratio. That is, 50% was based on the level of educational funding, 40% was based on the amount of external research funding, and 10% was based on the number of PhD graduates (Van Daalen et al, 2014, Kalpazidou Schmidt et al, 2006).

In June 2009, a political agreement was reached on a new distribution model for the allocation of organisational funding to universities. This new model also includes bibliometric indicators and was introduced stepwise over the period 2010-2012 at the expense of the weighting for external research funding. In 2013 it was agreed to continue the distribution model for five more years (Grimpe, 2015; Van Daalen et al, 2014).

From 2012 onwards instead of the 50-40-10 model, 45% of the (2% of variable) funding funds was distributed according to earned education related indicators, 20% was distributed according to research activity financed by external funds (project funding from research councils or EU funding), 25% was distributed based on bibliometric indicators, while 10% was based on the number of PhD graduates (Kalpazidou Schmidt, 2010; personal communication Kalpazidou Schmidt, 2015).

The principles for the bibliometric indicator are summarised in a report published by the DASTI in October 2009 (Grimpe, 2015). This system drew inspiration from the Norwegian system (Fosse Hansen, 2009).

There are several publication types such as books, book chapters, journal articles, PhD-and doctoral-theses, and patents and each publication is worth points – at two levels (the elite level and the other level) (De Boer et al, 2015). The system is journal based: publications are counted in around 20,000 peer reviewed journals and publishing houses. The journals and publishers have been selected by 68 panels of researchers. They are divided into two groups, where publications in the top 20% receive a higher weight than publications in the other outlets (Fosse Hansen, 2009).

The most recent results for the bibliometric indicators are published on the homepage of DASTI (Grimpe, 2015). In addition to the performance based funding the government has also engaged in the signing of performance contracts with the Danish universities (Grimpe, 2015).³²

8. Estonia³³

Project vs institutional funding

Estonia's funding for publicly performed research is allocated for 26% and 74% in the form of organisational level (institutional) funding and project funding respectively. Its share of organisational level (institutional) funding is thus comparatively low.

In the Estonian RDI system public RDI funding appropriations for directly funding of RDI institutions 16% in 2011 and 19% in 2015 and 26 % in 2016 were allocated in the form of performance based funding. This part is organizational funding and it was introduced in 2005. The rest was allocated as competitive funds in the form of personal research grant and institutional research grant (not institutional or organizational in the frames of terminology used in this report).

Timing and method

In Estonia, annual organizational funding is allocated on the basis of universities and PROs' R&D performance indicators.

The general criteria for allocating organizational level funds for RDI institutions (baseline funding in Estonian legislation) are set in the 16.12.2004 amendments to Organization of Research and Development Act³⁴ (introduced in 1997). The detailed conditions are specified by a decree of the Minister of Education and Research. Annual baseline funding is allocated to R&D institutions, provided they have received a regular positive evaluation.

The following methodology (calculation formula) is used: the first 5 % is allocated to humanitarian research of national significance, while the remaining 95 % is distributed according to performance criteria. Among these, 50 % refer to the number of high level publications in internationally recognized journals, the number of high level research monographs and the number of registered patents and patent applications; 40 % refer to the amount of financing of R&D from other sources i.e. targeted research, commissioned by enterprises, municipalities, ministries, etc. The 10% remaining refer to the number of doctoral graduates.

To avoid unexpected sharp changes in funding flows, the performance in the three last years is taken into account. From 2017 a new formula incentivising more cooperation with private and public institutions will be introduced. The weight of financing R&D from other sources will be raised from 40 to 50% while the weight of high level publications and patents will reduced from 50 to 40%.

Modality of the assessment

To carry out the regular evaluation of R&D institutions, the Minister of Education and Research forms a 3-16-member evaluation committee composed of foreign experts. The evaluation of the applications for research funding is organized by the Estonian Research Council.

Proposals from the beginning of January 2015 targeting both the high share (ca 80%) of project based funding and the confusion about research funding instruments were put forward. They suggest that the current “triple breakdown” into institutional research funding, personal research funding and baseline funding should be replaced by a “dual breakdown”, which would consist of stable operating subsidies to research institutions (baseline funding) and competition-based research grants. Their proportion should be shifted from the existing ratio of 20:80 to 50:50 (MER news, 5.01.2015) by raising the amount of baseline funding. To provide more stability for operational expenses during the transition period, the funding allocated through competitive mechanisms should also grow but in a slower pace (MER, 2015). A growth in the budget for institutional base-line funding has been announced, rising from EUR 9.2 million in 2015 to EUR 13.9 million in 2016.

9. Finland

According to the data collection in the PREF study, Finland’s public allocations for publicly performed research is allocated for 56 % and 44% in the form of organisational level (institutional) funding and project funding respectively.

Already in 1994, the Finnish government introduced performance agreements for higher education institutions. The formula based allocation of competitive research funding system that was in place since 1998 was changed in 2010 (Hicks, 2012) and again in 2013, while for polytechnics a new funding formula was introduced in 2014 (Boer et al, 2015). While the performance contracts and the funding allocation model cover many of the same indicators, not all the indicators included in the performance contract are used to based funding allocation decisions on and vice versa (Boer et al, 2015).

The core funding of universities is based on a fixed formula. Before 2010, the key components of the system on which the allocation of organisational funding was based considered the following indicators: university’s operational expenditure in the last year of the previous performance agreement period; new students and other targets set in performance contracts; education appropriation, research appropriation including graduate schools, number of PhD degrees and number of completed PhD degrees and societal services appropriation, project funding and the number of Centres of excellence, funded from the Academy of Finland, amount of external resources (Auranen & Nieminen 2010).

The formula was adjusted in 2013 and, in comparison to the formula used in the 2010–2012 performance period, puts more emphasis on quality, effectiveness and internationalisation. The funding formula comprises three main parts: education, research, and other education and science policy objectives (Boer et al, 2015; Saarnivaara, 2015). Regarding research, the funding model takes into consideration bibliometric indicators (peer reviewed publications), the number of doctoral degrees awarded and the internationalization of students and staff. Universities also sign a performance contract.

The variables in the funding allocation models for universities and polytechnics have different weights. For universities the education, research and strategic development pillars accounting for 41%, 34% and 25% respectively. For universities, alignment with overall education and science policy goals is thus an important ingredient (25%) of the funding agreement. Here, the strategy of the university (10%), its orientation towards specific disciplinary fields (8%) and the carrying out of national duties (7%) are fed into the funding model (source: www.minedu.fi in Boer et al, 2015; Saarnivaara, 2015).³⁵

The government funding for polytechnics, universities of applied science, will be allocated on a 70-30% ratio on the basis of the number of degrees awarded and other performance indicators (Saarnivaara, 2015). This ratio is applied after the subtraction of the share for strategic funding (2.5% of the total funding); though it appears that in 2016 there will be no strategic funding. Performance indicators cover the pillars education (85%), research (15%), and strategic development. Indicators relate to degrees conferred, student progress, research productivity, external research funding (including by the Academy of Sciences and TEKES, an innovation agency), contract income, and internationalisation (student mobility) (source: www.minedu.fi in Boer et al, 2015).

Funding levels

In 2013, the share of organisational level funding for research in HEIs was 42%. According to the new performance and quality based funding model of universities the Government appropriations for universities (90% of HEI organisational level research funding) are directed especially on the basis of completed qualifications and credits as well as scientific publications and competitive project funding (10% of HEI organisational level research funding) is revised to better support improvement in the quality of teaching and research (Saarnivaara, 2015).

10. France³⁶

According to the data collection in the PREF study France’s public allocations for publicly performed research is allocated for 79 % and 21 %in the form of organisational level (institutional) funding and project funding respectively. These data differ from Eurostat figures, which indicate a ratio of 93% for organisational funding and 7% for project funding. These deviations will be explored further. Independently of their differences, both Eurostat and PREF figures show a high share of organisational level (institutional) funding.

The French Ministry of Higher Education and Research established the AERES National Agency in 2008 to undertake, at national level, the evaluation of research in both research organisations and universities. This organisation has been replaced in 2013 by the High Council of the Evaluation of Research and Higher Education (HCERES).

The main objective of HCERES is the evaluation of all French research teams and research Centres. The evaluations provided are based on a peer-review assessment that includes, but is not exclusively based on publication and citation analysis.

The main criteria used in the evaluation are as follows (EC, 2010):

- Quality of research (mainly based on the scientific production of the lab);

- International recognition of the lab (by using some quantitative indicators, such as the number of citations etc.);

- Participation in national and/or international research networks or programmes; research grants obtained from ANR (Agence Nationale de la Recherche) at a national level, from the European Commission or from other European or international funding organisations;

- Risk taking in research, and openness to interdisciplinary interfaces (frontier research);

- Openness towards societal challenges;

1. Investment in the dissemination of scientific culture; valorization (patents);

2. Number of full research fellows (chercheurs) in the lab, compared to other members (university faculty) who teach and are active in research for 50% of their time (enseignants-chercheurs);

3. In some domains (mathematics, for example), the supervision of PhDs is also taken into consideration in evaluating research.

The evaluation committees convened by HCERES are asked to identify excellence among the teams and to compare teams active in the same field of research by ranking them. Each team is usually given one of the following grades: A+, A, B and C.

The main consequence of this evaluation system is the allocation of funding to the research teams by the Ministry of Higher Education and Research and by the research organisations.

11. Germany³⁷

The German funding system is rather complex and precise shares of project vs. institutional funding of R&D are difficult to predict reliably. According to the data collection in the PREF study Germany’s public allocations for publicly performed research is allocated for 64 % and 36 %in the form of organisational level (institutional) funding and project funding respectively.³⁸ These data are in line with Eurostat figures.

The legal basis for the allocation of public funds for R&D is the ‘Freedom for Science’-Article 5(3) of the German constitution (‘Grundgesetz’ GG). Further, rules for joint funding by federal and state governments are laid out in Article 91b GG of the constitution and in the Federal Budget Code (‘Bundeshaushaltsordnung,’ BHO). Article 91b GG of the constitution has been changed based on votes of both chambers of parliament in December 2014. This change enables the federal government to be permanently involved in the funding of universities (Sofka, 2015).

Public funding for R&D has two primary components in Germany: organisational level (block) funding and project funding. Organisational level funding covers essential financial demands of universities in topics or fields for which third-party funding is hardly available, especially fundamental research. In contrast, project funding is directed at a particular goal with typically short to medium-term time horizons (see BMBF, 2012: pp. 53ff for details on ‘how R&D funding in Germany works’) (Sofka, 2015).

The funding of education and research at university is devolved to a large extent to the sixteen states (Länder) which are highly autonomous in matters of education policy. In summary, the Länder’s public funding typically consist of three possible procedures:

An incremental/discretionary/non-competitive part which is mainly based on the previous’ year funding and corrected for inflation. During the past years, this approach to funding has gradually become less important. In order to achieve goals of the state government, like internationalization and gender equality, the state government can financially award well-performing institutions. This type of funding is generally non-competitive (Van Daalen et al, 2014).

Van Dalen et al (2014) provide an overview of how formula based funding developed in 9 Lander over time. They observe an increase over time in the number of Länder that integrate an indicator-based formula into their funding program. Moreover, the individual shares increased as well for most Länder.

This indicator-based part of the annual budget consists of both a teaching and a research component. Typically, the research component carries more weight for (research) universities than for Fachhochschulen, but the exact ratio varies by Länder (e.g. Berlin applies a 50/50 ratio for universities and a 80/20 ratio for Fachhochschulen). The teaching component often consists of the number of students and graduations, whereas the research component is often distributed on the amount of external funding and the number of PhD graduations (Van Daalen et al., 2014).

On top of this, Länder started to implement state-wide pacts and individual target-agreements as a complementary steering instrument. An important thing to note about these target-agreements is that they are not directly linked to financial rewards and/or penalties (Van Daalen, 2014; see also De Boer et al (2015) for further description of the German situation).

The Pact for Research and Innovation (‘Pakt fuer Forschung und Innovation’) has been a major driver for organisational level (institutional) funding for public R&I in Germany in recent years. The initial agreement between federal and Laender governments encompassed the years 2005 to 2010 and was extended in 2009 for the years 2011-2015 (‘Pakt II’). The Pact for Research and Innovation increased the institutional funds annually by 5% for the German research foundation DFG and the main non-university research organisations Max Planck Society (MPG), Fraunhofer Society (FhG), Helmholtz Association (HGF), and Leibniz Association (WGL). Accordingly, institutional funding for these institutions (including funds for implementing the Initiative for Excellence through DFG) is 92% higher in 2015 compared with 2005 (GWK, 2015 in Sofka, forthcoming). R&I stakeholders in Germany, such as the Expert Commission on Research and Innovation (EFI) have welcomed the pact not just for the increase in funds but for planning secure planning coordinates which facilitates strategic decision making (EFI, 2014 in Sofka, forthcoming). Federal and Laender governments have agreed in December 2014 to extend the Pact for Research and Innovation until 2020 with annual budget increases of 3%.³⁹

Part of the pact agreement is the commitment of DFG and the main non-university research organisations to report annually to the Joint Science Conference (‘Gemeinsame Wissenschaftskonferenz’, GWK in Sofka, forthcoming) of federal and Laender governments. These reports culminate in a monitoring report which documents developments and practices (GWK, 2015). An initial stage in the allocation of institutional funding is the allocation across the main research organisations which have distinct R&I profiles and organize the allocation of funds among their member institutes internally (GWK, 2015 in Sofka, forthcoming):

12. Greece⁴⁰

According to the data collection in the PREF study Greece’s public allocations for publicly performed research is allocated for 50 % and 50 % in the form of organisational level (institutional) funding and project funding respectively. Its share of organisational level (institutional) funding is thus comparatively high. These data are in line with Eurostat figures.

Organisational funding in Greece is exclusively block funding and not based on any organisational assessment. There has been hefty opposition to evaluations and accreditations discouraging efforts to link assessments to organisational funding. Therefore, while both HEIs and PROs are subject to evaluations, the evaluation results are not linked to the budget allocation. An effort to link incremental funding to excellence in the ‘90s was abandoned very soon. As salaries are fixed based on formal criteria (rank and seniority) and investments and operational budgets are distributed with equal growth (or reduction) rates on the previous year’s budgets there is no room for rewarding merit and excellence.

13. Hungary⁴¹

Timing and method

Several issues were elaborated and discussed in the past five years by the Hungarian Rector’s Conference and other stakeholders to reform the higher education system and its financing. After the first reading by the government in mid-October 2014, the state secretariat of the Ministry of Human Capacities responsible for higher education published the draft higher education concept entitled “Changing gears in higher education. Directions of performance based development of higher education”. This new concept foresees major changes in the way the HEIs are financed but does not specify how the new funding system, based on more concentrated resource allocation, will support the achievement of international quality standards in HEIs. The foreseen performance indicators of HEIs are: i) cost of teaching by specialisations, ii) cumulated number of credits, iii) increase of the competence level of students, iv) number of diplomas, v) drop-out rates and vi) scientific measures (i.e. number of publications in foreign languages, technology transfer projects, patent applications, PhDs awarded, scientific prizes).

Modality of assessment

The organisational funding allocation in Hungary is mostly based on educational metrics. Government decree 50/2008 (III.14.) regulates the calculation of the institutional funding of HEIs. The basic principles are the following:

• education support that is calculated based on the number of full time students inscribed, the type of education, level of education (i.e. BA, MA, PhD); the norms per student are differentiated by specialisations (e.g. medicine, engineering, humanities).

• research support that is calculated based on the number of teachers and researchers (FTE); number of state financed PhD students (FTE); number of qualified staff out of teachers (FTE); number of teachers that get qualified (i.e. PhD and higher scientific qualifications) in the current and previous two years; and the number of awarded scientific qualifications in the current fiscal year.

• maintenance support that is based on the size of the infrastructure.

Funding levels

The austerity measures had a major impact on the institutional funding of the higher education organisations which decreased by one-third between 2009 and 2013. The total sector got an institutional funding of €424m (HUF123b) in 2013. In 2014 the institutional funding of HEIs remained at the same level as they received a total of €469m (HUF136b). In 2015 the central budget will provide only HUF7b (a mere €23m) more compared to 2014 (a total of HUF143b). It should be noted that the decrease of institutional budgets was quite uneven among the HEIs, as the shrinking of their budget depended on the evolution of the number of students and the specialisation of the respective HEIs. The priority was given to natural science and engineering specialisations, while the social science faculties suffered the largest reductions. At the same time, the HEIs got access to substantial funding from different Operational Programmes of the Structural Funds 2007-2014.

14. Ireland⁴²

Project vs institutional funding

According to the data collection in the PREF study Ireland’s public allocations for publicly performed research is allocated for 32 % and 68 %in the form of organisational level (institutional) funding and project funding respectively. Its share of organisational level (institutional) funding is thus comparatively low. These data are in line with Eurostat figures.

Timing and method

There has been no legislative reform with respect to the legal framework for the allocation of funding for research and development via competitive or institutional funding channels in recent years.

The main element of organisational level funding is the block grant provided to the HEIs by the Higher Education Authority. The block grant covers core teaching and research activities within institutions; it is important to note that the internal allocation of funds between teaching and research is a matter for each institution.

Modality of the assessment

The allocation by the Higher Education Authority of organisational level funding to the HEIs is determined on a formula basis. The allocation is based on a standard per capita amount in respect of weighted EU student numbers in four broad subject price groups. Student numbers in the four groups are weighted to reflect the relative cost of the subject groups.

Five per cent is “top-sliced” from the aggregate grant for all higher education institutions and reallocated on the basis of the number of PhD degrees awarded (75%) and the share of research income by academic staff member (25%) (Forfas, 2013). The research component of the block grant was estimated by Forfás (2013) at €162.5m in 2013.

Funding for Public Research institutes is normally determined on the basis of annual negotiations between the research organisations and their parent Government Department (ministry).

15. Italy⁴³

Project vs institutional funding

According to the data collection in the PREF study Italy’s public allocations for publicly performed research is allocated for 95 % and 5% in the form of organisational level (institutional) funding and project funding respectively. Its share of organisational level (institutional) funding is thus comparatively (very) high.

Timing and method

In 2013, organisational level funding for public HEI and research centres amounted to approx. €60m of which €38m (62%) were allocated to R&D activities and €22m (38%) to administrative activities (Italy Country Report, 2015). The share of institutional funding allocated on the basis of performance criteria rose from 7 % in 2009 to 13.5 % in 2013 and is set to increase further in the coming years (European Commission 2015a).

The Italian research system has traditionally been characterized by an institutional block funding allocation mechanism based on education metrics. Since 2014, important changes have been implemented towards more performance-based funding mechanisms on the basis of a metric informed peer review exercise called VQR which was coordinated by ANVUR and completed in 2013.⁴⁴ The ANVUR was created in 2010 following a law passed in 2006 with the aim to improve the performance of the Italian research system. A previous evaluation of universities, the VTR, for the period 2001-2003 was performed in 2006, but had limited financial consequences in comparison to the UK RAE on which it was modelled: only about 2% since 2009 (Geuna and Piolatto, 2016). The first FFO (ordinary funding plans) introduced in 1993 did allocate a share of funding on input/output measures, though these were initially mainly based on input indicators such as student numbers. Assessments of research outputs have only been introduced in this funding allocation mix in more recent years (Geuna and Piolatto, 2016).

At the end of 2014 the new ordinary funding plan (FFO) for universities was published by the Ministry of Education, Universities and Research (MIUR), and the ‘Stability law’ on budget allocations for 2015 was approved by Parliament (MIUR, 2014). These measures introduce a €150m increase of FFO over 2013 values that however incorporates ‘merit funds’ and other resources that were previously in separate budget lines. At the same time the government spending review cuts €34m from university purchases of goods and services.

Modality pf the assessment

The funding plan of MIUR introduces two new mechanisms for the distribution of funds among universities. First, 20% of the FFO is distributed among universities on the basis of a ‘standard cost’ per student, with a new (but not yet tested) mechanism of resource allocation. Second, 18% of the FFO will go to ‘better performing’ universities, and is distributed in the following way:

- for 70% on the basis of their performance in the ANVUR quality assessment review (ANVUR, 2013);

- for 20% on the basis of their recruiting policies (scientific production of the professors that are recruited or promoted assessed by ANVUR);

- for 10% on the basis of the relevance of international teaching activities, combining the presence of foreign students and the courses followed abroad by local students.

The ANVUR based its assessment on the best research outcomes obtained by each organisations (universities and research institutes in the seven years from 2004 to 2010. The approximately 185,000 publications (of which 70% journal articles) by 130 organisations are evaluated partly by submitting them to international experts (in 20 rather than the previous 14 panels) who appraise their scientific quality, and partly by analysis of the citations received from third parties and examination of the impact they have made in their respective field of research (ANVUR, 2013 and Ancaiani et al, 2015 in Geuna and Piolatto, 2016; Nascia, 2015). Apart from the quality and quantity of submitted output (weighted 50%), the final indicator of unit research quality is calculated also on the basis of the ability to attract external funding. the number of international collaborations, patents registered, the quality of new recruitments and promotions, the number of doctoral students, spinoffs, museums and archaeological sites, third-party activities and performance improvement compared to the VTR 2001-2003 assessment (Nascia, 2015; Geuna and Piolatto, 2016).⁴⁵ A VQR for the period 2011-2014 is currently being prepared.⁴⁶

While peer review was the dominant assessment methods in the social sciences (except economics), arts and humanities and several fields of engineering, bibliometric based approaches (based on journal impact factor and citation measures) were more important in the life and natural sciences, other engineering fields, economics and statistics – even if also in these fields a substantial number of outputs (between 25 and 48%) were peer reviewed – the decision on whether or not to peer review an output was based on whether or not the outcomes of the two bibliometric indicators converged or not (ANVUR, 2013;Geuna and Piolatto, 2016).

According to the European Commission (2015a), effective implementation of the RPBF regimes is made more difficult by the overall decrease in higher education funding and the restrictive rules that limit the yearly change in the amount of funds allocated to each university.

16. Latvia⁴⁷

Project vs institutional funding

According to the data collection in the PREF study Latvia’s public allocations for publicly performed research is allocated for 25 % and 75% in the form of organisational level (institutional) funding and project funding respectively. Its share of organisational level (institutional) funding is thus comparatively low.

Timing and method

According to the Law on Scientific Activity, the state shall allocate base financial resources to state scientific institutions, state institutions of higher education and the scientific institutes of state institutions of higher education that are registered in the Register of Scientific Institutions. The amount is allocated in accordance with the procedures specified by the Cabinet and includes resources for:

• the maintenance of scientific institutions (buildings and equipment maintenance, payment of public utility services, remuneration of administrative and support staff);

• payment to the scientific staff;

• development of the scientific institutes to support them in achieving the aims set out in their operational strategies.

According to the Law, scientific institutes that receive the base financing shall have their activities evaluated every six years. Institutional base financing is calculated and administered by the Ministry of Education and Science based on the data submitted to it annually by the research institutions.

The amount of base research funding that is granted to the research organisations is calculated according to a formula that considers three variables from the previous year: the sum of the average operation costs and research personnel costs multiplied by the development coefficient. The first variable reflects the office space operating costs that are obtained by taking the amount of fixed space (15 m² per each individual employed in research) and multiplying it by the number of FTE researchers, then multiplying it by a standard calculation of the premises’ fixed operation costs and further multiplying it by a coefficient of 2.0 (in the case of natural and engineering sciences) or 1.3 (for social sciences and humanities). The second variable in the formula reflects the number of FTE personnel employed by the organisation in the previous year multiplied by half of a professor’s minimum annual income as set by state regulations. The development coefficient varies in the interval from 1 to 12 and includes the contribution of last year’s outputs, such as peer-reviewed scientific papers in various categories, monographs, patents and doctorates produced, as well as the number of international research and development projects generated.

Large-scale reforms of HEIs and PROs are currently under way to improve the quality and relevance of public R&D. As part of this process, research institutions have been assessed by international experts (in co-operation with the Nordic Council of Ministers and NordForsk, undertaken by Arnold et al (2014)) and the results came out in 2014. According to the assessment, only 15 out of 150 evaluated research institutes and research departments of HEIs received the highest evaluation and were recognized as international players. As a result, in 2013, the Cabinet of Ministers accepted amendments in Regulation No. 1316 “Order of calculation and allocation of institutional funding to research institutes”. The amendments revised the criteria for granting base financing to scientific institutes in order to reduce institutional fragmentation, achieve scientific excellence and ensure that innovation and technology transfer takes place. Namely, changes to provisions on allocating the base financing to scientific institutions took place in 2014. The changes stipulate that additional financing⁴⁸ will be granted to the scientific institutions that have received an evaluation of “4” and “5” in the international assessment of science, while those that have been evaluated with “1” and “2” and do not participate in the consolidation processes of scientific institutions will not receive the base financing as of 2016. Fostering the consolidation process, the scientific institutions with the lowest marks are asked to seek out opportunities to integrate with stronger institutions. The final goal is that only 20 competitive scientific institutions are awarded with base financing in 2020 (as opposed to the 44 that were awarded this funding before the initiative). There is thus a certain performance based element in Latvian base funding of PROs already.

In addition, a new quality-targeting higher education financing model has been developed, based on the recommendations from a recent World Bank study (2014), and performance-based institutional funding was piloted in 2015. Namely, as described in the NRP 2015 of Latvia, a three-pillar financing model was proposed as the most suitable solution. The three pillars are the three funding forms:

1. Base financing (institutional financing to ensure the functioning of education and research);

2. Performance-based financing (financing that is allocated when study outcomes and research results are achieved);

3. Innovation financing (future development-oriented financing that promotes the specialisation of institutions and their profile development).

The second pillar’s (PBF) implementation has been supported by a new budget sub-programme which allocated €5.5m in 2015 and will allocate a further €6.5m in both 2016 and 2017. A set of criteria for the performance-based financing of scientific activity has also been developed. It is expected to direct the financing in order to facilitate an increase in human resources in the research and development of technologies and increase the international competitiveness of Latvian research.

Modality of the assessment

A set of criteria for the performance-based financing of scientific activity (pillar 2) has been developed. The following indicators with across-the-board relevance for universities are considered as part of formulas (Regulation No. 1316 “Order of calculation and allocation of institutional funding to research institutes”): number of research staff FTE (MAs and PhDs), number of graduates, bibliometric indicators (publications) and third party funding of research and teaching (industry funded and international research).

17. Lithuania⁴⁹

Project vs institutional funding

According to the data collection in the PREF study Lithuania’s public allocations for publicly performed research is allocated for 67 % and 33 % in the form of organisational level (institutional) funding and project funding respectively. Its share of organisational level (institutional) funding is thus comparatively high.

Background

In 2009, the Lithuanian Government adopted the decision on the method for allocation of budgetary appropriations for R&D for public higher education and research institutions which stipulated that a higher share of organisational funding should be linked to research performance (this decision was amended in 2010 and 2012). The Decision established that in 2010, 40% and in 2011 and subsequent years, 50% of organisational funding will be allocated to public HEIs on the basis of results of assessments of R&D activities. The remaining 50% as of 2011 are allocated as block funding on the basis of the “normative number of staff” that is approved for each institution by the decree of Minister of Education and Research. The “competitive” half of organisational level funding from 2012 onwards is reallocated every three years taking into consideration the results of assessment of R&D activities.

Modality of the assessment

The ministerial decree adopted in November 2012 stipulates that assessment of R&D activities is based on four criteria:

a) funding received from participation in international research projects;

b) funding received from R&D contracts with business companies;

c) public funding from participation in joint R&D projects with business companies (funding of business subcontracts);

d) results of evaluation of research production.

The latter focuses on publications and patents and covers research production for each year separately. It is carried out every three years by LMT and is based on bibliometric informed peer review. These criteria are given unequal weights for assessment of R&D activities in different fields of science. For example, results of evaluation of research production are given the highest weight in social sciences and humanities (80%) as well as physical and biomedical sciences (55%). Assessment of R&D activities in other fields of science mostly depends on institutions’ capacities to attract funding from privately and internationally funded R&D projects.

It was expected that linking public institutional funding with the capacity to attract additional funding should create incentives for institutions to increase the relevance of their research programmes.

18. Luxembourg⁵⁰

According to the data collection in the PREF study, Luxembourg’s public allocations for publicly performed research is in 2013 allocated for 70 % and 30 % in the form of organisational level (institutional) funding and project funding respectively. These data slightly differ from Eurostat figures, according to which organisational funding represent 82% of research funding and project funding 18%.⁵¹ The share of organisational level (institutional) funding is thus comparatively high.

Luxembourg does not have a system of performance based funding, but does use “Performance Contracts”. These contracts specify the proportions of (third party) project vs organisational level, funding for Luxembourg’s University and PROs. The research funding allocated through the performance contract has been based on the following indicators for the last years:

Table A3. Performance Contract Indicators and Targets

Indicator | 2011-2013 target | 2011-2013 achieved | 2014-2017 target
Number of scientific publications (having an impact factor over 2) | 558 | 775 | 1,040
Number of doctoral theses completed | 91 | 90 | 167
Number of patents applied for | 23 | 34 | 45
Number of spin-offs created | 6 | 4 | 10

Source: MESR

19. Malta⁵²

According to the data collection in the PREF study Malta’s public allocations for publicly performed research is allocated for 99 % and 1 %in the form of organisational level (institutional) funding and project funding respectively. Its share of organisational level (institutional) funding is thus comparatively (very) high.

In Malta the allocation of public research funds for higher educational institutions, government departments and research organisations does not involve performance considerations, certainly not in any formal sense. There exists no algorithm for allocation of public funds, and there is no history of institutional performance assessments to feed into such a mechanism. Funding is allocated primarily on the basis of what was allocated in previous years, and there has been no mention of changing this approach either in the national strategy or in any other document.

20. The Netherlands

According to the data collection in the PREF study The Netherlands’s public allocations for publicly performed research is allocated, in 2013, for 70 % and 30 % in the form of organisational level (institutional) funding and project funding respectively. These data are in line with Eurostat figures.

A study by researchers from the Dutch Central Plan Bureau (CPB) indicates that 23 % of Dutch science funding is based on a performance based funding systems. Roughly half (11%) of this is allocated on research related criteria. According to Hick’s definition (i.e. RPBF is based on ex post research output assessments), the Dutch system would not qualify as a RPBF. However because it does allocate funding on the basis of the number of PhD defences (a research related output), we do include it in the limited RPBF category in this report as was the case for e.g. Austria. It is, however, different in nature from the systems basing their funding decisions on publication output assessments.

Since the implementation of the new Law on Higher Education and Science (MoESC, 2012), in 2012 7% of funding for higher education institutions has been based on performance agreements. This reform replaced an earlier trial of contract funding, during which contracts were signed with the university of applied science sector as a whole (Boer et al, 2015). The latter experience with collective agreements has indicated that they were not sufficiently aligned to the strategic targets of individual universities (Boer et al, 2015). The effect of the new system still needs to be assessed. The association of universities, VSNU, has already indicated in the press that it preferred not to continue the system of performance contracts.

Separate from the funding allocation system, universities are also regularly evaluated at the level of departments/schools. This peer review based research assessment system was implemented in the late 1980 but rather than being linked to university funding, the assessment is used to support the development of national and university strategies. According to Geuna and Piolatto (2016) it does generate a competition for reputation among the university departments. In 2015 a decision was made to change the research output criteria and no longer include the volume of publications as a criterion in evaluations due to concerns over the adverse effects of publication pressure. However funding allocation decisions are not directly tied to these evaluations and this therefore does not classify as performance based funding under Hicks’ definition (2012).

Modality of the assessment

In 2012 Dutch universities and university medical centers received around 3.8 bn euros from the central government. 41 per cent of this organisational level funding is allocated on the basis of education related criteria. 44 per cent is allocated on the basis of research parameters. The remaining 15 per cent is allocated to Academic Medical Centers. The following criteria are used to allocate the education and research part of organisational funding (Van Dalen et al, 2014).

Education funding: (1.6 bn in 2012) funding allocation is based on student numbers and degrees (65%). The remaining 35% of education funding is based on university specific percentages and amounts which are set by the government (Van Dalen et al, 2014).

Research funding: The part of university funding which is allocated on the basis of research parameters includes criteria for degrees (15%), PhD defences (20% or 93 thousand euro per defended PhDs), around 5 % is spend on the funding of (often inter-university) graduate schools. The remaining 60% is allocated on the basis of block funding based on historical considerations, though 2 % is set directly by the government (Van Dalen et al, 2014). In sum the only clear research output variable considered in this formula is the number of PhD defences.

Apart from the research universities, the Netherlands also has a system of universities of applied science. These organisations are not very research intensive (in total these 37 universities spend around 100 million euro on research) and they are funded almost entirely on the basis of education related criteria.

21. Poland⁵³

According to the data collection in the PREF study Poland’s public allocations for publicly performed research is allocated, in 2013, for 40 % and 60 %in the form of organisational level (institutional) funding and project funding respectively.

Poland introduced performance based funding models in 2008. Organisational funding is divided based on the outcomes of nation-wide organisational assessments, using criteria defined by the KEJN (Committee for Evaluation of Scientific Research Institutions) and takes place every four years.

Modality of the assessment:

The detailed assessments include: counts of publications taking into account impact factors of specific academic journals, patents, revenues from industry co-operation and external R&D funding, normalized by numbers of R&D employees of an organization, scientific awards of researchers, number of PhD degrees awarded, patents, and financial results of commercialization of research results. In 2013, the evaluation criteria were substantially modified to further promote organizations conducting world-class research, and the evaluation process is performed using a central IT system POL-on[1] to eliminate the risks of human error or duplication of records for researchers working at more than one scientific organization. The first assessment was carried out in 2013. The assessment in 2017 will be carried out according to updated methodology prepared after consultations with stakeholders. Only a draft of this methodology is currently available[2]. Most important changes stem from including the strategic objectives of the Ministry of Science and Higher Education such as quality, innovation, interdisciplinarity as well as openness and internationalisation of Polish science. More points will be granted for the participation in international research projects with a special stress on Horizon 2020, receiving the HR Excellence in Research logos, the indicator on economic benefits from commercialisation was complemented with the assessment of the usefulness of the solution for the Polish economy. Changes include also the specific publication patterns in the social sciences and humanities

Funding levels:

Based on the outcomes of the evaluation, organizations fall into specific research categories. The best performing research organisations receive + A or A category, good ones B and the least performing C category. The organisational assessments are carried out at the level of separate faculties (not entire universities, i.e. worse-performing parts of a university cannot benefit from successes of other departments/faculties).

These categories and the assigned amount of organisational funding is calculated based on the status and the number of full-time researchers employed by the organization (statutory funding), with dedicated part of funds assigned for young researchers and doctoral students and costs of infrastructure maintenance. The organisational funding is expected to be used for purposes related to research and publication of research results. Beneficiary organizations apply each year for the funding, outlining ongoing research projects which would be supported from the budget, and afterwards report the results accomplished.

A formally defined algorithm determines the level of organisational funding based on: (a) the outcomes of the most recent organisational assessment and (b) the level of funding, which were granted based on previous assessment. However, part (b) of the algorithm has been gradually decreasing since 2010 and disappeared altogether in 2015 with the new ordinance on financing statutory activities of scientific units[3]. In 2014 block funding represented only 77% of the amount received in 2013. Therefore in order to alleviate negative consequences, in the intermediate period (till 2017) there are maximum threshold for increasing or decreasing the funding for a research institute per number of full-time researchers. This is done to avoid to abrupt shocks to the system.

The faculties receive this funding directly from the Ministry of Science and Higher Education. In addition to the Performance Based allocation of research funding, the budget for funding public higher education is also allocated on the basis of a complex funding formula. The latter does include some research performance related variables in order to differentiate between universities. This part of the funding of Higher Education are not analysed in this report.

22. Portugal⁵⁴

According to the data collection in the PREF study Portugal’s public allocations for publicly performed research is allocated, in 2013, for 53 % and 45 % in the form of organisational level (institutional) funding and project funding respectively. These data differ from Eurostat figures, which indicate a ratio of 78% for organisational funding and 22% for project funding. These deviations will be explored further.

Timing and method

In Portugal, the main type of funding for public research is traditionally based on a block-funding mechanism based on education metrics. This segment of the R&I system has however lost its relative importance over the decades. This long-term process has continued more recently, as the funds allocated under this item have declined from €176m in 2011, to €154m in 2013, and to €125m in 2014. The funding of these entities did not depend on efficiency or effectiveness criteria, but it is rather based on tradition and the number of tenured personnel in each higher education institute (RIO Country Report 2014). Evaluations of R&D units were carried out in 1996, 1999, 2002, 2007 and 2013. In the past these evaluations did not have direct effects on the allocation of organisational level funding.

Modality of the assessment

A new funding arrangement, which will be effective from 2015 onwards (and based on the 2013-2015 evaluation⁵⁵), will produce a multi-tiered system of unviersities. This funding arrangement could be classified as a performance based funding system on the basis of a peer review assessment. A total of 322 R&D units were involved in an evaluation exercise aimed to provide funding to research units according to their academic performance and the strategic plan they submitted. The evaluation criteria were the following: Productivity and contribution to the National Scientific and Technological System, Scientific and technological merit of the research team, Scientific merit and innovative nature of the strategic programme, feasibility of work plan and reasonability of the requested budget, impact of the scientific, technological and cultural output. Seven panels, with a total of 73 experts, were in charge of conducting the evaluation.

The research units were classified in 5 categories (outstanding, excellent, very good, good, and fair, poor). The process was carried out in two different phases. 45% of the units evaluated did not pass to the second stage of the evaluation and are expected to receive less than €1k per researcher, per year. The remaining 55% of the 322 research units under evaluation, which integrate 10,992 researchers, making up 62% of the researchers in all the evaluated units, were selected for the second stage of evaluation. The results of the second stage of the evaluation process were disclosed by the Portuguese Foundation for Science and Technology (FCT) in December 2014, with 11 units ranked as ‘Outstanding’, 52 as ‘Excellent’ and 104 as ‘Very Good’.

Funding levels

The funding to be assigned to the research units evaluated in the second stage is about €70m per year. The funding per researcher rises exponentially for the units in the two top levels, with those in the ‘Outstanding’ bracket expected to receive €13m per year (FCT website, https://www.fct.pt/apoios/unidades/avaliacoes/2013/index.phtml.en). The implication of this exponential rise, is that most of the research units will now receive much less funding than before, posing a severe strain on their survival.

23. Romania⁵⁶

Timing and method

The PREF study has not yet completed its data collection for Romania. According to Andreescu and Georghiou around 59% of public funding was allocated institutionally in 2013, increasing from approx. 45% in the period 2009-2012.

Institutional funding is directed towards:

• The programme of the Romanian Academy, having as recipients the institutes of the Romanian Academy. The allocation is highly proportional to the number of researchers (the full procedure is not available on the institution’s website).

• The Nucleu programme, having as recipients the National R&D Institutes. Each institute has its own Nucleu programme (portfolio of projects), reflecting the specific R&D strategy of the institute, also including objectives related to the development of the economic sector/ branch corresponding to the institute’s profile. The funding decision taken by MESR reflects a prioritisation of those specific Nucleu programmes, based on the joint analysis of the previous performance of the institutes and the relevance of their R&D portfolio in relation to the development priorities of the corresponding sector/branch (the full procedure is not available on institution websites).

Institutional evaluation is extensively discussed in Romania, but a coherent system with clear implications for Performance Based Funding has not been implemented yet. There is an evaluation procedure for the “entities part of the national RDI system”, adopted by Government Decision 1062/October 2011. The procedure was applied only to the national R&D institutes (32 in number).⁵⁷

For the period 2014-2020, the National Strategy (2014) attempts to correlate the criteria and amounts for funding going to national R&D institutes, Romanian Academy institutes, and universities. A dedicated programme for institutional funding was included in the draft National RDI Plan 2014-2020, which can de facto integrate the resources now spent through the Nucleu programme and add a similar stream for universities, in correlation with the financing stream of the Romanian Academy. The actual structure of the programme was not available by March 2015 and no action plan has so far been adopted.

Modality of the assessment

In 2013, the evaluation following the adoption of the Government Decision 1062/October 2011 classifies the institutes in four categories –A, B, C, D-, in turn divided in sub-categories (e.g. A+, A, A-). The results of this first evaluation have been criticized for including all the institutes in the “A” category. No information is available regarding the funding implications of this classification.

The Romanian Academy uses a similar methodology for the periodical evaluation of its institutes. The results are not publicly available.

Universities do not receive institutional funding for RDI. However, education metrics are used for the evaluation of the quality of teaching activities. A system based on “quality indicators” (i.e., 17 composite indicators referring to the teaching process, teaching staff, research etc.) is used to determine the funding allocated to universities. While theoretically they could determine up to 30% of the funding allocated, their real impact has been rather limited. (CNFIS 2013).

An alternative model based on “university classification and programme rankings” was implemented in 2011 and 2012, but it was contested by several universities for its lack of transparency (Hotnews 23 December 2013). The “university classification and programme rankings” were also not connected directly to R&D financing, but rather conditioned the right of universities to organise master and doctoral studies.

24. Slovakia⁵⁸

The Slovak Republic uses two types of institutional funding: (1) block funding and (2) performance based institutional funding. According to Hicks (2012 citing Geuna and Martin, 2003; and OECD 2010) the Slovak system is based upon a metrics informed peer review, but this is questioned by other experts which indicate the system is more comparable to the Czech system. Both the Czech and the Slovak systems are deemed to lie at the interface between quantitative metrics based assessments and metrics informed peer review based assessments.

The 2014 formula for allocating performance based funding to universities (€118.5m) gives a 43% weight to the R&D quality of a university (as established by the last accreditation), a 22.5% weight to the share of a HEI in the national research/arts output, a 10% weight to the share of a HEI in the national number of PhD students, a 10% weight to the share of a HEI in domestic grants, a 10% weight to the share of a HEI in foreign grants and the rest is determined by other factors (Balaz, 2015).

A considerable share of the Performance based funding allocation is thus based on the results of evaluations carried out by the Accreditation commission. This Commission approved new, more stringent rules on 15 April 2013. The new rules put more emphasis on research excellence in terms of scientific papers published/cited in journals listed in the international scientific databases (WOS, SCOPUS), research monographs published with high-quality publishers and international patents.

The last evaluation round took place in 2014-2016 and covered period 2008-2013. The Accreditation Commission evaluated 4891 study programmes in 22 Higher Education Institutions (HEIs) by end of 2015. Five HEIs may lose their ‘University’ status and be downgraded to simple ‘Higher Education Institutions’. The Accreditation Commission also did not approve 1756 study programmes, and abolished some 90 dislocated branches of the HEIs out of the total 110⁵⁹.

In April 2104 the Accreditation Commission also launched a pilot project to identify the top research teams in Slovak universities. 77 teams applied for evaluation. The Accreditation Commission used data provided by the Web of Knowledge and identified 37 top teams in 2015. Universities with research teams that are deemed excellent may receive more funding from public resources in the future (Balaz, forthcoming).

25. Slovenia

According to the data collection in the PREF study Slovenia’s public allocations for publicly performed research is allocated for 69 % and 31 %in the form of organisational level (institutional) funding and project funding respectively.

Slovenia uses an ex ante evaluation mechanism for the allocation mechanism of project/institutional funding. The largest share of the basic research is funded through Research Group Programme funding, a system established in 1999 to secure stability in the funding of basic research (Udovic, 2015).

The funding is allocated on the basis of a public call issued by the Slovenian research agency, but since it provides long-term support (up to 6 years) it is more stable than typical research project funding. The SRA indicates this is a clear competitive funding scheme, since it uses periodical calls and the applications are submitted by existing as well as new research groups. The applications (research proposals) are evaluated with assistance of external evaluators (Udovic, 2015).

Since this appears more a centre of excellence approach with ex ante evaluations of proposals, the Slovenian system is not considered to be a Performance Based System under the definition that was applied in this report. (Udovic, 2015).

26. Spain

In Spain, institutional funding is generally allocated in the form of block funding. Research institutions do not receive a variable/competitively allocated organisational level research funding. Organisational level funding of universities is mainly based on education metrics and the payment of salaries for the teachers and researchers (civil servants) (Fernandez-Zubieta, 2015).

From 1987 to 1993, an “incremental” criterion was applied for funding: based on the previous year expenditures and considering changes in teaching and administrative personnel. Hicks (2012) and OECD (2010) consider the Spanish Sexenio programme⁶⁰ as a sort of Performance Based Funding. Since this programme assesses voluntary applications from individual researchers, who when successfully assessed on the basis of among others research output criteria, receive a (modest) salary increase, it is not considered a form of performance based funding in this report which restricts itself to organisational level funding.

The “Severo Ochoa Centres of Excellence” and “María de Maeztu Units of Excellence” Support is a programme of the Secretary of State for Research, Development and Innovation of the Ministry of Economy and Competitiveness that aims to promote excellence in scientific research in Spain. It selects a limited number of existing centres and units that perform cutting-edge research and are among the world’s best in their respective areas. The impact and international scientific leadership of these centres and units are essential criteria. Selection is based on an ex ante evaluation of proposals and though the assessments of these proposals take into account current/past performance it is therefore not considered a RPBF instrument. The evaluation and selection process is based on international peer review by three field specific committees.

Since 1995, regional governments have been responsible for university funding and applied different versions of these mechanisms (Pérez Esparrells & Utrilla de la Hoz, 2008 in Fernandez-Zubieta, 2015). According to the EUA (2015), one of these regions, Catalonia, has implemented a Research Performance Based Funding System.

27. Sweden

According to the data collection in the PREF study Sweden’s public allocations for publicly performed research is allocated for 69 % and 31 % in the form of organisational level (institutional) funding and project funding respectively.

Since the 1990s Sweden has had a research funding system in which a larger share of funding to public research institutions and particularly universities is allocated through competitive means: comprising project funding and organisational level funding linked to organisational assessment. A new system for the allocation and redistribution of the appropriations for research and postgraduate education to the university sector was introduced in Sweden in 2009 (Jacob, 2015).

The proportion of public funding that was annually redistributed along the new model was initially 10%, and increased to 20% from 2014 onwards. The aim of the model is to reward quality in research and give university management the incentives to take measures to increase quality and relevance of the research undertaken at their institutions. This is done by allocating parts of the research appropriations on the basis of quality, based on two indicators: publications/citations, and external funding of research (Jacob, 2015). The assessment of research output is based on a field normalised bibliometric assessment (Kalpazidou Schmidt, 2010; Hicks, 2012).

This system also gives incentives to the universities to cooperate with the private sector since the more external R&D funds they receive from i.e. Swedish firms, the more they receive from the state through the redistribution of appropriations. The impacts are visible: as a result of the introduction of the new system, external funding from foreign firms to Swedish universities, which has been more or less constant up to 2008, starts to grow steadily. Other indicators considered include e.g. gender balance among academic staff (Jacob, 2015).

At present Sweden investigates possible future developments of the resource allocation system, which would rely on peer/expert review as well as performance/output with the surrounding society. In 2013 the Swedish Research Council was given the task to propose a new performance based model on research funding that should include peer review instead of only indicators. As a preparation a report was produced (Quist et al. 2013) which presented an overview of some existing national evaluation systems (UK, all the Nordic countries except for Iceland, New Zealand, Belgium (Flanders), Netherlands). In December 2014 the Swedish Research Council delivered the full proposal to the government named Research Quality Evaluation in Sweden (Forskningskvalitetsutvärdering I Sverige - FOKUS). (Jacob, forthcoming). According to this model proposed by the Swedish Research Council and currently under scrutiny, no data or indicators will directly control resource allocation, but rather all evaluation material will be assessed by panels that set grades and submit explanatory statements for their grading (Jacob, 2015).

In order to translate the evaluation results to resource allocation, the Swedish Research Council proposes that a weighting of 70 % (scientific/artistic quality): 15 % (quality enhancing Factors): 15 % (impact outside academia) is used in the calculation model together with a so-called combined volume measure, taking into account the HEI’s share of the total block grant and each research area’s share of the given HEI’s research staff. The calculations for the three components scientific/artistic quality, quality enhancing factors and impact outside academia will be combined to give the total sum of the performance-based funding allocated to each respective HEI. The HEI is then free to allocate the funds internally as it sees fit (Jacob, 2015).

VINNOVA has also been charged with developing a system for evaluating outreach and impact from universities. VINNOVA intends to present the final proposal in 2016. The Swedish Research Council and VINNOVA have discussed the need for closer coordination between the two proposals (Jacob, Forthcoming).

28. The United Kingdom

Project vs organisational level (institutional) funding of public R&D

According to the data collection in the PREF study the United Kingdom’s public allocations for publicly performed research is allocated for 47 % and 53 %in the form of organisational level (institutional) funding and project funding respectively.

Timing and method

The government’s main tool for funding university research lies in the grants made through the Higher Education Funding Council for England (HEFCE) and its equivalents in Scotland (Scottish Funding Council), Wales (HEFCW) and Northern Ireland. The HEFCE block grant is allocated to HEIs on the basis of a mix of formula and specific allocations. The teaching part of this grant is based on education metrics (number of students), while research performance stands as the main determinant of research funding. The reduction in the teaching part of the grant is compensated, in England but not in e.g. Scotland, by a substantial increase in student fee income for all HEIs. There are no institutional level performance contracts (De Boer et al, 2015).

The UK was the first country to introduce a RPBF system in 1986 with the explicit goal of increasing selectivity in the allocation of public resources (OECD, 2010; Geuna and Piolatto, 2016). Organisational level funding in the UK is nowadays almost always allocated based on organisational assessment. The main stream of support is allocated to the universities in the HE Sector, in the form of a grant from the Higher Education Funding Council for England (HEFCE) and its equivalent bodies in the devolved administrations. This is allocated on the basis of the Research Assessment Exercise – RAE, a peer review process which produces ‘quality profiles’ for each submission of research activity made by HEIs. There were four RAEs (in 1992, 1996, 2001 and 2008) that have been supplanted by the Research Excellence Framework (REF). The REF will be used for funding allocation from 2015 onwards. This performance based organisational level funding represents about 6% of total England HEI income, and 20% of research income from all sources (Cunningham, 2015).

Modality of the assessment

Funding levels for institutions (which are made on a subject oriented ‘cost-centre’ basis and which may apply at a sub-departmental level) are used on an annual basis for the allocation of funding. Each unit is evaluated on its research quality level, which in turn determines performance-based allocations. Initially, units were given a single score. From 2008 onwards units received a profile, based on the percentage of the unit assessed to be at five potential levels: unclassified, 1*) nationally excellent, 2*) internationally excellent, 3*) world class, 4*) world leading (Boer et al, 2015)⁶¹.

A units research output is assessed through a system of peer review. For the 2013 REF, panels were able to use bibliometric data to help inform their decision-making, although every research output (journal article) submitted is assessed on the basis of its intrinsic quality rather than its bibliometric score (Boer et al, 2015). Few top down instructions were provided on the use of these indicators, except for indications not to use e.g. journal impact factors as a proxy for publication quality,and the use the panels made of them was therefore different.⁶²

The first element is the assessment of the research outputs from eligible researchers. Each unit selects its staff members to submit from those staff employed at that university at the census point. A selected staff member can submit up to four publications to which they have contributed in the review period. Each of these is examined by at least 2 panel members (Boer et al, 215).

The second element of the calculation is societal impact. Prior to 2013 the third element covered ‘Esteem Indicators’. One of the main changes implemented in the REF was that in this system each unit also submits an Case Study on societal impact (Boer et al, 2015)

The third element that is assessed is the research environment offered by the unit of assessment. The environment score is currently calculated on the basis both of the number of research students (PhDs) within the department as well as a report in which the unit presents its strategy (Boer et al, 2015)

The overall assessment is based for 65% on the assessment of the quality of the research output, for 20% on societal impact, and for 15% on an assessment of the research environment. These elements are combined into an aggregate score. (Boer et al, 2015)

Funding levels

“The funding weight for research outputs in 2014/15 is calculated on the basis of 3 points per 5% rated as world leading (4*) and 1 point per 5% rated as world class (3*). Units whose profiles falls in the other levels do not receive points tied to additional funding. In order to reflect the differences in expenditures in different fields of research, the scores are weighted by field. Laboratory subjects, including engineering, receive a weighting of 1.6. Subjects such as history receive a weighting of 1.3 whereas e.g. research in the humanities receives a weighting of 1. The total performance based funding an institution receives is calculated as the sum of the total adjusted funding weights of its constituent departments. This institutional funding weight is then used to calculate the share of the £1bn mainstream performance based funding. The remaining funding is distributed separately” (Boer et al, 2015).

29 Norway

According to the data collection in the PREF study Norway’s public allocations for publicly performed research is allocated for 51 % and 49 %in the form of organisational level (institutional) funding and project funding respectively. These data are in line with Eurostat figures.

Norway is another European countries with a long tradition in performance based funding of universities. The analysis below is taken from the RIO Country Report 2015 written by Espen Solberg (Forthcoming). For further information on the historical evolution of the Norwegian system see a.o. Sivertsen (2015), Kalpazidou and Schmidt (2012), Frohlig and Sivertsen (2010) Michelsen & Aamodt (2006).

Higher education

The majority (70%) of institutional funds to HEI are given as block funding. The remaining 30% is distributed on the basis of reported student performance, research performance, and strategic research considerations. Until 2002 most funding of Norwegian higher education institutions was based on input indicators, such as student enrolment and number of staff. Since 2003 a new funding structure has been in place for these funds, which consists of three core components:

1. basic funds which refers to block funding without detailed specifications of its use. This component initially amounted to about 60% of institutional funding (on average for all HEIs), but has now increased to around 70%

2. a teaching component, in which funds are distributed on the basis of reported student performance; this component initially amounted to about one-quarter of institutional funding and has increased somewhat;

3. a research component, which currently stands for 6% of institutional funding. This component is distributed according to performance-criteria related to research activities. More specifically, these performance criteria consist of the following four dimensons:

a. Scientific publications (calculated as “publication points” according to a national standard based on a distinction between two levels of scientific journals – level 1 and level 2)

b. PhD candidates

c. (Competitive) funding from The Research Council of Norway and Regional Research Funds

d. Funding from EU frameworks programmes

Revisions of the funding criteria

In January 2015 an expert committee appointed by the Government presented its analyses and recommendations regarding the future design of public funding to Norwegian higher education institutions (Hægeland et al 2015). The committee did not recommend any radical reforms, but proposed a set of adjustments in the design, weighting and selection of performance based indicators. A central and much debated question was whether the large historically based 70% share of block funding should be subject to a “deconstruction” and allocated according to new criteria and considerations. The committee did not recommend this. Instead, they proposed to introduce a new element of contract based financing, which was suggested to account for 5% of total HEI funding.

In its budget proposal for 2016, the Government has followed up parts of the suggestions put forward by the committee. As from 2016, the performance indicators will be modified while maintaining the main design and structure of the system. The recent adjustments consist mainly of:

- a stronger focus on obtained grades in the indicators for student completion (instead of merely rewarding student points in the existing system)

- technical adjustments in the indicators for scientific publications in order to remove disincentives for scientific co-publishing

- incentives for EU R&D funding will be extended to include all sorts of EU-funding, including education programmes

- A new incentive for rewarding external funding and income from contract research will be introduced in order to increase the institutions’ interaction with industry and public sector

In addition, the Government announces an intention to introduce scientific citations as a supplementary indicator for stimulating research quality.

Research institutes

As mentioned above, a new system for institutional funding of research institutes was implemented in 2009. Part of the basic institutional funding is now based on a set of performance indicators. However, the basic funding only account for 10-15 per cent of total funding for most institutes. Furthermore, the share of basic funding which is subject to performance based allocation varies between 2.5% for primary sector institutes, 5% for environmental institutes and 10% for social science institutes and technical industrial institutes.

As a consequence, the result based funding at stake only account for a few percentages of the institutes’ total budget. Nevertheless, it seems that the performance based component has a real effect on priorities and strategies for the management of the 50 research institutes which are subject to the system (DAMVAD, 2012). The current system includes the following four performance based indicators:

- Scientific publications

- Total income from contract research

- Funding from international sources (EU and beyond)

- Number of completed PhDs (hosted and/or co-funded by the institute)

Funding of Health trusts/hospitals

In 2004, following a major organizational reform of the Norwegian Health care sector, a new system of performance research funding was introduced. The system builds largely on the system that was introduced in the higher education sector a few years before (see above).

The current system is based on two main funding streams:

1) An earmarked basic funding which is more or less evenly split between five Regional health authorities (representing app. the five geographical NUTS 2 regions). The exception is the South-East region, comprising the Oslo area, which receives 2/5 of the basic funding, due to its relative size and importance. This basic funding accounts for 40% of total funding to the Health trusts/hospitals.

2) The remaining 60% is distributed according to a performance based system, where scientific publications and completed PhDs constitute the main indicators

FRIDA

The Norwegian Performance Based assessment system depends on a National Research information system (FRIDA) which aims to collect data on publications (including journal articles, books etc), the CV’s of scientific authors, funding information etc. Norway was a pioneer in setting up such a system and may be an example for other countries. More information can be found in various publications and presentations by Gunnar Sivertsen (e.g.: 2009, 2015).

30 Switzerland

According to the data collection in the PREF study Switzerland’s public allocations for publicly performed research is allocated for 28 % and 72 %in the form of organisational level (institutional) funding and project funding respectively. These data are in line with Eurostat figures.

Switzerland is one of the countries with a Binary university system in which a considerable share of research funding is concentrated in the Federal Institutes of Technology. There is no formal RPBF regime in place in Switzerland but use is made of performance contracts. Also institutional funding allocation by the regional and federal level is tied to input indicators (numbers of students and / or acquired project funding). The text below is taken from the Swiss RIO Country Report 2015 drafted by Lepori et al (forthcoming).

The organization of the Swiss higher education system leads to a differentiated pattern of funding by type of HEIs (FITs, Cantonal universities, Universities of Applied Sciences). While competition is relatively limited, this structure is effective in differentiating level of funding and funding conditions by type of HEIs. This approach is consistent with the new Higher Education Coordination and Funding act, which explicitly states that levels of funding (and the relative balance between research and education) can differ depending on HEI mission.

It should be remarked that institutional funding of HEIs is in most cases jointly attributed for research and education, while the decision on the effective use of resources is left to the HEIs themselves.

a) Federal Institutes of Technology. FITs are funded by the Confederation through a block grant, which is linked to a performance contract between the Confederation and the ETH Rat. These are rather soft arrangements, but FITs are expected to maintain their position among the best reputed international universities and are therefore subject to strong quality pressures.

b) Cantonal universities. Cantonal universities are mostly funded by their home Canton and co-financed by the Confederation. The Cantonal core grant is usually negotiated directly between the Canton and the University, a performance contract having been introduced in a number of cases. Overall, this grant has a strong historical component for what concerns R&D, whereas student’s numbers are important for educational allocation (co-funding by other Cantons is based on a fixed rate by student). Federal co-funding is based on a formula based on the number of students (70%) and the acquired project funds (30%). The share of different funding sources varies rather strongly by university.

c) Universities of Applied Sciences are mostly financed by the Cantons: the core grant includes a (formula-based) educational component and a negotiated research component, which is however much lower than for universities. Federal funding is purely student-based. Therefore, UAS have a much lower level of institutional research allocation than Universities and are expected to fund most of their R&D from project funds.

This allocation model has been quite stable since the late ’90s.

31 Iceland

Funding for universities and research institutions in Iceland mainly takes the form of block funding (around 80%), with only general provisions as to the way it is to be spent. Governmental appropriations to research institutions are generally not based on comprehensive policy on budgetary targets, but rather on specific needs of institutions and industries. Funding is allocated primarily on the basis of what was allocated in previous years.⁶³

32 United States of America

According to Van Daalen et al (2014), Federal support to research in Higher Education Institutions in the US is by and large allocated in the form of project funding. This would leave little room for allocating institutional funding on the basis of a Performance Based Regime. The state governments do provide institutional funding to the colleges and universities that constitute their Higher Education Systems.

Historically the US states provide institutional funding in relation to the number of full time (equivalent) students the universities enroll. This model has changed in many states to incentivize universities to help students to complete degree programmes (i.e. in many states it now also considers degrees awarded). Increasingly states are including criteria that are tied to the goals and priorities of the states higher education policy makers in the funding allocation mix.

“Thirty-two states—Arizona, Arkansas, Colorado, Florida, Illinois, Indiana, Kansas, Louisiana, Maine, Massachusetts, Michigan, Minnesota, Mississippi, Missouri, Montana, New Mexico, New York, Nevada, North Carolina, North Dakota, Ohio, Oklahoma, Oregon, Pennsylvania, South Dakota, Tennessee, Texas, Utah, Virginia, Washington, Wisconsin, and Wyoming—have a funding formula or policy in place to allocate a portion of funding based on (primarily education) performance indicators such as course completion, time to degree, transfer rates, the number of degrees awarded, or the number of low-income and minority graduates. Five additional states—Connecticut, Georgia, Iowa, South Dakota, and Vermont—are currently transitioning to some type of performance funding, meaning the Legislature or governing board has approved a performance funding program and the details are currently being worked out.”⁶⁴

In most of the US states the funding allocation formula adopted to distribute funding to HEI are thus based on quantitative input and education indicators, potentially tied to the strategic objectives of the states government such as addressing labour market shortages for STEM students or for promoting the share of disadvantaged students.

We do not consider these funding allocation mechanisms to fall under the Research Performance Based Funding programmes adopted in this project: research funding allocation decisions are (at least partially) based on an ex post evaluation of research output (Hicks, 2012).

Most of the information for this paragraph was taken from the NCSL site⁶⁵ which provides an overview of the funding allocation system in place in the different US states university systems. No information is provided on this site for the university of California system, which is important considering the large role it plays in the research output of the US system. Also in the university of California system institutional funding is mainly tied to student numbers.⁶⁶

33 Mexico

Mexico does not have a system of Research Performance Based Funding allocation. Its public funding allocation constitutes a mix of institutional block funding and project funding. As in the case of some Southern European and Central and Eastern European Countries it is characterised by a large role for the National Council of Science and Technology which allocates funding primarily to its own research institutes.

In Mexico, funding for R&I comes predominantly from two sources: the National Council of Science and Technology (CONACYT) and the Sectoral Ministries, such as Public Education, Energy, Agriculture, Rural Development, Fisheries and Food, Health and Social Security.⁶⁷ The CONACYT budget constitutes approximately 45% of the total Government R&I budget. In 2015, the total budget of CONACYT was 54% higher than in 2012, reaching €1,911.40m (MXN 33,706.75m) in 2015. In 2014 a reengineering process in some of the competitive programmes was finalised. The distribution between project and institutional funding barely changed in the period.⁶⁸ Project funding amounted to 31.61% of total funding in 2011, to 34.48% in 2012 and to 34.58% in 2013.

CONACYT’s budget is divided into budget allocated to CONACYT (about 80% of the total) and budget allocated to the CONACYT centres (about 20% of the total). Budget allocated to CONACYT is then split between its own operational budget (between 3% and 4% of the total) and project funding (also called substantive programmes). The competitive R&I funding is used to finance R&I projects and individual grants. It takes the largest part of CONACYT’s budget (about 77% of total funding) and in 2015 it was 50% higher than in 2012.

34 Australia

The Australian Research Council has been involved in Research Performance Based Funding for well over a decade (Butler, 2003). The system in use has undergone various iterations. From a simple bibliometrics approach, a ranked journal listing approach similar to Denmark and Norway was developed. This was then replaced in the new Excellence in Research for Australia (ERA) framework in 2012[4]. The ERA adopts a metrics based peer review assessment of Fields of Research within an institution, to come to an assessment of all research outputs produced in Australian universities vis a vis national and international benchmarks. To this end, the ARC has developed discipline specific indicators together with the respective scientific stakeholder community.

One of the objectives of the ERA is to inform funding allocation decisions for the Sustainable Research Excellence (SRE) block grants. The first full round of ERA occurred in 2010 and the results were published in early 2011. This was followed by two rounds in 2012 and 2015 respectively. In its National Innovation and Science Agenda (2015) the Australian government announced the introduction of impact and engagement assessments to assess the benefits flowing from university research. Such impact assessment indicators will be developed, again in consultation with the respective stakeholder community, by the ARC in the course of 2016. Following a pilot assessment in 2017, national level assessment and reporting will take place in 2018.

35. China

The elite level of the Chinese research system, responsible for most high impact publications, consists on the one hand of institutes from the Chinese Academy of Science and on the other of research universities. Universities had, at the start of the reform process in the early 1980s, hardly any role in research (Liu and White, 2001) but since the mid-1980s the Chinese government has started a programme to bolster their research capabilities. It did this for example through the set-up of State Key Laboratories - perhaps comparable to the centres of excellence set up in many European member states, which received some designated funding to bolster their research capabilities, attract researchers from abroad and provide training to researchers from other Chinese organisations (see e.g. Jin et al, 2005).

Still in the 2000s, organisational level (block) funding was more important for the institutes of the Chinese academy of Science than for universities. In the Knowledge Innovation Programme of the Chinese Academy of Science a selected group of institutes received additional organisational level funding to upgrade their research capabilities. At the same time a number of institutes were closed and/or merged (Suttmeier, Cao and Simon 2007; Jonkers, 2010). The “CAS Innovation 2020” initiative builds on the Knowledge Innovation Programme since 2011.

With the aim to upgrade the research capabilities of Chinese universities the government launched two programmes. The 211 and the 985 programme. Through the 211 Project or “High-level Universities and Key Disciplinary Fields” programme 107 [currently 116] universities were allocated each around 400 million RMB to improve teaching, learning and research (Ma, 2007 in Zhang et al, 2013). Local and provincial governments also invested in the selected universities as part of this programme (Ministry of Education, 2007 in Zhang et al, 2013). In total almost 37 bn RMB was invested in these universities in the period 1995-2005 (Zhang et al, 2013) and the programme has continued since.

The 985 or “World Class Universities” programme is also a partnership between the central government (ministry of education and other ministries with their own universities) and local governments. The latter were supposed to match central government funding. In the first phase of the 985 programme (1999-2003) the 14 billion RMB was allocated to [9]⁶⁹ universities and in phase II (until 2007) 39 universities received 18.9 billion RMB (Ministry of Education 2011 in Zhang et al, 2013). The third phase is from 2010-2015, reflecting the integration of the 985 programme in the five year plans (Luo, 2014). Not all universities receive the same amount of funding. In the first phase, the two elite universities Beijing University and Tsinghua University each received 1.8 billion RMB whereas the others received between 0.1 and 0.6 billion. In the third phase Beijing and Tsinghua university received 4 billion RMB, whereas the other universities from the top nine received 1.5-2.64 billion RMB respectively. The more recent 985 universities received less (Luo, 2014). The 985 universities are concentrated in the eastern regions and especially Beijing, Shanghai, Guangzhou and the capital cities of eastern provinces. That there are still seven universities in Western and Central China might be a reflection of the aim to also build up capacities in these parts (Zhang et al, 2013).

China does not have a Research Performance Based Funding. While it is true that the original 985 universities are the universities which publish most and most high impact publications, levels of funding are not thought to be directly tied to assessments of output. The choice to select universities in e.g. the 985 programme is partially political as suggested by the geographical spread. Nonetheless the 985 programme and to a lesser extent the 211 programme, is clearly meant to concentrate resources in selected organisations which is also one of the aims of Research Performance Based Funding. Rather than RPBF the two (large scale) programmes could be seen as an alternative strategy to increase concentration among universities in an attempt to create world class research universities. This approach has led to the development of a binary or rather multi-tier system of universities. The Shanghai Jiaotong university ranking was set up partially to monitor developments in the performance of Chinese universities and to benchmark them with the leading universities in the world. At the sub-organisational level, State Key Laboratories and Open Laboratories in universities and CAS institutes are subject to periodical evaluations by the NSFC. These assessments can lead to the closure or upgrading of these Laboratories (Jin et al, 2005).

References

Abramo, G., Cicero, T., D’Angelo, C.A, 2012. The dispersion of research performance within and between universities as a potential indicator of the competitive intensity in higher education systems. Journal of Infometrics, vol.6, 2, 155-168.

Adams, J., & Gurney, K. (2010). Funding Selectivity, Concentration and Excellence: How Good is the UK’s Research?. Oxford: Higher Education Policy Institute.

Aghion, P., Dewatripont, M., Hoxby, C., Mas-Colell, A., and Sapir, A. (2010). The Governance and Performance of Universities: Evidence from Europe and the US. Economic Policy, (January), 7–59.

Alexander, S., 2015. RIO Country Report 2014: Luxembourg, Editor: European Commission, Joint Research Centre, Institute for Prospective Technological Studies, Publications Office of the European Union.

Andersen, LB., & Thomas Pallesen, T., “Not Just for the Money?” How Financial Incentives Affect the Number of Publications at Danish Research Institutions, International Public Management Journal Volume 11, Issue 1, 2008, pages 28-47

ANVUR, 2013. Rapporto sullo stato del sistema universitario e della ricerca. Available at: http://www.anvur.org/attachments/article/644/Rapporto%20ANVUR%202013_UNIVERSITA%20e%20RICERCA_integrale.pdf

Arnold, E., Knee, P., Angelis, J., Giarraca, F., Grinice , E., Javorka, Z., Reid, A., 2014, Latvia Innovation System Review and Research Assessment Exercise – final report, http://izm.gov.lv/images/zinatne/ZISI/Latvia-systems-review_2014.pdf last accessed 8-10-2015

Auranen, O., Nieminen, M., 2010. University research funding and publication performance- an international comparison. Research Policy, 39, 6, 822-834

Avotins, V., 2015. RIO Country Report 2014: Latvia, Editor: European Commission, Joint Research Centre, Institute for Prospective Technological Studies, Publications Office of the European Union.

Balaz, V., 2015. RIO Country Report 2014: Slovakia, Editor: European Commission, Joint Research Centre, Institute for Prospective Technological Studies, Publications Office of the European Union.

Bitard, P., 2015. RIO Country Report 2014: France, Editor: European Commission, Joint Research Centre, Institute for Prospective Technological Studies, Publications Office of the European Union.

Bleiklie, I., & Michelsen, S., 2015. Political-Administrative Regimes and University Governance, http://ecpr.eu/Filestore/PaperProposal/48e6c9c2-7460-43a9-aaf7-557bdb35e4cf.pdf last accessed 8-10-2015

Braun, D., 2003. Lasting tensions in research policy-making – a delegation problem. Science and Public Policy, 30, pp. 309–321.

Butler, 2003. Explaining Australia’s increased share of ISI publications – the effects of a funding formula based on publication counts, Research Policy, 32 (2003), pp. 143–155

Butler, L., 2010 Impacts of performance-based research funding systems: A review of the concerns and the evidence page 127–165 in OECD (2010), Performance-Based Funding for Public Research in Tertiary Education Institutions, Workshop Proceedings, OECD: Paris.:http://www.oecd-ilibrary.org/education/performance-based-funding-for-public-research-in-tertiary-education-institutions/impacts-of-performance-based-research-funding-systems-a-review-of-the-concerns-and-the-evidence_9789264094611-7-en last accessed 8-10-2015

CRDI (2013b) Methodology of Evaluation of Research Organizations and Evaluation of Finished Programmes (valid for years 2013 - 2015), http://www.vyzkum.cz/FrontClanek.aspx?idsekce=695512, accessed on 16/04/2015.

Claeys-Kulik, A.-L., Estermann, T., 2015. Define Thematic Report: Performance-based funding of universities in Europe. European University Association

CNFIS (2013), Starea finantarii invatamantului superior si masurile de optimizare ce se impun (The state of higher education financing and the measures of optimization to address the challenges), available at http://www.cnfis.ro/wpcontent/uploads/2012/08/CNFIS-Raport-public2013-final.pdf.

CRDI, 2014. Návrh výdajů státního rozpočtu České republiky na výzkum, experimentální vývoj a inovace na rok 2015 s výhledem na léta 2016 a 2017, http://vyzkum.cz/FrontClanek.aspx?idsekce=677284, accessed on 16/04/2015.

Cruz-Castro, L., Bleda, M., Derrick, GE., Jonkers, K., Martinez, C., and Sanz-Menendez, L., 2011. Issue brief.: Public sector research funding. https://www.innovationpolicyplatform.org/sites/default/files/Public%20Sector%20Research%20Funding_0_0.pdf

Cruz-Castro, L., Jonkers, K., Sanz Menendez, L., 2016, The internationalisation of European Public Research Organisations, in Lewin and Nedeva, M., Towards European Science, Springer

Cruz-Castro, L., Sanz-Menendez, L., 2007, Research Evaluation in Transition, individual versus organisational assessment in Spain in The Changing Governance of the Sciences Volume 26 of the series Sociology of the Sciences Yearbook pp 205-223

Cunningham, P., 2015. RIO Country Report 2014: UK, Editor: European Commission, Joint Research Centre, Institute for Prospective Technological Studies, Publications Office of the European Union.

Cuntz, A., 2015. RIO Country Report 2014: Austria, Editor: European Commission, Joint Research Centre, Institute for Prospective Technological Studies, Publications Office of the European Union.

De Boer, HF., Jongbloed, B., Benneworth, P., Cremonini, L., Kolster, R., Kottmann, A., Lemmens-Krug, K., Vossensteyn, H., 2015. Performance-based funding and performance agreements in fourteen higher education systems - Report for the Ministry of Education, Culture and Science, Enschede: CHEPS, University of Twente https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0ahUKEwiC95apyejLAhUChSwKHev2CIcQFgghMAA&url=https%3A%2F%2Fwww.rijksoverheid.nl%2Fbinaries%2Frijksoverheid%2Fdocumenten%2Frapporten%2F2015%2F03%2F01%2Fperformance-based-funding-and-performance-agreements-in-fourteen-higher-education-systems%2Fperformance-based-funding-and-performance-agreements-in-fourteen-higher-education-systems.pdf&usg=AFQjCNHY9PufYaEvEGqej9y1Q8vj3VHBjQ&bvm=bv.118353311,d.bGg&cad=rja

Debackere, K., Glänzel, W., 2004, Using a bibliometric approach to support research policy making: The case of the Flemish BOF-key, Scientometrics, Volume 59, Issue 2, pp 253-276

DeBackere, K., Veugelers, R., (eds) 2015. Vlaams indicatorenboek 2013, https://www.ecoom.be/sites/ecoom.be/files/downloads/indicatorenboek2013.pdf last accessed 8-10-2015

Den Hertog, P., 2015. RIO Country Report 2014: Netherlands, Editor: European Commission, Joint Research Centre, Institute for Prospective Technological Studies, Publications Office of the European Union.

Derrick, GE., and Pavone, V., 2013, Democratising research evaluation: Achieving greater public engagement with bibliometrics-informed peer review, Science and Public Policy, 40 (5): 563-575.

Domagoj, R, 2015. RIO Country Report 2014: Croatia, Editor: European Commission, Joint Research Centre, Institute for Prospective Technological Studies, Publications Office of the European Union.

Dora, 2012. The San Francisco Declaration on Research Assessment (DORA), http://www.ascb.org/dora/

Dory, T, 2015. RIO Country Report 2014: Hungary, Editor: European Commission, Joint Research Centre, Institute for Prospective Technological Studies, Publications Office of the European Union.

Duchêne, V., 2015. RIO Country Report 2014: Belgium, Editor: European Commission, Joint Research Centre, Institute for Prospective Technological Studies, Publications Office of the European Union.

Edquist, C & Johnson, B., Institutions and Organisations in Systems of Innovation, Chapter 2 in Edquist, C. (Ed.), 1997. Systems of Innovation: Technologies, Institutions and Organizations. London: Pinter Publishers/Cassell Academic, 432 pp

ERAC Peer Review of the Spanish Research and innovation System, 2014.

Estermann, T., Bennetot Pruvot, E., Claeys-Kulik, A.-L., 2013. Designing strategies for efficient funding of higher education institutions in Europe. European University Association.

European Commission, 2010. Assessing Europe’s University-Based Research. EUR 24187 EN, Science in Society 2008 Capacities, 1.4.1. European Commission, Brussels.

European Commission. (2011). Supporting growth and jobs–An agenda for the modernisation of Europe’s higher education systems. COM (2011), 567.

European Commission, 2012. Communication from the commission to the European parliament, the council, the European economic and social committee and the committee of the regions a reinforced European research area partnership for excellence and growth. Com/2012/0392 final, available at http://eurlex.europa.eu/legal-content/EN/TXT/?uri=CELEX:52012DC0392 last accessed 8-10-2015

European Commission, 2015a. Staff Working Document, Country Report Italy 2015.

European Commission 2015b, Staff working document, country report Latvia 2015

Allspach-Kiechel, V., Kancewicz-Hoffman, J., Walter, N., White, O., Worms. JC., , 2015, ESF Report on the Evaluation Process of the Foundation for Science and Technology (FCT) R&D Units, European Science Foundation https://www.fct.pt/apoios/unidades/avaliacoes/2013/docs/ESF_FCT_report_Final.pdf

FCT, 2015, FCT Evaluation of R&D Units: https://www.fct.pt/apoios/unidades/avaliacoes/2013/

Fernández Zubieta, A., 2015. RIO Country Report 2014: Spain, Editor: European Commission, Joint Research Centre, Institute for Prospective Technological Studies, Publications Office of the European Union.

Forfás (2013), State Investment in Research & Development 2012-2013, Dublin, Ireland. Available from: http://www.tara.tcd.ie/bitstream/handle/2262/70200/18122013_State_Investment_in_RD_2012-13-Publication.pdf;sequence=1 Accessed 18 February 2015.

Fosse Hansen, H., 2009. Research Evaluation: Methods, Practice, and Experience. Danish Agency for Science, Technology and Innovation (downloaded on June 9, 2011

Frølich, N. (2008). The politics of steering by numbers: debating performance-based funding in Europe. NIFU STEP.

Geuna, A., 2001. The Changing Rationale for European University Research Funding: Are There Negative Unintended Consequences? Journal of Economic Issues, 35, 3, pp. 607-632.

Geuna, A., Martin, B.R., 2003. University research evaluation and funding: an international comparison. Minerva 41, 277–304.

Geuna, A., Piolatto, M., 2016, Research assessment in the UK and Italy: Costly and difficult, but probably worth it (at least for a while), Research Policy, Volume 45, Issue 1, Pages 260–271

Gheorghiu, R.C., 2015. RIO Country Report 2014: Romania, Editor: European Commission, Joint Research Centre, Institute for Prospective Technological Studies, Publications Office of the European Union.

Good, B., Vermeulen, N., Tiefenthaler, B., Arnold, E., 2015. Counting quality? The Czech performance-based research funding system, Research Evaluation doi: 10.1093/reseval/rvu035

Gläser, J., Laudel G., Hinze, S., Butler, L., 2002, Impact of evaluation-based funding on the production of scientific knowledge: What to worry about, and how to find out http://s3.amazonaws.com/academia.edu.documents/30993064/02ExpertiseGlaeLauHinBut.pdf?AWSAccessKeyId=AKIAJ56TQJRTWSMTNPEA&Expires=1459865530&Signature=QkucBae1Iq4iHLnQRcJJrxRQwoE%3D&response-content-disposition=inline%3B%20filename%3DImpact_of_evaluation-based_funding_on_th.pdf

Grimpe, C., 2015. RIO Country Report 2014: Denmark, Editor: European Commission, Joint Research Centre, Institute for Prospective Technological Studies, Publications Office of the European Union.

Hackett, E., Peer review and social science research funding, in ISSC/UNESCO World Social Science Report 2010, Paris: UNESCO Press.

Hazelkorn, E., 2010. Pros and cons of research assessment, in ISSC/UNESCO World Social Science Report 2010, Paris: UNESCO Press

HEFCE, 2015. Review of the UK’s interdisciplinary research http://www.hefce.ac.uk/pubs/rereports/Year/2015/interdisc/Title,104883,en.html last accessed 8-10-2015

Herbst, M., 2007. Financing Public Universities. Higher Education Dynamics, vol. 18. Springer.

Hicks, D, 2009. “Evolving Systems of Multi-University Research Evaluation”, Higher Education 57(4): 393-404.

Hicks, D., 2012. Performance-based university research funding systems, Research Policy, Volume 41, Issue 2, 2012, Pages 251–261

Hicks, D., Wang, J., 2009. Towards a Bibliometric Database for the Social Sciences and Humanities—A European Scoping Project. Final Report on Project for the European Science Foundation

Hicks, D., Wouters, P., Waltman, L., De Rijcke, S., Rafols, I, 2015. Bibliometrics: The Leiden Manifesto for research metrics, Nature, 22 April 2015

Ingwersen, P., Larsen, B., 2014, Influence of a performance indicator on Danish research production and citation impact 2000-12, Scientometrics, 101: 1325-1344

Jin,BH, Rousseau, R., Sun, XX., 2005, Key labs and open labs in the Chinese scientific research system: qualitative and quantitative evaluation indicators, Research Evaluation (2005) 14 (2): 103-109.

Jacob, M, 2015. RIO Country Report 2014: Sweden, Editor: European Commission, Joint Research Centre, Institute for Prospective Technological Studies, Publications Office of the European Union.

Jongbloed, B., Lepori, B., Huisman, J., Boer, D. H., Dill, D. D., & Souto-Otero, M. (2015). The funding of research in higher education: mixed models and mixed results. The Palgrave International Handbook of Higher Education Policy and Governance, 439.

Jonkers, K., 2010, Mobility, Migration and the Chinese Scientific Research System, London: Routledge

Kalpazidou-Schmidt, E. (2010). Les systèmes nordiques d’enseignement supérieur dans l’espace européen de l’enseignement supérieur et de la recherche. Education et sociétés, (2), 45-62.

Kalpazidou Schmidt, E., 2012. University reform in the nordic countries, http://www.zam.go.jp/n00/pdf/nf010007.pdf

Kalpazidou Schmidt, E., Langberg, K., Aagraard, K., 2006. Funding Systems and Their Effects on Higher Education Systems COUNTRY STUDY – DENMARK, PAris OECD

Klincewicz, K, 2015. RIO Country Report 2014: Poland, Editor: European Commission, Joint Research Centre, Institute for Prospective Technological Studies, Publications Office of the European Union.

Krücken, G., Meier, F., 2006. Turning the university into an organizational actor. In In: Globalization and Organization: World Society and Organizational Change, edited by Gili Drori, John Meyer, and Hokyu Hwang. Oxford, Oxford University Press, 2006, 241-257.

Kyvik, S., Structural Changes in Higher Education Systems in Western Europe, Higher Education in Europe, Vol. XXIX, No. 3, October 2004

Lepori, B., van den Besselaar, Dinges, van der Meulen, Potì, Reale, Slipersaeter & Theves, 2007a. Indicators for comparative analysis of public project funding: concepts, implementation and evaluation. Research Evaluation, 16(4), 243-255.

Lepori, B., (forthcoming). RIO Country Report 2015: Switzerland, Editor: European Commission, Joint Research Centre, Institute for Prospective Technological Studies, Publications Office of the European Union.

Lepori, B., Benninghoff, M., Jongbloed, B., Salerno, C. & Slipersaeter, S., 2007b. Changing models and patterns of higher education funding: Some empirical evidence. In A. Bonaccorsi & C. Daraio (Eds.) Universities and Strategic Knowledge Creation. Specialization and Performance in Europe (pp. 85-111). Bodmin, Cornwall: MPG Books Limited

Liu, XL, White, S, Comparing innovation systems: a framework and application to China’s transitional context, Research Policy, Volume 30, Issue 7, August 2001, Pages 1091–1114.

Luo, Y., Building World-Class Universities in China in Jung Cheol Shin and Barbara M. Keim (eds) Institutionalization of World Class University in Global Competition, pp 165-184 Springer Verlag, https://books.google.be/books?id=CbjIORBaL-QC&pg=PA172&lpg=PA172&dq=985+china+phase+III&source=bl&ots=5qy2ce1hoe&sig=BlROqxZswbj4BEv4qrNyP_5fDvE&hl=nl&sa=X&redir_esc=y#v=onepage&q=985&f=false

Malek, J., Hudeckova, V., Matejka, M., 2014. System of evaluation of research institutions in the Czech Republic. Procedia Computer Science 33: 315-320

Marginson, S., 1997. Steering from a distance: power relations in Australian higher education. Higher Education 34, 63–80.

Martin B., 2011. The Research Excellence Framework and the impact agenda, are we creating a Frankenstein Monster?, Research Evaluation, 20(3) 247-254

Martin, Ben R and Richard Whitley 2010. The UK Research Assessment Exercise: a case of regulatory capture? In Reconfiguring Knowledge Production: Changing Authority Relationships in the Sciences and their Consequences for Intellectual Innovation, eds R Whitley, J Gläser and L Engwall, pp. 51–80. Oxford: Oxford University Press.

Martin, T, 2015. RIO Country Report 2014: Ireland, Editor: European Commission, Joint Research Centre, Institute for Prospective Technological Studies, Publications Office of the European Union.

MINECO, 2015, Apoyo y acreditación de Centros de Excelencia «Severo Ochoa» y a Unidades de Excelencia «María de Maeztu», http://www.idi.mineco.gob.es/portal/site/MICINN/menuitem.7eeac5cd345b4f34f09dfd1001432ea0/?vgnextoid=cba733a6368c2310VgnVCM1000001d04140aRCRD last accessed 8-10-2015

Ministry of Education and science (2014). Information report “On the Latvian science structural reform implementation until July 1, 2015 http://tap.mk.gov.lv/mk/tap/?pid=40324956 last accessed 8-10-2015

Ministry of Education Flanders, 2015, Besluit van de Vlaamse Regering betreffende de financiering van de Bijzondere Onderzoeksfondsen aan de universiteiten in de Vlaamse Gemeenschap, http://data-onderwijs.vlaanderen.be/edulex/document.aspx?docid=14492#24 last accessed 13-10-2015

MinOC&W, 2012, Wet Kwaliteit in verscheidenheid hoger onderwijs, http://wetten.overheid.nl/BWBR0033693/geldigheidsdatum_08-10-2015

Mira Godinho, M., Corado Simoes, V., 2015. RIO Country Report 2014: Portugal, Editor: European Commission, Joint Research Centre, Institute for Prospective Technological Studies, Publications Office of the European Union.

MIUR, Decreto ministeriale 4 novembre 2014, n.815. Decreto criteri di ripartizione del Fondo di finanziamento ordinario (FFO) delle università per l’anno 2014. L. 23/12/2014 n.190 Disposizioni per la formazione del bilancio annuale e pluriennale dello Stato (Legge di stabilita’ 2015).

Moed, H. F. (2008). UK Research Assessment Exercises: Informed judgments on research quality or quantity?. Scientometrics, 74(1), 153-161

Nascia, L., 2015. RIO Country Report 2014: Italy, Editor: European Commission, Joint Research Centre, Institute for Prospective Technological Studies, Publications Office of the European Union.

North D., 1990. Institutions, Institutional Change and Economic Performance, Part of Political Economy of Institutions and Decisions, Cambridge University Press

OECD, 2010. Performance-Based Funding for Public Research in Tertiary Education Institutions, Workshop Proceedings, OECD: Paris.

Osuna, C, Cruz-Castro, L, and Sanz-Menéndez, L 2011, Overturning some assumptions about the effects of evaluation systems on publication performance, Scientometrics 86:3, 575-592

Oswald, 2007. An examination of the reliability of prestigious scholarly journals: evidence and implications for decision-makers, Economica, 74, pp. 21–31

Paliokaite, A., 2015. RIO Country Report 2014: Lithuania, Editor: European Commission, Joint Research Centre, Institute for Prospective Technological Studies, Publications Office of the European Union.

Račić, D., Švarc, J., 2015. RIO Country Report 2014: Croatia, Editor: European Commission, Joint Research Centre, Institute for Prospective Technological Studies, Publications Office of the European Union.

Radosevic, S., Lepori, B., 2009. Public research funding systems in central and eastern Europe: between excellence and relevance: introduction to special section, Science and Public Policy, 36 (9): 659-666

Rafols, I., Leydesdorff, L., O’Hare, A., Nightingale, P., Stirling, A., 2012. How journal rankings can suppress interdisciplinarity. A comparison between innovation studies and business & management, Research Policy Volume 41, Issue 7, September 2012, Pages 1262–1282

Reale, E., Lepori, B., Nedeva, M., Thomas, D. Primeri, E., Chassagneux, E. and Laredo, P. (2013) Understanding the Dynamics of Research Funding Collaboration in the European Research Area. JOREP Final Report, European Commission: Luxembourg. ISBN 978-92-79-29661-1 Doi 10.2777/10945ResearchItaly, 2015 https://www.researchitaly.it/en/understanding/overview/assessment/ downloaded 08/10/2015 last accessed 8-10-2015

Rinia, E.J., Van Leeuwen, T.N., Van Buren, T.N., Van Raan, T.N. Influence of interdisciplinarity on peer-review and bibliometric evaluations in physics research, Research Policy, 30 (3) (2001), pp. 357–361

Ruttas-Küttim, R., 2015. RIO Country Report 2014: Estonia, Editor: European Commission, Joint Research Centre, Institute for Prospective Technological Studies, Publications Office of the European Union.

Saarnivaara, V.-P., 2015. RIO Country Report 2014: Finland, Editor: European Commission, Joint Research Centre, Institute for Prospective Technological Studies, Publications Office of the European Union.

Sandström, U., Heyman, U., van den Besselaar, P., 2014. The Complex Relationship between Competitive Funding and Performance. In Ed Noyons (ed), Context counts: Pathways to master big and little data - STI, Leiden: CWTS. pp 523-533

Science Metrix 2015. data related to the study “Analysis and regular update of bibliometric indicators” financed under FP7 (tender N° 2009/S 158-229751) finalized by Science Metrix in 2015. http://ec.europa.eu/research/innovation-union/index_en.cfm?pg=other-studies

Sivertsen, 2009, A Bibliometric Funding Model based on a National Research Information System, http://www.issi2009.org/agendas/issiprogram/public/documents/ISSI%202009%20Sivertsen%20Vista-100730.pdf downloaded 08/10/2015

Sivertsen, G., & Schneider J. W. (2012). Evaluering av den bibliometriske forskningsindikator. Oslo: NIFU. 75 p. (Rapport 17/2012, in Norwegian).

Sivertsen, G., 2015. NIFU Norway, The integration of bibliographic data, institutional information systems and national data sources for the purpose of funding and evaluation. The latest developments in the Scandinavian countries, STI conference Lugano

Sofka, W., 2015. RIO Country Report 2014: Germany, Editor: European Commission, Joint Research Centre, Institute for Prospective Technological Studies, Publications Office of the European Union.

Soete, L.,, Cowey, L., Langfeldt, L., O’Carroll, C., Preissler, S., Gaczynski, M., Garcia, CE,, Henriques, L., Mahr, A., Sorčan, S., 2015. Peer Review of the Bulgarian Research and Innovation system, Horizon 2020 Policy Support Facility, DG Research and Innovation, Luxembourg: Publications Office of the European Union.

Sörlin, S. (2007), “Funding Diversity: Performance-based Funding Regimes as Drivers of Differentiation in Higher Education Systems”, Higher Education Policy 20, 413-440 (1 December 2007) | doi:10.1057/palgrave.hep.8300165.

Spruyt E., Engels, T., 2013. Het nieuwe BOF-besluit en de implicaties ervan voor de universitaire onderzoeksfinanciering en -beleidsvoering, in: Tijdschrift voor onderwijsrecht en onderwijsbeleid - ISSN 0778-0443, p. 55-67

Srholec, M., 2015. RIO Country Report 2014: Czech Republic, Editor: European Commission, Joint Research Centre, Institute for Prospective Technological Studies, Publications Office of the European Union.

Steen, J. v., 2012. “Modes of Public Funding of Research and Development: Towards Internationally Comparable Indicators”, OECD Science, Technology and Industry Working Papers, 2012/04, OECD Publishing. http://dx.doi.org/10.1787/5k98ssns1gzs-en.

Suttmeier, P., Cao, C., Simon, F., 2006, China’s Innovation Challenge and the Remaking of the Chinese Academy of Sciences, Innovations Vol 1, No3 pp 78-97

Talib, D., Stelee, A., 2000 The Research Assessment Exercise: strategies and trade-offs, Higher Education Quarterly, 54, pp. 68–87

Tapper, T., Salter, B., 2003. Interpreting the process of change in higher education: the case of the Research Assessment Exercises. Higher Education Quarterly 57, 4–23.

Taylor, J., 2011, The assessment of research quality in UK. Universities: peer review or metrics? British Journal of Management, 22 (2011), pp. 202–217

Todeva, E., 2015. RIO Country Report 2014: Bulgaria, Editor: European Commission, Joint Research Centre, Institute for Prospective Technological Studies, Publications Office of the European Union.

Todorova, A., Pérez, S.E., 2015. Stairways to Excellence Country Report Bulgaria. Editor: European Commission, Joint Research Centre, Institute for Prospective Technological Studies, Publications Office of the European Union.

Tsipouri, L., 2015. RIO Country Report 2014: Greece, Editor: European Commission, Joint Research Centre, Institute for Prospective Technological Studies, Publications Office of the European Union.

Udovic, B., 2015. RIO Country Report 2014: Slovenia, Editor: European Commission, Joint Research Centre, Institute for Prospective Technological Studies, Publications Office of the European Union.

UEFISCDI, CNFIS, 2013. Starea finanţării învăţământului superior şi măsurile de optimizare ce se impun (The currest state of the higher education financing system and the optimization challanges), http://www.cnfis.ro/wp-content/uploads/2012/08/CNFIS-Raport-public2013-final.pdf.

Van Daalen, R., Mehmood, S., Verstraten, P., van der Wiel, K., 2014. Public funding of science: An international comparison’, CPB Background document, Netherlands Bureau for Economic Policy Development, The Hague

Warrington, B., 2015. RIO Country Report 2014: Malta, Editor: European Commission, Joint Research Centre, Institute for Prospective Technological Studies, Publications Office of the European Union.

Whitley, R., 2000. The intellectual and social organisation of the sciences (second ed.)Oxford Universidy Press, Oxford

Whitley, R., 2007, Changing governance of the Public Sciences, The Consequences of establishing research evaluation systems for knowledge production in different countries and different fields in Whitley, R., Glaser J (Eds) The Changing Governance of Science, Dordrecht: Springer

Wilsdon, J., Allen, L., Belfiore, E., Campbell, P., Curry, S., Hill, S., Jones, R., Kain, R., Kerridge, S., Thelwall, M., Tinkler, J., Viney, I., Wouters, P., Hill, J., Johnson, B., The Metric Tide - the independent review of the role of metris in research assessment and management, London: Sage

Wilsdon, J., Allen, L., Belfiore, E., Campbell, P., Curry, S., Hill, S., Jones, R., Kain, R., Kerridge, S., Thelwall, M., Tinkler, J., Viney, I., Wouters, P., Hill, J., Johnson, B., 2015. The Metric Tide, Report of the independent review committee on the role of metrics in research assessment and research management http://www.hefce.ac.uk/news/newsarchive/2015/Name,104464,en.html

World Bank. 2013, Input for Bulgaria’s Research and Innovation Strategies for Smart Specialization

World Bank, 2014. Higher Education Financing in Latvia: Final Report. http://viaa.gov.lv/lat/izglitibas_petijumi/petijums_ai_finansesana/ last accessed 8-10-2015

Zhang, H., Patton, D., Kenney, M., 2013, Building global-class universities: Assessing the impact of the 985 Project, Research Policy, Volume 42, Issue 3, April 2013, Pages 765–775

List of figures

Figure 1. Research Performance-Based Funding Systems

Figure 2 Evolution of MS share of the EU publication output 2000-2006-2013

Figure 3. Share of publications among top 10% most cited (field weighted)

List of tables

Table 1. The share of project vs organisational level funding

Table 2. Dominant assessment approach for the allocation of Performance Based funding in the EU28

Table 3. Overview of Performance Based Funding Systems in several EU member states

Table 4. Funding evolution in the MS (PPS 2005 prices) (Eurostat 2015)

Table A1. Table A1. Parameters included in the calculation of BOF allocation formula

Table A2.

Table A3. Performance Contract Indicators and Targets

Notes

¹ https://rio.jrc.ec.europa.eu/

² Public research centres are public research organisations which play an important role in the performance of scientific research, PRCs include the academies of sciences in the Central and Eastern European countries and the research councils in Italy, Spain and France (Cruz Castro et al, 2011)

³ In case a RPBF provides incentives on output indicators only, researchers can be incentivised to “slice up” their results and publish several papers rather than a single one.

⁴ Since organisational level funding tends to be allocated on the basis of a set of institutions (rules) an alternative argument could be made for retaining this label. In case this is done, however, the definition should be adapted and it would be a challenge to distinguish it from project funding which is also allocated within an institutional framework.

⁵ Academic stakeholder organisations argue that RPBF should, instead of being sliced from the core funding, should include additional funding (EUA, 2015). This approach is not always followed by the Member States.

⁶ Latvia has recently introduced a performance based funding system which is described in more detail in the annex. It is not yet reflected in the overview table.

⁷ Institutional funding of universities in Spain is a regional competence. There is at least one region (Catalonia) which has introduced some form of Performance Based Funding system.

⁸ It is relevant to note that some countries do not distinguish between R&D budgets of higher education organisations and the education budgets in the calculation of HERD as prescribed by the Frascati manual. In Poland, neither scholarships for doctoral students nor the salaries of academics working in HEI or PROs are considered in HERD/GOVERD. The R&D expenditures in these sectors therefore appears relatively small in comparison to other countries where this is done. This may also affect the relative share of project and institutional funding though the exact nature of this effect is unclear as the inclusion of doctoral grants would increase project funding whereas the inclusion of salary expenditures would increase institutional funding. Salaries and scholarship are paid from a different budget line which is not so strictly linked to performance as the part of the research budget linked to RPBF.

⁹ see annex 1 for brief reports of the EU Member States as well as Switzerland, Norway, Australia, Mexico and the US

¹⁰ See e.g. http://www.nilf.no/publikasjoner/Notater/2012/n201204hele.pdf

¹¹ Most systems use education metrics in the allocation of higher education funding to universities. What is referred to in table 2B-2E refers to systems which (partially) use education metrics to also allocate research funding.

¹² Spain is a special case as at the national level it uses individual level assessment on the basis of self-reported data, to give researchers a (modest) variable bonus to their fixed salaries (Osuna et al, 2011; Cruz-Castro and Sanz, 2007). This system is not used to allocate competitive organisational level funding and we therefore did not include Spain as one of the RPBF systems. However, higher education funding in Spain is a regional level competence and it is possible that some regions have implemented RPBF type funding systems: a recent report from the European University Association indicates that Catalonia has used performance based funding to allocate organisational level funding to universities but it is unclear whether this is research or education based.

¹³ Peer review is used more in assessing the publication output in the social sciences and humanities. In the natural science fields the assessments are mainly based on journal (impact factor) based bibliometrics

¹⁴ The French Ministry of Higher Education and Research established the AERES National Agency in 2008 to undertake, at national level, the evaluation of research in both research organisations and universities. This organisation has been replaced in 2013 by the High Council of the Evaluation of Research and Higher Education (HCERES).

¹⁵ https://www.researchitaly.it/en/understanding/overview/assessment/

¹⁶ A number of other countries are developing approaches to assess broader societal impact, though this is not necessarily tied to the allocation of funding at present see e.g. https://www.rathenau.nl/nl/publicatie/evaluating-societal-relevance-academic-research-guide.

¹⁷ Not all academics are selected by the host university to submit to the peer review assessments. Those who are not are deemed “research inactive” a somewhat negative and not necessarily accurate term. (Comment by Paul Cunningham, 2016)

¹⁸ “One could argue that it is not the sector as a whole, but only the selected ‘research active’ members of the academic community. Thus, the system is open to a certain amount of gaming as institutions seek to maximise their REF return rates.” (Comment by Paul Cunningham, 2016)

¹⁹ These evaluations are not inconsequential even if they do not have a direct link to funding

²⁰ This might also be due to the (non standardised and undirected) use of bibliometrics by review panels (Derrick and Pavone, 2013)

²¹ The new to be introduced Czech system would instead be based on a five yearly assessment.

²² Jo Johnson, the UK Minister for Universities and Science, has appointed in December 2015 a committee under the chairmanship of Lord Stern to review the way Britain allocates its funding for university research. The intention is to explore among others to which extent metrics can replace peer review to make the process less costly https://www.gov.uk/government/news/government-launches-review-to-improve-university-research-funding.

²³ For the data on project vs organisational level (institutional) funding please check table 1 for methodological notes.

²⁴ Based on Cuntz, 2015.

²⁵ Note that research funding through the Hercules scheme will be administered by the FWO from 2016 onwards

²⁶ See http://www.gallilex.cfwb.be/document/pdf/40180_000.pdf (last consulted 10/2015).

²⁷ http://data-onderwijs.vlaanderen.be/edulex/document.aspx?docid=14492#24

²⁸ Source: http://data-onderwijs.vlaanderen.be/edulex/document.aspx?docid=14492#24

²⁹ Based on Todeva, 2015 and Todorova (forthcoming)

³⁰ Based on Racic & Svarc (2015)

³¹ Based on Tsipouri, 2015.

³² http://ufm.dk/uddannelse-og-institutioner/videregaende-uddannelse/universiteter/styring-og-ansvar/udviklingskontrakter

³³ Partially based on Ruttas-Küttim (2015).

³⁴ Amendments valid from 01.01.2005.

³⁵ This model applies to the government funding for universities. In addition to this, each university has its own unique funding model for internal monitoring & allocation of funds. Furthermore, each university has its own set of indicators e.g. for strategic monitoring component. https://www.google.fi/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=0ahUKEwiqgOiwm9vJAhWGECwKHTv3Dy8QFggdMAA&url=https%3A%2F%2Fwww.tek.fi%2Ffi%2Fcmis%2Fbrowser%3Fid%3Dworkspace%253A%2F%2FSpacesStore%2F6c6e50c6-81b3-4b1d-b50b-0c0bae5eab3e%26filename%3Dcmisattachments%2FTutkimustoiminnan%2520vaikuttavuus%2520yliopistojen%2520rahoitusmallissa_selvitys.pdf&usg=AFQjCNGDuWXwAD6aLLJbEktu0ZG9FbhHrQ&sig2=3NKuazm4dOy9blZb3qfcHg

³⁶ This section is partially based on European Commission (2010).

³⁷ This section is partially based on Sofka (2015) and Sofka (forthcoming).

³⁸ Note that regional funding which plays an important role in institutional funding of universities in Germany may not have been fully taken into account yet in the reported figure.

³⁹ http://www.gwk-bonn.de/fileadmin/Papers/PFI-III-2016-2020.pdf (8/2015)

⁴⁰ Based on Tsipouri (2015)

⁴¹ Based on Drory, 2015

⁴² Based on Martin (2015)

⁴³ Partially based on Nascia (2015)

⁴⁴ http://www.anvur.org/index.php?q=schema-dm-vqr-definitivo&lang=it

⁴⁵ https://www.researchitaly.it/en/research-evaluation

⁴⁶ http://www.tara.tcd.ie/bitstream/handle/2262/70200/18122013_State_Investment_in_RD_2012-13-Publication.pdf;sequence=1

⁴⁷ Partially based on Kulikovskis and Stamenov (forthcoming)

⁴⁸ The government provided (on limited competition basis) an additional €9.9 m of funds to support excellent institutes to develop their strategy and to integrate weaker institutions until November 2015. The reform also entails an increase by 10% of the calculated basic infrastructure grant to those research institutions which received an evaluation score 4 or 5 (i.e. are among excellent science organisations) since 1 January 2015 (Avotins, 2015).

⁴⁹ Based on Paliokaite (2015)

⁵⁰ Based on Alexander (2015)

⁵¹ This discrepancy will be checked.

⁵² Based on Warrington (2015)

⁵³ Based on Klincewicz, 2015. See also Soete et al, 2015.

⁵⁴ Based on Mira Godinho & Corado Simoes (2015)

⁵⁵ The FCT (2013) assessment was evaluated by a panel coordinated by the European Science Foundation (Allspach-Kiechel et al, 2015).:

⁵⁶ Based on Gheorghiu (2015) and Andreescu & Georghiou (forthcoming)

⁵⁷ The Advisory Board for Research and Development and Innovation began evaluating research institutes based on new rules, contained in the Government Decision no. 97/2016. Source: http://cursdeguvernare.ro/a-inceput-evaluarea-institutelor-de-cercetare-pe-baza-noilor-norme-cine-nu-obtine-nota-3-pierde-certificarea.html (article published on 14 March 2016).

⁵⁸ This annex is partially based on Balasz (2015. Forthcoming)

⁵⁹ http://spravy.pravda.sk/domace/clanok/365842-pat-slovenskych-univerzit-moze-prist-o-titul/

⁶⁰ See Osuna et al (2011) and Cruz-Castro and Menendez, 2007 for more detailed analyses of this programme.

⁶¹ The following page provides the exact definitions of the quality levels, the definition of impact (reach and significance) and the criteria for assessing research environment http://www.ref.ac.uk/panels/assessmentcriteriaandleveldefinitions/

⁶² According to personal communication with an analyst of the role of metrics in research evaluation exercises

⁶³ This annex is based on Hulda Skogland’s forthcoming RIO country report on Iceland.

⁶⁴ http://www.ncsl.org/research/education/performance-funding.aspx

⁶⁵ http://www.ncsl.org/research/education/performance-funding.aspx

⁶⁶ https://www.insidehighered.com/news/2013/01/31/university-california-rethinks-how-it-funds-campuses

⁶⁷ Includes Communication and Transport, as well as the General Attorney of the Republic

⁶⁸ Note that data on project vs institutional funding for Mexico is not necessarily compatible with the information collected through the PREF study.

⁶⁹ Zhang et al (2013) report that 34 universities took part in the first phase, most other sources indicate that it was the top 9 universities.

[1] http://polon.nauka.gov.pl/ankieta-jednostki

[2] http://www.bip.nauka.gov.pl/g2/oryginal/2015_06/edd7e651544a4961aab9a394eaea8e1b.pdf

[3] http://www.dziennikustaw.gov.pl/du/2014/1941/1

[4] http://www.arc.gov.au/era-faqs