THE CONCEPT OF THE IMPLEMENTATION OF THE PERFORMANCE-BASED RESEARCH FUNDING TOOL

Topicality. The implementation of the Performance-based research funding (PBRF) in EU Member States has been a priority for the development of research and innovation in last time. This financial mechanism refers to the type of competitive organizational and institutional allocation of research funding.Aim and tasks. The aim of the article is to develop the theoretical basis of Performance-Based Research Funding (PBRF) assessment to increase the effectiveness of scientific research. Also the aim of the article to analyze the implementation of this mechanism in in Ukraine.Research results. Given the formation of Academic Freedom in the article, it will be clarified: the structure of the evaluation of the financing of research based on the results, the constellations of scientific activities, the category of research activities, and the hypotheses concerning the effectiveness of the national research evaluation systems (NRES). The taxonomy of the indicators of the efficiency of subsidiary activity in Ukraine, New Zealand, Sweden is considered.The following factors explain efficiency differences between science systems: The level of competition; Share of project funding; Performance based funding systems; National evaluation systems.The following activities are excluded from the definition of research except where they are used primarily for the support, or as part, of research and experimental development activities: preparation for teaching; the provision of advice or opinion, except where it is consistent with the PBRF’s Definition of research; scientific and technical information services; general purpose or routine data collection; standardisation and routine testing (but not including standards development); feasibility studies (except into research and experimental development projects); specialised routine medical care; the commercial, legal and administrative aspects of patenting, copyrighting or licensing activities.Conclusions. Theoretical basis of Performance-Based Research Funding (PBRF) assessment to increase the effectiveness of scientific research in Ukraine is developed. The Performance-based research funding is governed by the following set of principles: comprehensiveness; respect for academic traditions; consistency; continuity; differentiation; credibility; efficiency; transparency; complementarity. The Methodology for evaluating the effectiveness of scientific institutions of the National Academy of Sciences of Ukraine, in our opinion, can be supplemented by 3 elements: Moderation Panel. Comprehensive peer-review panel and PBRF audit.

РУБЕЛЬ О.Є.д-р екон.наук, с.н.с.ст.наук.співроб.Інститут проблем ринку та економіко-екологічних досліджень НАНУ Французький бульвар, 29, м.Одеса, Україна, 65044 Е-mail: rubeloleg@gmail.comORCID: 0000-0001-5104-5602 Problem statement and its connection with important scientific and practical tasks.The implementation of the Performance-based research funding (PBRF) in EU Member States has been a priority for the development of research and innovation in last time.This financial mechanism refers to the type of competitive organizational and institutional allocation of research funding, based on an ex post evaluation of research results.This approach differs from other approaches to the financing of scientific activities, in which funding is mainly provided on a non-competitive basis or competitive project financing, for which the allocation of funds is carried out in a preliminary (ex ante) assessment mode.
Analysis of recent publications on the problem.This topic is deeply studied in the applied developments of specialists from leading foreign scientific centers in Europe, Asia and America.In particular, the foundations of scientometrics were developed by F. Galton, J. Bernal [1], G. Holton, D. Price [2], its various aspects were covered also by Y. Garfield, S. Bradford, A. Pritchard, L. Resing, D. Hawkins.Quantitative analysis is taken from expert reports of the Joint Research Center (JRC) of the European Commissionn.The reports are based on the basis of national case studies and academic academic literature.[3].Qualitative analysis complemented by quantitative data and analysis of the material.Performance-based financing mechanisms are one of the main ways to include elements of competition to allocate of funding at the institutional level.The report of European Commission (2011) "Supporting growth and jobs -An agenda for the modernisation of Europe's higher education systems" recommends an increase in financing mechanisms related to efficiency that introduce an element of competition [5].Source: [19, p. 365-384] The above definition of autonomy shows that institutional autonomy generally means the autonomy of the university from the state, and this leads to an increased power of the universities management.Larger power of top management is often related to the introduction of forms of NPM leading to a decrease of the autonomy of the academic staff, which is also obvious in the EUA report.Several authors have argued that this may lead to less creative and more middle of the road and risk-avoiding research.This research will less often result in top cited papers, and therefore lowers the performance and the efficiency of the institution.Data on academic freedom are hardly available, but we will address this issue also in the empirical part, using the few data we could find.For the purposes of the PBRF, research is original investigation undertaken in order to contribute to knowledge and understanding and, in the case of some disciplines, cultural innovation or aesthetic refinement.
It typically involves enquiry of an experimental or critical nature driven by hypotheses or intellectual positions capable of rigorous assessment by experts in a given discipline.
It is an independent, creative, cumulative and often long-term activity conducted by people with specialist knowledge about the theories, methods and information concerning their field of enquiry.Its findings must be open to scrutiny and formal evaluation by others in the field, and this may be achieved through publication or public presentation.In some disciplines, the investigation and its results may be embodied in the form of artistic works, designs or performances.
Research includes contribution to the intellectual infrastructure of subjects and disciplines (for example, dictionaries and scholarly editions).It also includes the experimental development of design or construction solutions, as well as investigation that leads to new or substantially improved materials, devices, products or processes.
The following activities are excluded from the definition of research except where they are used primarily for the support, or as part, of research and experimental development activities: -preparation for teaching; -the provision of advice or opinion, except where it is consistent with the PBRF's Definition of research; -scientific and technical information services; -general purpose or routine data collection; -standardisation and routine testing (but not including standards development); -feasibility studies (except into research and experimental development projects); -specialised routine medical care; -the commercial, legal and administrative aspects of patenting, copyrighting or licensing activities.
Research and development (RD) or research and technological development (RTD), refers to innovative activities undertaken by corporations or governments in developing new services or products, or improving existing services or products [11].

National evaluation system
Competitive project funding Academic Authonomy The PBRF Quality Evaluation is an assessment of research quality in our degree-delivering tertiary education organisations (TEOs) -universities, institutes of technology and polytechnics, wānanga, and private training establishmentsfor the purpose of determining the allocation of government funding.
Research produced within the tertiary education sector enables TEOs to play an important role in the creation, application and dissemination of knowledgecrucial ingredients for a knowledge-based economy and society.The PBRF Quality Evaluation assessment encourages high-quality research at our TEOs.Dynamic research cultures underpin and enhance degree-level learning, particularly at the postgraduate level.
The government's current aims for the PBRF are to: -increase the quality of research; -ensure that research continues to support degree and postgraduate teaching; -ensure that funding is available for postgraduate students and new researchers; -improve the quality of public information about research outputs; -prevent undue concentration of funding that would undermine research support for all degrees or prevent access to the system by new researchers; -underpin the existing research strengths in the tertiary education sector.
The PBRF is governed by the following set of principles: Comprehensiveness.(The PBRF should appropriately measure the quality of the full range of original investigative activity that occurs within the sector, regardless of its type, form, or place of output).Respect for academic traditions.(The PBRF should operate in a manner that is consistent with academic freedom and institutional autonomy).Consistency.(Evaluations of quality made through the PBRF should be consistent across the different subject areas and in the calibration of quality ratings against international standards of excellence).Continuity.(Changes to the PBRF process should only be made where they can bring demonstrable improvements that outweigh the cost of implementing).Differentiation.(The PBRF should allow stakeholders and the government to differentiate between providers and their units on the basis of their relative quality).Credibility.(The methodology, format and processes employed in the PBRF must be credible to those being assessed).Efficiency.(Administrative and compliance costs should be kept to the minimum consistent with a robust and credible process).Transparency.(Decisions and decision-making processes must be explained openly, except where there is a need to preserve confidentiality and privacy).Complementarity.(The PBRF should be integrated with new and existing policies, such as charters and profiles, and quality assurance systems for degrees and degree providers).Cultural inclusiveness.
The qualitative approach relies on information collected through a network of national experts in R&D contracted by the Joint Research Centre (JRC) of the European Commission.The reports provided were supplemented by material from national policy documents and analyses published in the academic literature.The resulting national case studies were reviewed by National Contact Points in the national administrations of each EU Member State (Jonkers and Zacharewicz, 2016) [3].
Budgetary Allocations for R&D, but provides a more fine-grained disaggregation by mode of allocation, managing organisations, and funding flows to performers.
While we base ourselves largely on the van Steen (2012) definitions, which are also employed by, we make an adaptation in the label of organisational level funding which is generally referred to as institutional

Develop and Test
Implement funding [17].The reason for doing so is that "institutions" refer to "rules, norms, habits and ways of doing things" as distinct from organisations.We therefore refer to organisational level instead of institutional funding.
In order to analyse the relationship between RPBF systems and university autonomy we consider the strategic actor-hood of universities and make a distinction between "internal autonomy" and "external autonomy" (Whitley, 2007; Cruz-Castro, Jonkers, & Sanz Menendez, 2016) [28][29].The analysis of organisational actorhood is more compatible with the understanding of universities as organisations rather than institutions.Since the 1980s the importance of project funding has increased in many European Member States.The other traditional of providing public support to research is through organisational level funding of universities and public research organisations (Fig. 3).

Fig. 3. Performance-based financing mechanisms: institutional approaches
Source: author's development Public research funding is generally allocated in two main ways, through project funding and through organisational level funding.Project funding is defined as "the total of national budgets in a given country, attributed to a group or an individual to perform an R&D activity limited in scope, budget and time, normally on the basis of the submission of a project proposal describing the research activities to be done."(Van Steen, 2012) [17].
Organisational level funding is defined as the total of national budgets in a given country, attributed to a research performing organisation, with no direct selection of R&D project or programmes and for which money the organisation has more or less freedom to define the research activities to be performed.To a large extent this block funding may be earmarked for particular expenditures such as infrastructure or researcher's salaries, especially in research systems where permanent researchers are civil servants (Cruz Castro et al, 2010) [29].It is this latter type of competitively allocated organisational level funding which we consider as performance based funding.As indicated in the introduction, many EU member states have implemented some form of performance based funding over the past decade and the share of organisational level funding which is allocated competitively on the basis of performance assessments has increased.
Analysts and policy makers have used different understandings of Performance Based Funding.It is therefore useful to further define what we mean in this paper by this funding allocation system by building on the definition developed by Diana Hicks (2012) [7].
Doing so will allow us to assess whether Member States have implemented a research performance based funding system for the allocation of organisational research funding.In order to be considered as RPBF, Member States funding allocation systems must have the following characteristics: Since organisational level funding tends to be allocated on the basis of a set of institutions an alternative argument could be made for 141 retaining this label.In case this is done, however, the definition should be adapted and it would be a challenge to distinguish it from project funding which is also allocated within an institutional framework.
The PBRF is a mixed-performance assessment; it employs both peer-review processes and performance measures.There three elements: periodic Quality Evaluationsthe assessment of the research performance of eligible TEO staff, undertaken by expert peer-review panels; postgraduate research degree completions (RDC) measurethe number of postgraduate research-based degrees completed in participating TEOs, assessed on an annual basis; an external research income (ERI) measurethe amount of income for research purposes received by participating TEOs from external sources, assessed on an annual basis.
Naturally, the development of a classification of the use of research indicators requires as a data source some kind of inventory of the various types of use to which indicators have been put.The literature contains no recent comprehensive overview of indicator use.The classification (fig.4) has fore main categories of use are identified: general science policy, funding allocation, management and organization, content decisions, and consumer information.Policy formulation use of research indicators as a source of information to support decision making, policy development, and the setting of policy goals.
Policy evaluation use of research indicators to evaluate policies or programs.The results of evaluations can in themselves feed back into policy formulation.
Inducement use of research indicators to create performance incentives that are not purely or primarily intended to have another function such as solving a budget allocation problem.
Block funding gives recipient institutions discretion to spend funding according to their own views.We include block funding with some general constraints on the purpose for which it is used (constrained block funding).Two types of block funding are distinguished.
Formula-based block funding use of research indicators as a variable in a funding formula next to other variables such as faculty and number of graduates Non-formula block funding use of research indicators without formulae, but, e.g., in negotiating contracts on which funding is based.
Earning of extra funding by institutions or individuals based on their research performance.A financial bonus or additional funding comes on top of basic funding and is not granted to everyone but only to those eligible.Financial bonuses or penalties can also result from contractual agreements which contain indicators as targets.
Program and project funding use of research indicators to decide about the funding of research programs or projects on the basis of project proposals.In some systems, indicators play an immediate role in the decisionmaking, while in other decision-making processes indicators are used to inform peer reviewers or decision makers: Internal funding use of research indicators for funding allocation within institutions.Internal funding can reflect external funding mechanisms such as formulae or contracts in which indicators can play a role.
Strategy use of research indicators by institutions in the formulation of their strategy, to decide upon such a strategy and to set aims to pursue.
Contract-based governance and steering (management by objectives) Use of research indicators in contracts between ministries and institutions or between institutions and departments to agree upon targets to be met.This concerns cases where no funding is involved.
Accountability use of research indicators by researchers, research groups, and research institutions to inform on their research activities to their higher management or to society at large.
Human resources management use of research indicators for the selection, hiring, promotion, and dismissal of personnel.
Quality assurance or quality improvement cannot only be achieved by funding or inducements, but also by means of quality assessment.Committees assessing quality might use research indicators as information about the research performance.

Reputation management use of research indicators to advertise strengths of a research institution or individual researchers
Selection of partners and members use of research indicators to inform institutions about the research performance of possible partners or candidates which apply for membership of professional associations.
Publication channel selection use of indicators such as the journal impact factor by authors to decide in which medium they will try to publish their work.

Research profile management use of research indicators by research institutions to manage their research profile and evaluate their strengths and weaknesses
Journal and database management use of research indicators to manage or support journals and bibliographic databases.
Library collection management use of research indicators by librarians to inform themselves about publications which should be adopted by their institutions.
Research indicators such as rankings are proxies for quality of institutes and serve different groups of consumers as important sources of information.
Compare the evaluation system in points.An analogue of the technique we met in New Zealand [27].
The evaluation of an eligible staff member's research performance is based on information contained within an EP, which has three components:

ЕКОНОМІЧНІ ІННОВАЦІЇ Том 20, Вип. 4 (69) 143
The research output component: This comprises up to four NROs, as well as up to 30 other research outputs.The research output component has a 70% weighting.For a research output to be eligible for inclusion, it has to have been produced (for example, published, publicly disseminated, presented, performed, or exhibited) for the first time within the assessment period.For the 2012 Quality Evaluation the period was 1 January 2006 to 31 December 2011.Research outputs are also required to satisfy the PBRF definition of research.
The peer rewew component: This comprises the recognition of a staff member's research by their peers (for example, prizes, awards, invitations to speak at conferences) and has a 15% weighting.
The contribution to the research environment component: This comprises a staff member's contribution to a vital high-quality research environment (for example, the supervision of research students, the receipt of research grants) and has a 15% weighting.
The assessment of involves scoring each of its three elements.In determining the appropriate score, the panels draw upon generic descriptors and tie-points (encapsulating the standard expected for a particular score) that apply to every panel, together with panel-specific guidelines.
The rating scale has the following characteristics: The scale for each component has eight steps (0-7), with 7 being the highest point on the scale and 0 being the lowest.A score of 0 indicates that no evidence has been provided in the EP for that component.Only whole scores are allocated (the use of fractions is not permitted).The descriptors and tie-points for each of the three components of an EP are used to assist with the scoring.The scoring associated with the Quality Categories is as follows: -"A" (indicative of a total weighted score of 600-700); -"B" (indicative of a total weighted score of 400-599); -"C" or "C(NE)" (indicative of a total weighted score of 200-399); -"R" or "R(NE)" (indicative of a total weighted score of less than 200).Similar approaches (4 gradations) are found in the Ukrainian document: "Methodology for evaluating the effectiveness of scientific institutions of the National Academy of Sciences of Ukraine" is developed in accordance with the Concept of Development of the National Academy of Sciences of Ukraine, approved by the decision of the Presidium of the National Academy of Sciences of Ukraine dated December 25, 2013 № 187.
The methodology is intended for the state certification of research institutes and universities the scientific institutions of the National Academy of Sciences (hereinafter referred to as the "Institution"), equated to them, to which the action applies the current Provision on the state attestation of scientific research (scientific and technical) institutions.
"Methodology for evaluating the effectiveness of scientific institutions of the National Academy of Sciences of Ukraine" is developed in accordance with the Concept of Development of the National Academy of Sciences of Ukraine, approved by the Presidium of the NAS of Ukraine dated December 25, 2013 No. 187.
The methodology is intended for the state certification of scientific research institutes and equivalent scientific institutions of the National Academy of Sciences of Ukraine (hereinafter referred to as the Institution), which are subject to the provisions of the current Regulations on state attestation of scientific research institutions.The method defines: the basic principles and criteria for evaluating the effectiveness of the Institutions; principles for determining the composition of commissions for evaluating the effectiveness of the Institutions; Separate organizational issues related to the use of this Methodology.The method can also be used during: Planned periodic and extraordinary inspections of scientific and scientific-organizational activities of the Institutions at the Presidium level of the National Academy of Sciences of Ukraine; preparation of proposals for optimization of the network of Institutions; determination of annual volumes of basic financing institutions.In assessing the most important publications issued by the Office, the criteria for assessing the quality of scientific results presented in table 1 shall be guided.

Table 1 Criteria for assessing the quality of scientific results in Ukraine
The level of quality Definition Description

A Very good
The results meet the international standards of excellent level.From the point of view of originality, significance and scientific ethics, these scientific results can be compared with very good works at the international level.The work at this level is of great interest to the international academic community.

Continuation of Table 1 B Good
The results meet the international standards of high level.From the point of view of originality, significance and scientific ethics, these scientific results can be compared with good work at the international level.Research satisfies high international standards of excellence.Jobs of this level are published by internationally recognized publishers and magazines C Sufficient The results to a certain extent correspond to the national standards of high leveFrom the point of view of originality, significance and scientific ethics, these scientific results are of sufficient quality in order to satisfy, first of all, the needs of Ukraine.D Weak From the point of view of originality, significance and scientific ethics, these scientific results do not correspond to the requirements of professional editions.

Source: author's development
The Ukrainian methodology, in our opinion, can be supplemented by 3 elements: Firstly: Moderation Panel.The role of the Moderation Panel is to: ensure that the assessment framework is applied consistently across the panels, while at the same time avoiding a situation in which the judgements of the panels are reduced to a mechanistic application of the assessment criteria; provide an opportunity to review the standards and processes being applied by the panels establish mechanisms and processes by which material differences or apparent inconsistencies in standards and processes can be addressed by the panels; advise the TEC on any issues regarding consistency of standards across panels.Panels were informed, by their chairs, of the findings of the first Moderation Panel meeting held prior to panel meetings.Panels devoted considerable attention to the calibration of scores for each of the three Evaluation Panel components and discussed how they would consistently apply the tie-points and descriptors in the Guidelines.All panels undertook a systematic review of Evaluation Panel.Particular attention was given to those EPs where the total weighted score was close to a Quality Category boundary.Panels considered all Evaluation Panels where panel pairs were unable to reach agreement on preliminary scores.Panels examined Evaluation Panels that had unusual score combinations for their research outputs, peer esteem, and contribution to the research environment components.During panel meetings, all panel members had the opportunity to be involved in an Evaluation Panel's assessment (other than where this was prevented by conflict of interest or in exceptional circumstances as noted in the individual panel reports).Prior to the designation of final quality scores the panels undertook a holistic assessment process of all Evaluation Panels.Panel chairs and their secretariats took an active role in ensuring that panels complied with the PBRF assessment framework, panel-specific guidelines, and the Guidelines.Panel meetings were also attended by the moderators and the TEC internal auditor to further ensure compliance with the assessment framework and guidelines.
Secondly, the PBRF audit.The primary objectives of the PBRF audit methodology were to: determine whether participating TEOs had adequate systems and controls for submitting to the TEC; determine whether participating TEOs had adequate systems and controls for identifying and verifying PBRF-eligible staff for inclusion in the PBRF census; understand participating TEOs' preparedness for submitting accurate PBRF census and Evaluation Panels data; provide assurance to the TEC and the PBRF peer-review panels that the material presented in the research outputs component of Evaluation Panels and in the TEOs' staff-eligibility data was complete and accurate.Independent assurance on the processes for the assessment of Evaluation Panels was provided by the TEC's internal auditor.
Thirdly, Comprehensive peer-review panel -Group of experts who were selected to evaluate the quality of research as set out in an individual evidence portfolio.
Conclusions and perspectives of further research.Theoretical basis of Performance-Based Research Funding (PBRF) assessment to increase the effectiveness of scientific research in Ukraine is developed.
The Performance-based research funding is governed by the following set of principles: comprehensiveness; respect for academic traditions; consistency; continuity; differentiation; credibility; efficiency; transparency; complementarity.
The Methodology for evaluating the effectiveness of scientific institutions of the National Academy of Sciences of Ukraine, in our opinion, can be supplemented by 3 elements: Moderation Panel.The role of the Moderation Panel is to: ensure that the assessment framework is applied consistently across the panels, while at the same time avoiding a situation in which the judgements of the panels are reduced to a mechanistic

ЕКОНОМІЧНІ ІННОВАЦІЇ
Том 20, Вип. 4 (69) 145 application of the assessment criteria; The primary objectives of the PBRF audit methodology were to: determine whether participating TEOs had adequate systems and controls for submitting to the TEC; determine whether participating TEOs had adequate systems and controls for identifying and verifying PBRF-eligible staff for inclusion in the PBRF census; understand participating TEOs' preparedness for submitting accurate PBRF census and Evaluation Panels data; provide assurance to the TEC and the PBRF peer-review panels that the material presented in the research outputs component of Evaluation Panels.Comprehensive peer-review panelgroup of experts who were selected to evaluate the quality of research as set out in an individual evidence portfolio.
The subsequent research will be aimed at fully adapting the method to the Ukrainian institutional conditions.