DEVELOPING AN INTEGRATED MODEL FOR EVALUATING R&D ORGANIZATIONS’ PERFORMANCE: COMBINATION OF DEA-ANP

. Assessing the performance of the Research and development (R&D) organizations to achieve higher productivity, growth, and development is always a critical necessity. Therefore, developing a more accurate model to evaluate the performance is always required. For this purpose, this study is aimed at developing a decision-making model for evaluating R&D performance. The model comes up with determining the most proper evaluative criteria for assessing R&D organizations. Then, it integrates Data Envelopment Analysis (DEA) with Analytical Network Process (ANP) to assess R&D performance. This paper is aimed to develop an integrated model for evaluating R&D performance. The findings of the study show that the DEA-ANP model is an accurate and acceptable model for evaluating R&D organizations’ performance. JEL Classification:


Introduction
Research and development (R&D) is an indicator of productivity, growth, and firms' competition (Salimi & Rezaei, 2018). It is found that the economic performance of many European countries had delayed or stopped because of lacking R&D investments (Khoshnevis & Teirlinck, 2018). The role of R&D in creating economic value can be assessed by combining two factors: 1) the economic value produced by R&D achievements and 2) the value of the strategic infrastructure (Khoshnevis & Teirlinck, 2018).
-This paper seeks to develop a combined DEA-ANP method for evaluating R&D industry-based organizations, which have been neglected in the previous literature. -This paper develops some new indicators for evaluating R&D organizations that have not been paid attention in the previous studies (e.g., "the degree of satisfaction of researchers with their jobs in the organization" and "project operationalization rate") and tries to go beyond the traditional indicators. -This paper tries to define every indicator clearly, such that the indicators can be easily quantified and measured. Thus, the potential user can easily benefit from it. -This paper proposes a suggestion including a focus on improving some indicators e.g., "researcher's work experience" and "degree of satisfaction of researchers with their job in the organization" to promote the general efficiency of the R&D industry-based organizations. The following of this paper is organized such that Section 1 reviews the recent studies evaluating R&D organizations and the common indicators employed in the previous studies. Section 2 introduces the model and its procedure along with the indicators used in this study. Section 3 solves the model and presents the results and the efficiency of R&D organizations. Section 4 discusses the study and presents solutions for increasing efficiency in R&D organizations. Finally, in the last section, the study is concluded and the suggestions for further study are presented.

Literature review
So far, various studies have been conducted in the field of performance evaluation for R&D organizations. They employed different models for evaluation, and a few studies employed a DEA-ANP model for assessing the efficiency of R&D organizations. However, in this study, we use a DEA-ANP model with more comprehensive and practical indicators and introduce the most effective criteria for the efficiency of these organizations. Here, some recent studies investigating the performance of R&D organizations, many of them using the DEA approach as a basic method for evaluating the performance. As a case study of research institutes in the Chinese Academy of Sciences, Xiong et al. assessed the efficiency of R&D organizations by a dynamic DEA approach (Xiong et al., 2018). They showed that the static DEA method may undervalue the scores of R&D efficiency. Kim and Cho examined the qualitative efficiency of National R&D Projects by the DEA approach focused on Agricultural Research Area (Kim & Cho, 2018). This study suggests that for promoting R&D performance, both quantitative and qualitative natures of outputs must be paid attention during evaluating R&D efficiency.
Chen et al. investigated regional R&D efficiency by dynamic DEA model in different periods to territorial R&D systems in China (Chen et al., 2018). This study presents a dynamic logical structure with a new method from a long-term and systemic view based on the DEA approach. In a study toward EU countries, Karadayi and Ekinci assessed R&D performance using categorical DEA (Karadayi & Ekinci, 2019). They employed the DEA model combined with CRS and VRS techniques as well as categorical data. Moreover, Khoshnevis and Teirlinck evaluated the performance of R&D companies in Belgium by the VRS DEA model (Khoshnevis & Teirlinck, 2018). They show that R&D active firms face problems including lack of technical efficiency and lack of appropriate scale size, because the CRS-VRS efficiency average is low, besides the scale efficiency average is moderate. Also, Dobrzanski and Bobowski employed the VRS DEA model in R&D projects in ASEAN states (Dobrzanski & Bobowski, 2020). Furthermore, for state R&D projects, Park and Shin evaluated the technical efficiency as well as the ratio of technology gap to study the efficiency of Korean state sub-biotechnologies R&D projects from 2007 to 2013 by a meta-frontier DEA approach (Park & Shin, 2018). Sun et al. evaluated the R&D efficiency of the teachers in university by a two-phase DEA approach (Sun et al., 2018). This paper analyses the classic DEA approach and presents a two-phase DEA model to resolve the drawback of classic DEA, and then generates a novel input-output system to assess the R&D efficiency of teachers. Zuo and Guan measured the R&D efficiency of different zones by a parallel DEA game approach (Zuo & Guan, 2017). In this study, the game cross-efficiency implication is integrated into the parallel DEA approach, such that each DMU (sub-process) seeks to maximize its efficiency without affecting the cross efficiency of other units, resulting in an algorithm to provide the best game cross-efficiency values.
In a comparative study, Cao et al. evaluated R&D functional platforms' performance by a DEA approach (Cao et al., 2019). In this study, the K-means clustering technique is employed to rank the R&D operative platforms of thirty provinces in China. Qin et al. investigated the territorial R&D efficiency and its spill-over effects in China by a combined DEA and spatial Durbin model (Qin et al., 2019). In this study, according to the effects of the R&D value chain, they show that R&D spillovers occurred intra-regionally. Wu et al. studied the efficiency of the Taiwanese semiconductor industry in the R&D field by the DEA approach (Wu et al., 2019). They employed a hybrid CCR-BCC technique. Carrillo investigated scaling and ranking R&D efficiency in the country by the VRS DEA model (2019). In this paper, the R&D efficiency of the studied countries is evaluated by the DEA model, then the overall efficiency score is presented with the cross-efficiency technique. Belgin analyzed the research and development efficiency of Turkish zones by DEA model (Belgin, 2019). In this study, primarily the efficiency values are measured, then sensitivity analysis is performed to identify the most effectual variables on territorial research and development efficiency by pair t-test. Asmara et al. measured R&D performance by DEA model in the case of Indonesia (Asmara et al., 2019).
Yu et al. presented a framework for deriving investment priority in National Defense R&D using DEA based on TRA (Technology Readiness Assessment) technique (Yu et al., 2018). Lim and Jeon analyzed the performance of defense R&D projects based on DEA model (Lim & Jeon, 2019). This study analyzed the three-stage performance of projects for the defense R&D section based on the logical model employing the DEA model. In 2019, Gibson et al. evaluated industry collaborative research centers by HDM (Hierarchical Decision Modeling) method (Gibson et al., 2019). This study employs quantitative and qualitative criteria and has a generalizable model that is advanced based upon plan objectives and the results are verified by consulting with experts. In 2018, Salimi and Rezaei evaluated the R&D efficiency of firms using the "Best Worst" technique (Salimi & Rezaei, 2018). This study investigates R&D performance regarding the different significance values of R&D criteria, by a multi-criteria decision-making method named "Best Worst" technique (BWM) for identifying the weights (importance) of R&D criteria for fifty high-tech SMEs in the Netherlands.
Ersoyak and Ozcan presented a performance evaluation structure for R&D projects in the software sector by the Balanced Scorecard method (Ersoyak & Ozcan, 2019). This study employs a sequential mixed technique, in which key performance criteria are extracted from preliminary interviews with related experts. You and Jung conducted an analysis of system dynamics from the National R&D Performance Measurement System in Korea by the Peer review process (You & Jung, 2019). In this paper, a new method named MPUIC which is developed based on the Design Science Research Methodology (DSRM) and weighted scoring approach is employed and verified by a case study. Ge and Yang studied the R&D performance of the high-tech industries in China through the DEA model (Ge & Yang, 2017).
Finally, Wu et al. assessed the efficiency of R&D organizations due to capital stock by a dynamic combined DEA-ANP technique (Wu et al., 2016). This study is more similar to our study in terms of the model. In this study, first, a dynamic three-phase network DEA model is presented, which assesses the R&D performance, technology-based efficiency, and valuecreation efficiency of R&D systems in Taiwan from 2005 to 2009. Before incorporating window analysis and network DEA model for estimating dynamic efficiencies, the ANP method is applied to specify the relative importance of each phase. Then, a panel data regression is conducted to investigate whether the capital stock of patents, human resources quality, and service support influence the dynamic efficiencies of these organizations or not.
As can be seen, only the last study by Wu et al. employed a DEA-ANP hybrid model for R&D Organizations. However, our study uses another algorithm of the DEA-ANP hybrid model for another type of R&D organization along with more complete and comprehensive criteria to make a more accurate and practical evaluation.
Next, the indicators (inputs and outputs) employed in recent studies in this field are presented in Table 1. Of course, we should note that indicators of inputs and outputs are mostly defined in models based on the DEA approach. As we can see in Table 1, there are almost limited and repeated indicators used in evaluating the efficiency of R&D organizations. In this study, we try to evaluate the organizations in terms of more dimensions to fulfill this gap e.g., we define some criteria including "satisfaction of the researchers", "satisfaction of the customers (clients)", "researchers' work experience". So, the evaluation can be more comprehensive observing many aspects. Of course, recently it is shown that some issues like gender affect R&D efficiency. Kou et al. showed that a higher ratio of female R&D staff promotes R&D efficiency (Kou et al., 2020). They found that gender diversity in R&D groups can enhance innovation efficiency by obtaining informational and social advantages (Xie et al., 2020).

Method and data
In this section, we first briefly describe the two methods of DEA and ANP. We then present the proposed approach in this article, along with a full description.

Data Envelopment Analysis (DEA)
The method of data envelopment analysis was first presented by Dr. Rhodes at the University of Mellon in 1978 as a doctoral thesis. It was used to evaluate the academic achievement of US national school students. The first paper on data envelopment analysis was published that year by Charnes and Cooper in 1978 and the model presented herein became known as the CCR model (Calik et al., 2018;Karsak & Goker, 2020). In this model, the efficiency of each DMU is a fractional planning problem, i.e., the efficiency of each DMU is the maximum ratio of the weighted output to weighted input under some constraints. Assuming that x ij and y rj are inputs and outputs with limits greater or equal to zero, besides v i (i = 1,…, m) and u r (r = 1,…, s) are the weights related to the inputs and outputs respectively, and each decisionmaker also has at least one positive input component and one positive output component, thus we will have the following Equation: End of (1) Provided that: In data envelopment analysis, the efficiency score is calculated for each unit under study, which is a numerical range of zero and one, and it divides the units under study into two groups: "efficient units" and "inefficient units". A unit that scores one is efficient (e k = 1) and a unit that scores less than one is inefficient (e k < 1).

Network Analysis Process
The process of network analysis is the one invented by Thomas L. Saaty in 1996 (Tian & Peng, 2020). According to his definition, ANP is a more general and more complete model than AHP that allows analysis of various issues by making interactions between elements. In AHP, there are four conditions including inverse, homogeneity, dependency, and expectation conditions that in the network analysis process, the third hierarchical condition is violated, because, in a hierarchy, the dependencies must be linear from top to bottom or vice versa, which makes it impossible to analyze these issues showing an interrelationship between options and criteria. Figure 1 shows the structural differences between the two AHP and ANP models.
Therefore, to solve a problem by this method, first, a network including goals, criteria, sub-criteria, and options is created and then the relations between them are recognized and drawn. Then, by pairwise comparisons and the formation of the super-matrix, the option with the highest end-weight is selected as the best option. The network analytical process is illustrated in Figure 2.

Hybrid DEA-ANP method
The hybrid DEA-ANP method used in this paper has two main phases. The first phase consists of five basic steps: Step 1 From the solution of the above mathematical model, the values of kk e ′ (k = 1,…, n; k′= 1,…, n; k ≠ k′) are obtained and the matrix E is created by the number of k rows and k′ columns, such that the elements on the diagonal all get value one. After forming the pair comparison matrix E, the first phase proceeds with the following four steps: Step 2. Calculating the pair comparison matrix A ( ) In the ANP method, on the diagonal of the pairwise comparison matrix, the rank of the element kk a ′ reflects the evaluation of the unit k compared to the unit k′. Also, the relation- Step 3. Calculating the pairwise comparison matrix A′ ( ).

kk a′
After obtaining the pairwise comparisons matrix A, this matrix should be normalized. The new normalized matrix A′ is obtained by Equation (6) (dividing each element by the sum of its corresponding column elements).

kk a ′ ′′′
By normalizing the column vector A′′, it is found that the vector A′′′ is the complete ranking of the organizational units.
Phase Two -At this phase, based on the pairwise comparisons matrix E, a super-matrix is obtained from the interactions between organizations is calculated using the ANP method and the views of experts, which is called matrix W*. The final result of the algorithm and the efficiency of the units is obtained by multiplying the two matrices A′′′ and W* ( * ) kk k a w ′ ′′′ × . Figure 3 gives an overview of the steps of this method.  Step

Model indicators
Initially, by reviewing the literature of previous research and studies and also interviewing with relevant experts, more effective indicators on R&D efficiency in different organizations were identified. The experts in this study are managers of R&D organizations. Then, using these indicators, the inputs that represent the resources, and the outputs that represent the success and performance level of the DMUs generally were identified.
In this model, 6 input indicators (n = 6) and 7 output indicators (m = 7) are considered. Meanwhile, the number of organizations under study (DMUs) is 17. It is noted that the duration of each period is one year.
A very important point about these indicators is that according to previous literature, a relatively new set of indicators has been compiled in this study so that all indicators are clearly defined and described, which are easily measurable (Lack of a clear definition and also lack of a specific method for measuring indicators is one of the drawbacks of many previous studies in this area).
The input indicators of the model are as follows (see Table 2). This item is a qualitative indicator and should become quantitative. *

Number
Note: * This item is calculated as follows: A survey at the organization's website will be presented to the researchers at the end of the year and according to the different aspects and features of their job, they will be asked to express their satisfaction in three areas. Salary and benefits and rewards and how they are paid: Excellent (1) Good (2) Average (3) Weak (4) Very weak (5). Existence of equipment and facilities and suitable workspace for the implementation of projects: Excellent (1) Good (2) Average (3) Weak (4) Very weak (5). Training and updating the researchers and the attention of the organization to their scientific and practical improvement: Excellent (1) Good (2) Average (3) Weak (4) Very weak (5). For all three domains, the total score is calculated and divided by the number of responses and the per capita satisfaction of researchers in each domain as 3 numbers is obtained. Then the average of these three numbers is presented, which is the amount of final satisfaction of the researchers from their organization.
The output indicators are as follows (see Table 3). The number of projects that are sold (or used) for production or service organizations throughout the year, divided by the total number of completed projects throughout the year In this paper, 17 semi-state industry-university R&D collaboration agencies in 17 provinces of Iran supported by the Industry Ministry are studied. In the following, the values of inputs and outputs for these 17 agencies are presented as follows (see Tables 4-5).

Problem solving and results
As noted before, this study develops a combined DEA-ANP method to overcome some deficits in previous related studies, i.e. ordering and full-ranking problems caused by the DEA model and the whole hierarchy and subjective evaluations caused by the ANP method. This study tries to overcome these problems for evaluating the performance of industry-based R&D organizations. Now, based on the data obtained, we apply the steps of the DEA-ANP hybrid method to measure the efficiency of R&D organizations. In the first step of the first phase of this method, we compute the pairwise comparisons matrix E for the present organizations as shown in Table 6.
The rest of the solution is performed according to steps 2, 3, 4, and 5 described in the previous section, and the results of the first step including the hybrid method along with the same column vectors A′′′ (normalized column vectors A′′′) are shown in Table 7.
In the second phase of the DEA-ANP method, the super-matrix derived from the interactions between organizations (W*) is presented by the views of experts and the ANP method which is shown in Table 8.
Finally, the column vectors A′′′ from phase 1 and the matrix W* from phase 2 are multiplied, yielding the final results and efficiency values of the R&D organizations presented in Table 9.

Evaluating the validity of the results
Now, to validate the results, the final efficiencies obtained by the combined DEA-ANP method are compared with the efficiencies obtained by the standard DEA method (CCR model), which is shown in Table 10.
As shown in Table 10, the DEA method divides DMUs only into "efficient" and "inefficient" classes, whereas the DEA-ANP method calculates the efficiency values more precisely and practically. However, we can see that there is an uniform compatibility between the mutual efficiencies of DMUs by DEA method and DEA-ANP method, such that all DMUs whose efficiency is 1 in DEA method have efficiency value higher than 0.1 in combined DEA-ANP method and DMUs whose efficiency is lower than 1 in DEA method, have efficiency value lower than 0.1 in combined DEA-ANP method (except organization 9).
In the next step for validating the results, the final efficiencies obtained by the combined DEA-ANP method are compared with the super-efficiencies obtained by Andersen and Petersen (AP) model. By this method, it can be said that efficient units in the input-oriented model are valued more than one, and a detailed ranking can be presented. The results and rankings obtained by these two methods are presented in Table 11.
As shown in Table 11, there is also a tangible compatibility between the mutual efficiencies and rankings of DMUs by AP method and DEA-ANP method, such that all DMUs whose efficiency is more than 1 in AP method, have efficiency value higher than 0.1 in combined DEA-ANP method and DMUs whose efficiency is lower than 1 in AP method, have efficiency value lower than 0.1 in combined DEA-ANP method (except organization 4). As can be seen, DEA and AP methods almost validate the results obtained by the model in this paper. Moreover, statistically, we can test the compatibility between the model in this paper and DEA and AP methods by the Mann-Whitney U test respectively. Using the Mann-Whitney U test for this case, we found that the model in this paper and DEA and AP methods are mutually compatible with a P-value ≈ 0.1 (α = 0.05).
In addition, compared to the results of the study of Wu et al. (2016) which is relatively similar, the accuracy of the results in both studies are at the same level. Furthermore, it should be mentioned that the number of criteria employed in our study is more comprehensive, thus we can claim that our results are more practical and observe more aspects.

Evaluating the relationship between existing indicators with the efficiency by correlation analysis
In the following, the input and output criteria employed in the study were evaluated by the Pearson correlation test in SPSS software and then their correlation with the relevant efficiency was evaluated. It is worth noting that Halásková and Bazsová also used a correlation analysis to assess the input and output indicators concerning R&D efficiency by the DEA method for EU members (Halásková & Bazsová, 2016) In Table 12, we present the correlation and determination coefficient between existing indicators and the efficiency of R&D organizations.
In the above table, it should first be noted that the determination coefficient of the indicator indicates how much and what percentage of the organization's efficiency is predicted by that indicator (equal with the squared correlation coefficient). According to Table 12, 4 indicators including "Researcher's work experience" (I3), "Degree of satisfaction of researchers with their job" (I6), "Patents during a year" (O3), and "Project Operationalization Rate" (O5) have the most correlation with the efficiency, therefore, to improve the efficiency of R&D organizations, we should focus as much as possible on improving these indicators, since they have the greatest impact on increasing the efficiency of the R&D organizations. Based on the obtained results, the R&D managers should adopt the following policies and measures to improve efficiency: 1. Hiring more experienced researchers for optimizing the efficiency of organizations, reducing the costs, etc., as much as possible. 2. Providing the best occupational conditions for the researchers in terms of salary, fees, facilities, etc., as much as possible. 3. Selecting and implementing the projects that will have many customers and can be sold at a high price (in a high level of confidence). Therefore, the R&D managers and decision-makers must identify the needs of the market and select the appropriate projects to implement according to their capabilities.

Conclusions
In the field of evaluating the efficiency of the organization, one of the important points is to discover and consider new and applicable indicators for evaluating the organizations under study. As noted before, this paper develops some new practical indicators with a clear definition of them for the evaluation of R&D organizations; indicators that have not been considered in previous studies or have been less investigated. Among these indicators, we can note the "work experience of researchers", "the degree of satisfaction of researchers with their jobs in the organization", "project operationalization rate", and "client (project user) satisfaction". This leads to a more comprehensive, practical, and accurate model for evaluating the performance compared to related studies. Furthermore, this study developed a DEA-ANP hybrid model for evaluating the performance of industry-based R&D organizations, which has not been paid attention in previous studies. Finally, by performing Pearson correlation analysis on the employed indicators and the obtained efficiencies of organizations, to promote the whole efficiency of the R&D organizations, this study suggests the managers to focus on improving some more effective indicators. These indicators include "researcher's work experience", "degree of satisfaction of researchers with their job in the organization", "patents during a year", and "project operationalization rate". This suggestion somehow is more detailed compared to the suggestion of previous studies. For future studies, it is suggested that these indicators can be used in the form of new approaches such as DEA-AHP, TOPSIS-ANP, VIKOR-ANP combined models, or regression-based models for evaluating R&D organizations to verify the results of this study. The interval data can also be used in the DEA-ANP hybrid model to calculate fuzzy inputs and outputs for a more accurate evaluation of the efficiency. Moreover, by studying the various economic, industrial, organizational, even political, etc. features of R&D organizations, newer and more updated indicators can be determined and defined. Also, it is important to note that some indicators are also highly regarded at specific national and regional levels, but are not well defined globally and internationally.