FUZZY-BASED COMPOSITE INDICATOR DEVELOPMENT METHODOLOGY FOR EVALUATING OVERALL PROJECT PERFORMANCE

Construction companies develop and use composite indicators to evaluate their overall project performance. However, the conventional methodology of composite indicator development causes “the indiscrimination problem”, a low degree of performance discrimination due to low resolution of measurement, and “the redundancy problem”, an incorrect evaluation caused by interrelation among sub-indicators. To address these problems, we propose a novel methodology that uses fuzzy theories. The proposed methodology includes the utility function for normalizing, the fuzzy measure for weighting, and the fuzzy integral for aggregating. A retrospective case study on 52 real projects shows that our proposed methodology can help alleviate the indiscrimination and redundancy problems: the proposed methodology significantly improved the degree of performance discrimination (0.29 to 0.92) and changed ranks of underor overestimated projects by taking the interactions of sub-indicators into account. Our methodology can contribute more accurate evaluation of overall project performance with higher degrees of performance discrimination.


Introduction
Construction companies evaluate their project success by measuring project performance based on predetermined success criteria and comparing it with other projects. These success criteria include various aspects, such as schedule, cost, quality, and safety performance, and each aspect again has many sub-indicators to measure its performance (Dainty et al. 2003;Kumaraswamy, Thorpe 1996). To evaluate the overall project performance or project success, construction companies need to develop a composite indicator by normalizing, weighting, and aggregating the sub-indicators. Construction companies commonly use a categorical scale to normalize the values of sub-indicators with different measures, the budget allocation to weight the sub-indicators, and the additive aggregation function to aggregate the weighted sub-indicators. However, despite their simplicity in implementation and interpretation, these methods do not appropriately address the following problems that are inherent in these methods: -First, the categorical scale converts continuous values of the sub-indicators into discontinuous categorical values; in this process, the low resolution of measurement often impairs performance discrimination (Hand 2004). We refer to this as "the indiscrimination problem". For example, the progress rate of 81% in a given project A and the progress rate of 89% in a given project B are equally converted to 8 points in a categorical scale, meaning that these two projects lose their progress rate differences. -Second, the additive weighting method does not consider that the interaction among sub-indicators can cause "the redundancy problem" (Grabisch 1996). For example, in the area of safety performance, two sub-indicators, the number of accidents and the amount paid due to accidents in a project, are interrelated and move in a similar direction. The development of a composite indicator through mere adding of the weights of these indicators can lead to an incorrect estimation of the safety performance level because of redundancy of these two sub-indicators. We addressed these problems by developing a novel methodology that applies fuzzy theories in developing a composite indicator for evaluating overall project performance. Section 1 summarizes current state of the practice and state of the theory we reviewed. Section 2 describes the methodology we formalized to alleviate the indiscrimination problem and the redundancy problem during the overall project performance evaluation. Specifically, we propose the utility function as a replacement for the categorical scale to address the indiscrimination problem. We then apply the fuzzy measure and the fuzzy integral for weighting and aggregating the normalized sub-indicators to address the redundancy problem. In Section 3, we demonstrate the effectiveness of our proposed methodology through a retrospective case study using the cost performance data from 52 real projects of a construction company. Our results show that the degree of performance discrimination increased from 0.29 (using the conventional methodology) to 0.92 (using our proposed methodology), which explains the alleviation of the indiscrimination problem. Our results also show that many projects are over-or under-estimated in the schemes of the conventional methodology and are now positioned in different ranks due to the alleviation of the redundancy problem. Our methodology enables construction companies to evaluate their overall project performance more accurately by addressing the indiscrimination and redundancy problems.

Points of departure
We have reviewed the need for composite performance indicators and then observed the current practice of construction companies for developing a composite performance indicator. We found that construction companies suffer from the indiscrimination and redundancy problems that are described earlier. We then reviewed the existing methods for developing the composite indicators upon which our methodology builds and extends.

Performance evaluation in the construction industry
On one side of the performance evaluation spectrum, researchers in the construction discipline have identified project success criteria or key performance indicators (Al-Tmeemy et al. 2011;Cha, Kim 2011;Chan et al. 2004;Dainty et al. 2003;Lim, Mohamed 1999;Shenhar et al. 1997) and developed computer-aided systems to measure and track these indicators in a rapid and consistent way (Marques et al. 2011;Cheung 2004;Kumaraswamy, Thorpe 1996). This approach helps construction companies and governments pinpoint their systems' weaknesses and develop a managerial plan for addressing these weaknesses. This approach also provides researchers with valuable data for assessing the trends of specific criteria and the relationships among them.
On the other side of the spectrum, construction companies have also utilized composite indicators to measure and compare their overall project performance due to their usefulness as a communication tool (Freudenberg 2003) and a decision support tool (Saltelli 2006). That is, the overall performance of a project can only be measured using a composite indicator. Therefore, many researchers support the use of composite performance indicators as well as the use of individual indicators (i.e. project success criteria or key performance indicators). Lauras et al. (2010) and Marques et al. (2011) argue that project managers need to quantify project performance as a whole. Clivillé et al. (2007) point out that the performance management system should involve two kinds of performance metrics: elementary (i.e. individual indicators that represent different performance objectives) and aggregated (i.e. composite indicators that synthesize the elementary indicators into global objectives). Kumaraswamy and Thorpe (1996) suggest the use of "a project performance profile", which is composed of principal performance criteria and corresponding sub-criteria in a hierarchical structure. Landy and Farr (1983) argue that combined performance data are needed because the availability of overall performance ratings is useful for administrative decisions.

Current practice of developing composite indicators
To develop a composite indicator for evaluating overall project performance, construction companies often use a categorical scale for normalization, budget allocation for weighting, and an additive aggregation function for aggregation. Our observed case exemplifies the prevalent practice of composite indicator development. The two problems described earlier (i.e. the indiscrimination and redundancy problems) were observed in this case.
We investigated the cost performance of 52 real projects provided by a construction company in Korea (Table 1). To evaluate the cost performance of each project, the company measured three sub-indicators, i.e. the sales completion rate by percentage, the cost spending rate by percentage, and the work productivity in currency (Korean won). The following equations were used in the process: (1) (3)

The indiscrimination problem:
In an attempt to develop a composite indicator, the company used a categorical scale (range, 1-10) for normalizing the data. Because the sub-indicators have different dimensions and orientations (e.g. the larger the sales completion rate, the higher the performance score; the smaller the cost spending rate, the higher the performance score), the company normalized the sub-indicators into the same scale to make them comparable (Fig. 1). Although this categorical scale provided the construction company with an easy way In other words, since the scale had 10 categories, the company allocated each of the continuous values of the 52 projects to one of these 10 categories. Consequently, the discriminating power of the original measurement was lost once different values were allocated to the same category. This problem of indiscrimination worsens when the number of categories decreases. In addition, when the value of a sub-indicator changes slightly within a category over time, construction companies are not able to appreciate the change and respond to it adequately.

The redundancy problem:
Once the normalized sub-indicators were ready, the company evaluated the overall cost performance by weighting and aggregating them. The company used the budget allocation to establish the weights for these normalized sub-indicators (0.2 for the sales completion rate, 0.4 for the cost spending rate, and 0.4 for the work productivity). Although these weights have a critical impact on the results of project performance evaluation, they are heavily dependent on the opinions of the experts who participate in the budget allocation method, and therefore, they are out of the scope of this research. The company then used the additive aggregation function, which multiplied the subindicator values by the corresponding weights and summed all of the terms to evaluate the overall cost performance: (4) where: w s = 0.2, w c = 0.4, and w w = 0.4.
Although these methods are widely used in the development of a composite indicator (Saisana, Tarantola 2002), they assume preference independence, which Nardo et al. (2005) define as "given the sub-indicators, an additive aggregation function exists if and only if these indicators are mutually preferentially independent". If two or more indicators measure the same system behavior, and therefore violate the assumption of preference independence, a certain aspect of performance will be redundantly weighted (Freudenberg 2003;Grabisch 1996). In this observed case, the company used a metric of "work quantity completed" to measure the three sub-indicators (Eqns (1)- (3)). In addition, "work quantity planned" was used to measure the sales completion rate and the cost spending rate, while "budgeted unit price" was used to measure the cost spending rate and work productivity. To address the redundancy problem, these interrelations between the sub-indicators must be taken into account when the sub-indicators are weighted and aggregated. The company can perform a sensitivity analysis to be informed about the impact of the weights and modify them. However, this modification, without changing the overall methodology for evaluating project performance, does not solve the aforementioned problems.

Composite indicators development
A composite indicator is generally developed by (1) developing a theoretical framework, (2) selecting subindicators, (3) inventing a composite indicator, (4) testing the robustness of the composite indicator, and (5) reporting the results using the composite indicators (Freudenberg 2003;Nardo et al. 2005;Saisana, Tarantola 2002). Inventing a composite indicator, which is our focus in this research, again consists of three steps, i.e. normalization, weighting, and aggregation. Various methods have been developed for each step: normalization methods including z-scores, re-scaling, distance to the target, distance from the mean, and the categorical scale; weighing methods including the principal component analysis, the factor analysis, efficiency frontier, budget allocation, public opinion, the analytic hierarchy process (AHP), and randomly assigned weights; and aggregation methods including ranking, the additive aggregation function, and the percentage of differences over consecutive time periods. These methods provide evaluators (i.e. practitioners who evaluate overall project performance for construction organizations) with an opportunity to choose an appropriate set of methods based on the context of the evaluation and their purposes. In this context, researchers (Bai et al. 2011;Cha, Kim 2011;Park et al. 2009;Shouke et al. 2010) suggest various composite indicator models, which are different from the widely accepted model in the construction industry, i.e. a model that uses a categorical scale for normalization, the budget allocation for weighing, and the additive aggregation function for aggregation as explained in Section 1.2. However, research efforts that help construction companies choose an appropriate set of methods for addressing the indiscrimination and redundancy problems are still lacking.
Fuzzy theories, including the fuzzy measure and the fuzzy integral, can be utilized to address these problems due to their ability to model the interaction among subindicators (Grabisch 1996). The concept of the fuzzy measure was originally introduced by Choquet (1953) and then elaborated by Sugeno (1974) to extend the classical (i.e. probability) measure through relaxation of the additivity property. The fuzzy integral is essentially used as an aggregation or fusion operator in which the fuzzy measure exerts weight on various criteria or features (Grabisch 1997). The Choquet and the Sugeno integrals are the two well-known forms of the fuzzy integral. While the Sugeno integral is based on nonlinear operators (min and max), the Choquet integral is based on linear operators and is a natural extension of the Lebesgue integral (Liginlal, Ow 2006). Many researchers apply fuzzy theories in various disciplines such as enterprise intranet web sites evaluation (Tzeng et al. 2005), e-commerce strategies evaluation (Chiu et al. 2004), and generic projects evaluation of small and medium enterprises (Shen, Hsieh 2010).
In construction management disciplines, fuzzy theories have been often used to deal with uncertainties in design performance prediction (Fayek, Sun 2001) or labor productivity prediction (Fayek, Oduba 2005). Carr and Tah (2001) use fuzzy logic to represent and quantify the relationships among risk factors, risks, and their consequences, which are commonly described in natural language. Sun and Bi (2010) demonstrate the use of the fuzzy analytic network process in evaluating disaster reconstruction project performance. Although these studies provide valuable insights into the relationships between fuzzy theories and performance evaluation, they do not explicitly address the indiscrimination and redundancy problems in the context of construction project performance evaluation. In addition, research efforts that apply fuzzy theories to evaluate overall project performance and explain the effectiveness of the application are lacking.

Fuzzy-based methodology for composite performance indicator
There is a need for a novel methodology that helps construction companies develop a composite performance indicator that addresses the indiscrimination and redundancy problems. Therefore, we have defined three elements with which we build and then formalized our methodology for developing a composite indicator.

Elements of the proposed methodology
This section describes three elements of the proposed methodology, i.e. utility function, the fuzzy measure, and the fuzzy integral. In our proposed methodology, the utility function is utilized to address the indiscrimination problem, while the fuzzy measure and the fuzzy integral are utilized to address the redundancy problem.

Utility function
The utility is a scale of values that reflects the decision maker's preferences. The utility function is a graphical or mathematical function relating the values of various outcomes to the intrinsic value of a particular decision maker. The utility function is also referred to as the utility curve or risk preference curve. The utility value is measured in arbitrary units called utiles. The x-axis (the utility function's argument) is calibrated in directly measureable units. The y-axis origin and scale (expressed in utiles or utils) are arbitrary (Schuyler 1996).
The utility function can help address the indiscrimination problem because the y-axis can also have contin-uous values. For example, the sub-indicators of project performance described in Section 1.2 (i.e. the sales completion rate, the cost spending rate, and work productivity) are used in the x-axis. For the y-axis, a 0 to 1 scale can be used to normalize the different sub-indicator scales without affecting their discriminating power.

Fuzzy measure
The fuzzy measure can be used to model the interrelation between sub-indicators. The fuzzy measure of the union of two disjointed sets, g(A) and g(B), cannot be assessed by simply adding g(A) and g(B). Therefore, Sugeno (1974) proposes the λ-fuzzy measure, in which the λ parameter is used to represent the interrelations between combined sub-indicators (Eqn (5)): where .
The union of two disjointed sets has the following three different meanings according to the λ value: -Super-additivity: when λ is a positive value, is larger than the sum of g(A) and g(B). This can be used to increase the impact of the two sub-indicators on overall performance.
-Additivity: when λ is 0, is equal to the sum of g(A) and g(B), which means that there is no interrelation between the two sub-indicators.
-Sub-additivity: when λ is a negative value, is smaller than the sum of g(A) and g(B). This can be used to reduce the impact of the two subindicators on overall performance taking into account their interrelation. The Sugeno λ-fuzzy measure can be generalized for X = {x 1 , x 2 , …, x n } as the following equation: where g i = g({x i }), x i : an arbitrary sub-indicator, n: the number of the sub-indicators.
The value of λ is obtained through the boundary condition, g(X) = 1, which yields a polynomial equation with respect to λ, given by: The value of λ can be obtained by Eqn (7) using each g i with the weights of the sub-indicators x i . This is because one possible meaning of a fuzzy measure can be defined as the level of importance or the degree of belief of a single criterion in the overall evaluation of a system (Pham, Yan 1997). In addition, according to the fundamental theorem regarding the λ-fuzzy measure (Leszczyński et al. 1985), λ-value has the following three cases:

Fuzzy integral
In our proposed methodology, we suggest the use of the Choquet integral in aggregating the sub-indicators. The Choquet fuzzy integral, proposed by Murofushi and Sugeno (1989), has been used as a nonlinear aggregation tool in information fusion and data mining (Yang et al. 2005). This method provides the computational schemes for aggregating the sub-indicator values based on the λ-fuzzy measure described above. If h(x 1 ), h(x 2 ), …, h(x n ) are assumed to be a collection of input sources of h, and g is a λ-fuzzy measure, then the following Choquet fuzzy integral can be constructed:

The proposed methodology
We created a novel methodology for evaluating overall project performance by applying fuzzy theories in synthesizing multiple criteria. We do not underestimate the importance of collecting reliable raw performance data and designing relevant sub-indicators with regard to overall performance evaluation success; however, in this research, we focus on how to develop a composite indicator that addresses the indiscrimination and redundancy problems assuming that the raw data and sub-indicators are designed for a company's interests and given. The proposed methodology has the following three steps: -Step 1, Normalization. Sub-indicators are measured based on the raw data in construction projects and then normalized using the utility function. Because sub-indictor values have different scales, we suggest the use of the utility function as a normalization method so that a construction company can combine the values into a composite value. This function interpolates the values within a given category using the two boundary conditions that represent a company's perception of the utility. -Step 2, Weighting. The normalized values are weighed using the fuzzy measure. The method used to obtain λ-fuzzy measure values for the Choquet fuzzy integral is as follows. First, we determine the g i that is the importance measure or the contribution of each single sub-indicator to a composite indicator. In order to take into account the redundancy effect between sub-indicators raised in Section 2.2 of this paper, we assumed that the sub-indicators have super-additivity. This means that the sum of three sub-indicators' weights exceeds one ( , thus -1< λ < 0). Second, we calculate the value of λ using Eqn (7) given the g i determined above. Finally, we calculate the λ-fuzzy measure value of each g(A i ) using Eqn (6). -Step 3, Aggregation. The normalized values are then aggregated to produce a composite value (i.e. overall project performance) using Eqn (8). The weights of the sub-indicators are determined in Step 2 using the λ-fuzzy measure. An overview of the proposed methodology is shown in Figure 2. Our methodology allows construction companies to evaluate overall project performance with higher accuracy, i.e. higher precision by addressing the indiscrimination problem and higher validity by addressing the redundancy problem (Hand 2004).

Case study
We conducted a retrospective case study to demonstrate the effectiveness of the proposed methodology versus the conventional methodology. Following the steps that are defined in Section 2.2, we determined whether these steps can guide construction companies in the development of a composite indicator for evaluating overall project performance. This section also demonstrates how the indiscrimination and redundancy problems can be managed by the proposed methodology compared to those of the conventional methodology.

Case study overview
We used the values of three sub-indicators measured in the 52 projects that are presented in Section 1.2. We evaluated the cost performance of these projects using the following three different methodologies (i.e. conventional, alternative, and proposed) and analyzed the differences regarding their effectiveness for addressing indiscrimination and redundancy problems. The alternative methodology was devised to demonstrate the effectiveness of the proposed method's stepwise application: -Conventional. This methodology uses the categorical scale for normalization, the budget allocation for weighting, and the additive aggregation function for aggregation. The weights of sub-indicators are w s = 0.2, w c = 0.4, and w w = 0.4. -Alternative. This methodology uses the utility function for normalization but remains to use the budget allocation and the additive aggregation function for weighing and aggregation. The weights of subindicators are w s = 0.2, w c = 0.4, and w w = 0.4. -Proposed. This methodology uses the full combination of the elements that are described in Section 2.1: the utility function for normalization, the λ-fuzzy measure for weighting, and the fuzzy integral for aggregation. We asked three senior project managers of the company to determine the degree of influence of each sub-indicator without consideration of the constraint that the sum of these values must be one. As a result, these values are g s = 0.3, g c = 0.6, and g w = 0.5.

Application of the proposed methodology
This section demonstrates our application of the proposed methodology using the values of the three sub-indicators measured in 52 projects by a construction company in Korea. This paper describes the application of weighing and aggregation methods using project 4 (the fourth project) of the 52 projects, which was chosen arbitrarily.
Step 1: Normalization To address the indiscrimination problem during normalizing sub-indicator values, we developed utility functions that respond to different sub-indicator values (Fig. 3).
Step 2: Weighting We then weighted the normalized sub-indicator values (S, C, and W) based on the fuzzy measure. In the case of project 4, λ was calculated as -0.7062 using the Eqn (7). The calculation process is shown below: The values of normalized sub-indicator h(x i ) are listed in descending order as shown in Table 2. We obtained each g i from x i that was used by the company as shown in Eqn (4). The values of g(A i ) were also calculated using the λ and g i values as listed in Table 2. The calculation process for g(A i ) of the project 4 is shown below: Step 3: Aggregation This step involves obtaining a composite indicator using the fuzzy integral. The composite indicator evaluated in this case study was "the cost performance", which was composed of the three sub-indicators, i.e. the sales completion rate, the cost spending rate, and work productivity. For example, we calculated the cost performance of project 4 using Eqn (8) as shown in the following equation. Figure 4 shows a graphical representation of the equation:

Results
We compared the evaluation results with the three different methodologies defined above (Table 3). Our results show that the proposed methodology helps evaluate the overall project performance with a higher degree of performance discrimination and greater validity compared to the conventional methodology due to its alleviation of the indiscrimination and redundancy problems.
To measure the degree of performance discrimination, we defined the following metrics: We calculated the degree of performance discrimination for each methodology using Eqn (9). As a result, we determined that the performance discrimination of the proposed methodology is three times that of the conventional methodology (from 0.29 to 0.92). Therefore, with our proposed methodology that provides higher performance now have different ranks ranging from 12 th (project 26) to 24 th (project 8). Thus, our new ranks enable the construction company to differentiate these eight projects by taking the interrelation of the sub-indicators into account.

Expert feedback
To complement the comparison of the conventional methodology and our proposed methodology with a broader analysis of our research regarding its contributions and future research, we conducted a survey using five senior project managers in four major construction companies in Korea. The participants have an average of 9.2 years of project evaluation experience and 17.8 years of construction experience. In this survey, we demonstrated the two methodologies (i.e. the conventional methodology and the proposed methodology) and asked the participants to provide their opinions and suggestions.
In a result, we gained positive approvals from the participants, including the following comments: -"The proposed methodology is useful and has a strong power to discriminate projects in the same conditions. I intend to use this for evaluating projects in my company". -"The proposed methodology would be very helpful when companies need a higher degree of project discrimination during their project evaluations". -"Construction companies have been seeking more objective and accurate techniques for evaluating overall project performance. The proposed methodology is a timely advance in project evaluation".
We also received constructive suggestions that would provide us with key research directions to improve our methodology. The following future research areas were identified based on the suggestions: -Taking different project characteristics into account: The overall performance of a project is affected by many project characteristics, such as project type, project phase, contract type, and headquarter support. Therefore, our methodology for evaluating overall project performance must include these characteristics in the future. -Investigating under-and over-estimated projects in depth: Some projects show big differences between ranks in the scheme of the conventional methodology and ranks in the scheme of our proposed methodology. These projects should be more deeply investigated to clarify the reason. If the reason is related to a specific project characteristic, our methodology must be elaborated or modified to include the specific characteristic and its relationship to the overall project performance.
Choquet's fuzzy integral for project 4 example discrimination, construction companies can more sensitively evaluate overall project performance or project success. Because the improvement in performance discrimination mostly comes from application of the utility function, in this case study, the performance discrimination of the alternative methodology is similar to that of the proposed methodology.
Furthermore, the proposed methodology yields project ranks that are more valid than those produced by the conventional methodology because the proposed methodology takes into account the interaction of the three sub-indicators. In Figure 5, which shows the difference between the ranks based on the two different methodologies, the dots above line A represent over-estimated projects due to the interaction of the sub-indicators, while the dots under line A represent under-estimated projects in the schemes of the conventional methodology. In addition, B represents eight projects (projects 6, 8, 18, 21, 26, 35, 39, and 40) that are placed at the same rank (tie-15th) according to the conventional methodology. Using our proposed methodology, those eight projects  Table 3 Fig. 5. Comparison of project ranks by the conventional methodology and project ranks by the proposed methodology

Conclusions
The conventional methodology of the overall project performance evaluation in construction organizations consists of a combination of categorical scale, budget allocation, and additive aggregation function. Combined with the characteristics of sub-indicators of construction projects, this set of methods causes the indiscrimination problem (i.e. the degree of performance discrimination is low because of the low resolution of measurement) and the redundancy problem (i.e. an evaluation is incorrect because of a redundancy in two interrelated subindicators). Although many methods for normalization, weighting, and aggregation exist for the development of a composite indicator, research efforts to guide evaluators in choosing an appropriate set of methods that address these problems are still lacking.
To address these problems in evaluating overall project performance, we created a novel methodology that utilizes fuzzy theories that includes the following three elements: (1) the utility function for normalizing the values of sub-indicators; (2) the fuzzy measure for weighting the sub-indicators; and (3) the fuzzy integral for aggregating the sub-indicator values. We conducted a retrospective case study using 52 real construction projects to demonstrate that the proposed methodology can help alleviate the indiscrimination and redundancy problems: the proposed methodology significantly improves the performance discrimination among different projects (from 0.29 to 0.92) and changes the ranks of under-or over-estimated projects that was caused by the interrelated sub-indicators. With the development of a computational tool to reduce the burden of calculation, our proposed methodology can contribute the more accurate evaluation of overall project performance with higher degrees of performance discrimination.
Experts' feedback provides key research issues to be further studied for being used in real life application: taking into account different characteristics of projects and investigating under-and over-estimated projects in depth. In addition, while this research uses real project performance data, only three sub-indicators related to cost performance were tested. Thus, future research to expand the types of sub-indicators including qualitative ones is required. The use of a non-linear utility function that represents the preferences of construction organizations would also produce more appropriate normalization results.