A Multiple CriteriA Sorting Methodology with Multiple ClASSifiCAtion CriteriA And An AppliCAtion to Country riSk evAluAtion with

. in this paper, we propose an extension of the standard UtADiS methodology, an approach that originates from multicriteria decision aid (McDA) for sorting problems, such that it can handle more than one classification criteria simultaneously which possibly involves different predefined classes for alternatives. Moreover, we test the classification ability of the standard UtADiS methodology using the out-of-classification criterion approach, a new variant of the studies comprising out-of-time and out-of-sample testing methodologies. Results obtained in out-of-classification criterion testing are then compared with the classification ability of the Multiple classification criteria UtADiS (Mcc UtADiS). Finally, an application to country risk evaluation is performed. in this application, classifications of two credit rating agencies, Standard & Poor’s and Moody’s, are taken as two different classification criteria. Moreover, robustness of Mcc UtADiS method is tested through using several data sets. Results indicate that Mcc UtADiS involving more than one classification criteria performs very close to standard UtADiS with single classification criterion and performs better than the out-of-classification criterion tests. These results emphasize both the sensitivity of UtADiS models to the classification criteria and the importance of using a multiple classification criteria approach.

in solving this model. First group involves evaluation criteria y j , (j = 1,...,n), values for each alternative and the second group includes classification criteria values for each alternative. The standard UtADiS method involves n evaluation criteria and a single classification criterion which includes the predefined classes of alternatives. However, in a real world decision problem, sometimes it could be inevitable to take into consideration more than one classification criteria simultaneously. Our paper addresses an extension of UtADiS method in the case of more than one classification criterion which possibly involves different predefined classes for alternatives.
A typical real world problem of this type is country risk evaluation. in a country risk evaluation study of Doumpos and Zopounidis (2001), income levels of countries are used as the classification criterion. The classification criterion is obtained from World Bank and involves four groups; high-income economies, upper middle economies, lower-middle income economies, and low-income economies. However, credit rating agencies such as Moody's, investor Service, Standard & Poor's and Fitch also publish country ratings in a manner suitable for ordered classification (AAA to c). All of these companies are known as being trustworthy organizations and set the industry standards in country risk ratings. However, there are differences between their country classification ratings and it is not possible to decide which one is better than the other. if classification scheme of any one of these agencies was used as classification criterion instead of World Bank classification, it would be possible to obtain completely different sorting classifications of countries. For an investor, evaluating the credit risk ratings, it would be an important issue to take more than one institutions' rating into consideration simultaneously.
Although we focus our attention on the problem of country risk evaluation, our approach can also be applied to various fields such as economics, finance, management, etc., in which many experts, agencies or publishing houses provide results of their own classifications or rankings. Another real world problem example of this type is university rankings. There exist numerous studies on single classification criteria ranking and sorting in university programs (Keeney et al. 2006;Köksalan et al. 2010). Various organizations and publishing houses, such as Financial times, The Wall Street Journal, Business Week, the Economist, US news and World Report, Paristech, Academic Ranking of World Universities (ARWU), Research Assessment Exercise (RAE), Webometrics regularly publish rankings of universities, colleges, programs, etc. The published rankings have become very important to universities competing for research and education fields. For a decision maker, evaluating the university rankings, it would be an important issue to take more than one institutions' rating into consideration simultaneously.
in additon to the country risk evaluation and university rankings, other application areas of our approach can be listed as the evaluation for grants and loans, assessment of energy policies, analysis of credit risk and business failure in which, different decision analysts may define different; but at the same time, rational sets of classes for the conducted analysis. For instance, the credit risk assessment and the business failure prediction studies (Zopounidis, Doumpos 1999b; comprise predefined classification of financial managers in banking institutions which could result in different classifications from the view point of different financial managers or different financial institutions. The relation between our proposed approach and the group decision making for sorting decisions should also be discussed. According to Jelassi et al. (1990), McDA group decision process is intensely difficult due to the ill-structured, dynamic environment and the presence of multiple decision makers, each one of them having his or her own viewpoint on the way the problem should be handled and the decision to be made. There exist a limited number of studies that deal with sorting decision problematic in the context of group decision-making (Dias, climaco 2000;Jabeur, Martel 2007;Damart et al. 2007). McDA group decision methods generally aim to achieve consensus between the group members in handling interpersonal conflicts.
As a first impression, our proposed approach, that includes the idea of building a model taking into account multiple different assignments of the alternatives, also looks like a potentially appropriate tool for the group decision making process in sorting decisions. However, our aim in this study is slightly different. instead of trying to obtain a consensus between decision makers in group decision making perspective, we try to highlight the other dimension of the decision making process from the viewpoint of alternatives and third parties.
in today's dynamic and competitive environment, alternatives and third parties would also like to position their strategies according to the outcome of the decision making process. For instance in a country risk evaluation problem, countries as an alternatives of the problem and the international investors as third parties would want to understand the general convergence of the decision model including all the credit rating agencies' classifications simultaneously. They wish to know the weights of the evaluation criteria and scores of the countries simultaneously satisfying both credit rating companies' classifications. in that respect, they neither try to obtain a consensus between decision makers nor evaluate the equivalence of their corresponding classes. countries simply accept two rating agencies and the others as trustworthy institutions and try to adjust their evaluation criteria values so that their final score reaches the optimal level by satisfying two companies' classifications simultaneously. Therefore, our methodology, which is not designed as a group decision making tool, does not treat the multiple assignments as a whole, in other words, does not try to aggregate the different classifications explicitly. instead, it implicitly tries to find a utility function that minimizes the errors summed over all assignments, in order to reveal the general behaviour of the decision model in case of more than one classification criteria.
The effect of our approach can also be seen on the university classification problem. Universities as alternatives and the potential students as third parties generally would want to understand the general convergence of the decision model including all the university rating institutions' classifications simultaneously. Universities would always ask for the optimal evaluation criteria weight set in order to position in a reasonable class satisfying all institutions' classifications simultaneously. Students would also want to apply for a university that has a reasonably good class in more than one institution's classification. in our knowledge, our proposed study is the only research which tries to highlight this dimension of the decision making process from the viewpoint of alternatives and third parties as explained above. Our approach can also be generalized for supporting sorting type of group decisions by treating the multiple assignments as a whole. However at this point, this is out of the scope of our study.
The contribution of our paper is threefold. Firstly, the paper addresses the extension of the standard UtADiS methodology such that it can handle more than one classification criteria, which possibly involve different predefined classes for alternatives. The model still performs the analysis with minimum total classification error with multiple classification criteria. As a result of this extension, a linear programming based UtADiS methodology turns into a goal programming based multiple criteria sorting methodology which we named as Multiple classification criteria (Mcc) UtADiS throughout the paper. Secondly, the classification ability of the standard UtADiS methodology is tested using the out-of-classification criterion testing approach, which is a new variant of the studies comprising out-of-time and out-of-sample testing methodologies. Within the context of our study, results obtained in out-of-classification criterion testing are then compared with the classification ability of the Mcc UtADiS. Finally, various applications of Mcc UtADiS model are performed in country risk evaluation using two different classification criteria. in order to highlight the importance and necessity of multiple classification criteria based UtADiS methodology, we use country risk classifications of Standard & Poor's and Moody's as two different classification criteria and obtain UtADiS based sorting with minimum classification error covering both criteria. to check and verify the robustness of Mcc UtADiS methodology, we also repeat the country risk application with five different data sets including different evaluation criteria, varying number of countries, different classification criteria and different number of predefined classifications.
The paper is organized as follows. Section 1 briefly reviews the literature on UtADiS in a classified manner. in section 2, we introduce an extended version of UtADiS method; Mcc UtADiS. We also discuss the various limitations of the proposed model and define the out-of-classification criteria testing approach in this part. Section 3 provides an application of Mcc UtADiS on country risk rating evaluation. in sections 4 and 6, we test the robustness of Mcc UtADiS method using additional data sets and scenarios of classification criteria including varying correlations between them, respectively. The final section presents conclusions and recommendations for further research.

literature on utAdiS
Within the context of UtADiS, several studies have been performed over the last two decades. These studies can be grouped under the five different sub research streams, namely; application oriented studies of UtADiS, comparative studies with other techniques, model validation studies, development of model variations and combined models/DSS development studies. Using this classification scheme, the literature on UtADiS is presented in this section. Some studies fall into more than one group, due to multi stream contributions of these studies.

Application oriented studies
There are several applications of UtADiS methodology to various fields in the literature. The identification of acquisition targets in the EU banking industry (Pasiouras et al. 2007a), the reproduction of the auditors' opinion on the financial statements of the firms (Pasiouras et al. 2007b), development of credit risk assessment models for financial institutions using publicly available financial data (Baourakis et al. 2009) and development of classification models that could assist auditors in their decision to issue a qualified or unqualified opinion during the auditing of EU credit institutions (Gaganis et al. 2006) are some recent examples of these studies. The evaluation of the performance of mutual funds (Pendaraki et al. 2005), replication of the credit ratings issued by a rating agency (Doumpos, Pasiouras 2005), investigation of the relationship between client performance measures and the auditors' qualification decisions (Spathis et al. 2003), evaluation of credit applications in shipping industry (Dimitras et al. 2002), detecting falsified financial statements (Spathis et al. 2002), evaluation of Greek industrial SMEs' performance (Voulgaris et al. 2000), energy analysis and policy making (Diakoulakia et al. 1999), business failure prediction (Zopounidis, Doumpos 1999a), bankruptcy risk and business failure prediction (Zopounidis, Doumpos 1999b), are other application oriented studies of UtADiS methodology.

Model validation studies
Out-of-sample testing denotes the expected performance of the model on alternatives different from the ones used in model development and the out-of-time testing represents the expected performance of the model on future data for the same alternatives used in model development. numerous studies have been conducted to show the robustness of the UtADiS methodology using out-of-time and out-of-sample testing (Pasiouras et al. 2007b;Doumpos, Pasiouras 2005). Moreover, in order to investigate the performance of the UtADiS method several validation tests are conducted using the cross-validation approach (Pendaraki et al. 2005) and Jackknife model validation approach (Spathis et al. 2002). Results of these studies indicate that UtADiS methodology is always efficient in validation tests as well.

development of the model variations
Recently, several new variants of the original UtADiS method have been proposed (UtADiS i, ii, iii) to consider different optimality criteria during the development of the additive utility classification model Zopounidis, Doumpos 1999a;1997). M.H.DiS may also be considered as a variant of UtADiS. M.H.DiS method uses a hierarchical procedure in classifying the alternatives into the predefined classes Doumpos, Zopounidis 2001).

Combined models/dSS development studies
An integrated methodological framework, including UtADiS and goal programming, is developed by Penderaki (Pendaraki et al. 2005). Besides, PREFDiS (PREFerence DiScrimination) multicriteria decision support system to study sorting decision problems is an example of DSS based study. The model base of PREFDiS involves four McDA methods, namely the UtADiS method and three of its variants, UtADiS i, UtADiS ii and UtADiS iii (Zopounidis, Doumpos 2000).

Standard utAdiS methodology
in this section, we provide the methodology of standard UtADiS. in order to maintain the compatibility with standard UtADiS methodology, we use the model developed in the study of Zopounidis and Doumpos (1999b) and carry out the necessary modifications on this model including notational changes as well.
The UtADiS method is developed for the classification of a set of alternatives x i , i = 1,...,m, with evaluation criteria y j , j = 1,...,n, into predefined classes of classification criteria c k , k = 1,...,o (c 1 and c o contains most and least preferred alternatives, respectively). This classification is obtained by constructing an additive utility function U(x i ) and utility thresholds u k , k = 1,...,o-1, such that x i is assigned to class k c′ if u k <= U(x i ) < u k1 . Essentially, UtADiS based optimization procedure develops new classes 1 k c k ′ ⋅ =,...,o, in which the boundaries of the classes are formed by utility thresholds u k (here u 0 = 1 and u k = 0). let * j y and * j y be the less and the most preferred values of each evaluation criterion j y for all the alternatives. We denote weight of each criterion as μ j and marginal utility as j u′. j u′ is transformed as j (1) The global utility of an alternative x with all criteria is then: next, an optimization model is formulated to determine the marginal utility functions u j [y j (x)], criterion weights j µ , and the utility thresholds k u . This optimization problem is a linear programming model due to piecewise linear approximation in marginal utility calculations and the objective of this model is the minimization of the classification errors.
There are two possible misclassification errors as a result of optimization process; the over estimation error These two types of potential errors are better shown in Figure 2 for a three predefined classification group example. if the optimization procedure classifies an alternative to a lower class than the predefined class of this alternative then an over estimation error occurs. Similarly, if the optimization procedure classifies an alternative to a higher class than the predefined class of this alternative then an under estimation error occurs. Essentially, UtADiS based optimization procedure develops new classes k c′ , k = 1,...,o, in which the boundaries of these new classes are formed by utility thresholds u k (here; u 0 = 1 and u k = 0). if the predefined class and the model based class of an alternative is the same then there is no misclassification error for this alternative. Otherwise, the over estimation or the under estimation error occurs according to the methodology explained above. Fig. 2. Representation of misclassification errors for a predefined three classification group example The lP model for UtADiS can be expressed as the following: Objective function of the model minimizes the sum of all the misclassification errors. Furthermore, h k represents the number of alternatives in class c k . constraints (4)-(7) compare each utility ( ) i U x with the corresponding utility thresholds k u and defines the classification errors ( ) γ is a small positive number and provides that threshold 1 k u + is greater than k u . This optimization model determines the values of three variable groups, namely; alternative utilities ( ) i U x , criterion weights j µ , and the utility thresholds k u .

Multiple classification criteria (MCC) utAdiS methodology
in this part, we present the general concept of Multiple classification criteria (Mcc) UtADiS method and construct the model. As previously stated, the standard UtADiS method involves n evaluation criteria and one classification criterion that includes the predefined classes of alternatives. However, the Mcc UtADiS method includes more than one classification criteria, which possibly involve different predefined classes for alternatives. We define c r , r = 1,...,q as different classification criteria and c kr , k = 1,...,o, r = 1,...,q denotes the predefined classes of each classification criterion (c 1r and c or contains most and least preferred alternatives for each classification criterion, respectively). Similar to the standard UtADiS method, the classification of alternatives using Mcc UtADiS method is obtained by constructing an additive utility function U(x i ) and the multiple classification class utility thresholds um k , k = 1,...,o-1, such that x i is assigned to multiple classification criteria class k cm′ if um k <= U(x i ) < um k-1 . There are two possible misclassification errors as a result of optimization process; the over estimation error ..,q. These two types of potential errors are better demonstrated in Figure 3, which represents a hypothetical example with two classification criteria and three predefined classification groups for each of the two classification criteria. Figure 3a and 3b shows the standard UtADiS for each of the classification criteria separately. There are two different sets of thresholds u kr for each separate UtADiS including single classification criteria, as it can be easily seen from the figure; where Figure 3c represents the Mcc UtADiS methodology. in this case, a single set of thresholds um k is obtained by Mcc UtADiS methodology by taking into account two classification criteria simultaneously with minimum total misclassification error. As a result of this extension, a linear programming based standard UtADiS methodology turns into a goal programming based on multiple criteria sorting methodology.  The goal programming model of Multiple classification criteria (Mcc) UtADiS method can be expressed as following: in a similar manner with standard UtADiS model, objective function of the model minimizes the sum of all the misclassification errors. But in this generalized case, total misclassification error is obtained by taking into account two classification criteria simultaneously. By the way, it is possible to assign weight factor z r to a specific classification criterion in order to increase the importance of the criterion. Therefore, a linear programming based UtADiS methodology turns into a goal programming based multiple criteria sorting methodology. h kr represents the number of alternatives in class c kr . constraints (12)

discussions about the possible limitations of the MCC utAdiS approach
Our methodology finds a utility function that minimizes the errors summed over all assignments. One can argue that the inconsistencies in the assignments are not really taken explicitly into account. in other words, if apparent inconsistencies are involved between classification criteria, the probability of meaningful analysis would decreases. Actually, this is a general problem of whole family of McDA sorting models, when you take into consideration all the possible classifications as a whole. if one would apply UtADiS to the same alternatives and criteria but with a different predefined classification, the results can differ. in other words, using standard UtADiS model, someone may solve a problem with a predefined classification, obtain high levels of prediction and explanation abilities and conclude that the model has high explanation ability. But when the same model is solved with a different predefined classification, which could also be treated as an important classification criterion and should also be taken into consideration simultaneously, completely different results might be obtained if there are inconsistencies between classification criteria. At that point, our proposed model, at least, highlights this sensitivity of UtADiS model and proposes a potential extension to take different classifications into consideration. Additionally, if the inconsistencies between different assignments are high, as a natural result of this inconsistency, the explanation of Mcc UtADiS would be low as well.
Moreover, although the explanation ability of the model decreases as inconsistency between classifications increases, there would be significant amount of real world cases including consistent classification criteria as we mention in the first section. Finally, for some instances, although certain inconsistencies exist, the decision maker might still want to take into account all the classificiation criteria as a whole simultaneously. This situation could be inevitable especially in multi discipline group decision making processes. in order to show the effect of the inconsistency between classifications, we test Mcc UtADiS with varying correlations between two classification criteria in the last section.
Another important point that should be discussed is the number of classes in each classification criteria. Although the classification criteria contain the same number of classes in explanatory example in Fig. 3, our methodology also handles problems including classification criteria with different number of classes. According to our methodology, if the number of classes is not equal, our model uses the highest number of classes as a number of classes. For example, if we have two classification criteria with 3 and 2 classes respectively. Our approach assumes 3-class problem and third class of second classification criterion, which does not exist actually, is taken as a class with no alternative. This approach is better explained on Figure 4.
Abovementioned approach is just one of the possible alternatives of handling the problem of different number of classes. Actually, if classification criteria include different number of classes, the approach that can be employed directly depends on the problem's nature and the decision maker's preferences. if the number of classes is not equal, the problem can also be overcome by mapping a set of classes from one classification criterion into a single class in the other criterion, before solving the mathematical model. in order to allow the explicit description of the scope of the problem, user is also allowed to choose the number of classes to which he wishes to assign alternatives (e.g. if one criterion contains 3 classes and the other 6 classes, user can choose whether the final assignments should consider 3 or 6 classes).
As a natural extension of the problem of different number of classes, we should also discuss the equivalence of the same level of classes in different classification criteria. Even if there exists the same number of classes for each classification criterion, one can argue that the first class of the first classification criterion is not equal to the first class of the second classification criterion. Actually, again this is a general problem of whole family of McDA group decision models, when you take into consideration all the possible decision maker's class perception attitudes.

out-of-classification criterion testing
The standard UtADiS model is highly sensitive to predefined classification of the alternatives. in other words, a standard UtADiS model formulated and solved using one classification criterion would yield very different results if the model was built by taking another classification criterion into consideration. Additionally, the prediction ability of one model in explaining another model built with same criterion and alternatives but with different predefined classification seems to be considerably limited. to show this limitation we develop out-of-classification criterion testing and discuss its results in detail. numerous studies have been conducted to show the robustness of the UtADiS methodology using out-of-sample and out-of-time testing (Pasiouras et al. 2007b;Doumpos, Pasiouras 2005). in out-of-sample testing, UtADiS model is run using a set of alternatives (training sample). The thresholds k u , and the criteria weights j µ , obtained from this model are then used to test the explanation ability of this model on a different set of alternatives (holdout sample). Similarly, in out-of-time testing, UtADiS model is run using a set of alternatives (training sample) for a certain time period. The thresholds k u , and the criteria weights j µ , obtained from this model are then used to test the explanation ability of this model on the same set of alternatives with different time period (holdout sample). Eventually, out-of-sample testing denotes the expected performance of the model on alternatives different from the ones used in model development and the out-of-time testing defines the expected performance of the model on future data for the same alternatives used in model development.
in our paper, we also test the classification ability of the standard UtADiS methodology using the out-of-classification criterion approach, which can be counted as a new variant of the studies comprising out-of-time and out-of-sample testing methodologies. in out-of-classification criterion testing, standard UtADiS model is run using a certain classification criteria. The thresholds k u , and the criteria weights j µ , obtained from this model are then used to test the explanation ability of this model with different classification criterion. in other words, the thresholds and the weights from a model developed with a specific classification assignment are used to test the model against a different classification assignment. This is equivalent to comparing the recommendations of the model not with the original classification criteria grouping of the alternative, but with a different classification criteria grouping.
The procedure of out-of-classification testing is better shown on Figure 5, which represents a hypothetical example with two classification criteria and three predefined classification groups for each of the two classification criteria. Figure 5 (a1), represents the misclassification  indicates the misclassification errors resulting from standard UtADiS with classification criteria 2. As previously stated, in out-of-classification testing, the thresholds k u , and the criteria weights j µ , obtained from standard UtADiS model are used to test the explanation ability of this model on a different classification criteria. This methodology is shown on Figure 5. Figure 5 (b1) denotes the out-of classification criteria explanation ability of classification criteria 1 (Occ1). The figure shows the thresholds k u , from first classification criteria. However, the misclassification errors are computed using predefined classification groups of the classification criteria 2. Besides, Figure 5 (b2) represents the out-of classification criteria explanation ability of classification criteria 2 (Occ2). This figure shows the thresholds k u , from second classification criteria. Yet, the misclassification errors are computed using predefined classification groups of the classification criteria 1. As it can be seen on the figure, the magnitudes of misclassification errors increase in out-of classification testing cases.
Meanwhile, Figure 6 represents misclassification errrors of the Mcc UtADiS methodology. in this case, a single set of thresholds um k is obtained by Mcc UtADiS methodology by taking into account two classification criteria simultaneously with minimum total misclassification error.

An application of MCC utAdiS on country risk rating evaluation
According to Krayenbuehl (1985), country risk refers broadly to the likelihood that a sovereign state or borrower from a particular country may be unable and/or unwilling to fulfil their obligations towards one or more foreign lenders and/or investors. country risk has become a topic of major concern for the international financial community over the last two decades. The importance of country ratings is underscored by the existence of several major country risk rating agencies, namely the Economist intelligence Unit, Euromoney, institutional investor, international country Risk Guide, Moody's, Political Risk Services, and Standard and Poor's. These risk rating agencies employ different methods to determine country risk ratings, combining a range of qualitative and quantitative information regarding alternative measures of economic, financial and political risk into associated composite risk ratings. However, the accuracy of any risk rating agency in terms of any or all of these measures is open to question (Hoti, McAleer 2004).
Moody's country risk rating is defined as a measure of the ability and willingness of a country's central bank to provide foreign currency to service the foreign debt held by the government and other borrowers residing in that country. This rating is not a direct evaluation of the creditworthiness of the government, but rather an assessment of the foreign liabilities of the country as a whole. Unlike Moody's, Standard & Poor's defines its country risk rating as a measure of a government's ability and willingness to repay debt according to its terms. Standard & Poor's ratings are sovereign ratings as they address the credit risk of the government and not of the other borrowers of a country (Howell 2001). country risk evaluation, which involves different assessment of countries by various institutions, can be seen as an appropriate application area to test the validity of our proposed model. With the following applications, we aim to test the use of UtADiS as a sorting methodology in the case of more than one classification criteria for the alternatives. For this purpose, we use the country ratings of two international credit rating agencies, namely; Standard & Poor's and Moody's, as classification criterion for the countries which are taken as alternatives.
in this application, firstly, standard UtADiS models are developed and solved for a data set composed of 31 countries and 10 criteria by taking the Standard & Poor's and Moody's ratings separately as classification criterion. Then, a Mcc UtADiS model described in section 2 is developed and solved by taking the ratings of these agencies as two classification criteria. The results obtained in Mcc UtADiS approach are compared with those of standard UtADiS models.
More than 20 classification groups exist both in S&P and Moody's rating system. We reduce this large number of classification groups by combining them into 3 sub groups for simplification purposes. According to our classification approach used throughout the analysis, countries with credit ratings AAA, AA+, AA and AA-in the Standard & Poor's rating system are classified as class 1 countries. Similarly, for the Moody's rating system, countries with Aaa, Aa1, Aa2 and Aa3 rating scores are considered as class 1 countries. The agencies' ratings and their matches with our classifications used throughout the analysis are detailed in table 1. The table demonstrates that, three predefined classes (class 1, class 2 and class 3) are used for each classification. comparative ratings table of the different rating agencies is obtained from credit Suisse (2009) web site.
For our first data set consisting of 31 countries, UtADiS analysis is performed by taking into account 10 evaluation criteria and 2 classification criteria. These 10 evaluation criteria and their preference characteristics are listed in table 2. The term "Max" indicates that higher values of that criterion are prefered. in contrast, the term "Min" means that lower values of the criterion is favoured. As seen in table 2, inflation and external debt criteria are taken as "Min" criteria, which means that for the alternatives, the negative values of these criteria are used during the analysis.
As the first part of our analysis, two standard UtADiS models are developed and solved by taking each classification criterion into consideration separately. in other words, classifications obtained through Standard & Poor's ratings are taken as classification criterion and one model is solved according to this classification (henceforth we call this model as, cc1 UtADiS), then The predefined classes and the classes assigned by the model for each country are listed in Classes and Utility Scores columns of table 3 respectively. in Mcc UtADiS column, c ij denotes the classification of countries. Here, index i represents the predefined class of the country, whereas j represents the classification criterion. Our proposed model, Mcc UtADiS, takes both classifications into consideration simultaneously rather than assigning one class to each alternative. Actually, this is one of the origin points of our study as "What if there is more than one predefined classification for each alternative?". For some alternatives, classifications are the same according to both Standard & Poor's and Moody's classification, however, some alternatives belong to different classes with respect to classification criteria. As an example, Switzerland with c 11 , c 12 belongs to a class 1 according to both Standard & Poor's and Moody's ratings. On the other hand, Venezuela belongs to class 2 for Standard & Poor's based predefined classification, whereas it belongs to class 3 for Moody's based predefined classification as shown in table 3.
According to table 3, the utility threshold values are obtained as 0.3390 and 0.3011 for cc1 UtADiS Model. in the second standard UtADiS model, which is constructed using Moody's ratings as classification criterion, the utility threshold values are 0.3215 and 0.2919 respectively. in addition, with Mcc UtADiS approach, we obtain utility threshold values of 0.3233 and 0.2847.
When the utility scores in table 3 are analysed, one can easily conclude that there are some countries which are misclassified with respect to predefined classification. For example, Malaysia, which is a class 2 country in both Standard & Poor's and Moody's classifications, belongs to class 1 according to its utility score obtained through the models. These types of classification errors yield a total error score different than 0. This brings up the prediction ability and the explanation ability issues of the models. Here, the prediction ability refers to the degree of consistency between the predefined classes and classes assigned by the model. nevertheless, the explanation ability means the success of a model on explaining another model constructed with different classification criterion.
We measure the prediction ability of models through the comparision of classes assigned by the model with the predefined classes. For this purpose, we count the number of alternatives that have a utility score in accordance with their predefined groups and the number of misclassified alternatives.
While evaluating the explanation ability, we perform out-of-classification criterion testing mentioned in the Section 2. For our application, Occ1 model represents thresholds and the criteria weights obtained from cc1 model and the explanation ability of this model is then tested on a classification criterion 2. Similarly, Occ2 model represent thresholds and the criteria weights obtained from cc2 model and the explanation ability of this model is then tested on a classification criterion 1. After determining the correct predictions and mispredictions, we obtain the explanation abilities of the models as shown in table 4. it should also be mentioned that in Mcc UtADiS model, there is no single "original class" for each alternative; instead they have two predefined classifications according to each classification criteria separately. Therefore, Mcc UtADiS rows of the table 4 include original classes for each classification criterion separately. Although the data set includes 31 countries, the total number of countries in Mcc UtADiS rows of the table 4 is 62, which represent the original classes of 31 countries for two classification criteria separately. Our Mcc UtADiS model classifies Venezuella in class 3 in table 3. According to this classification, Venezuella is taken as misclassified for classificiation criterion one in which Venezuela is originally classified in class 2. On the other hand, the same country is taken as correctly classified for classificiation criterion two in which Venezuela is originally classified in class 3 as well.
in table 4, prediction and explanation ability scores for five different models are shown. Overall prediction and explanation ability scores of cc1, cc2, Mcc, Occ1 and Occ2 models are computed as 90%, 84%, 86%, 81% and 74% respectively. Results indicate that, explanation ability scores for the out-of-classification criterion tests are lower than the prediction ability scores of other models. This means that one model obtained through one classification criterion fails in explaining another model with same alternatives and evalu- ation criteria but with different classification criteria. However, combined use of multiple classification criteria, Mcc UtADiS approach, obtains close prediction ability scores with the standard separate models. Similar supportive results about the performance of Mcc UtADiS are obtained by evaluating the overall error values of the models, which are objective function values. As Mcc UtADiS methodology deals with two classification criteria simultaneously, the error function of this model includes error terms from both classifications. in other words, threshold values and utility scores of Mcc UtADiS are assigned to the alternatives by taking all predefined classifications into account at the same time by minimizing the classification errors. Therefore, the results of Mcc UtADiS implicitly include the holdout and the training results. So it is appropriate to compare the error value of this model with the sum of errors separately obtained from cc1 and Occ1 UtADiS models that give us the overall error of the classification criterion 1. Here, the error value of cc1 represents the prediction ability with one classification criterion and the error value of Occ1 represents the explanation ability of that classification criterion over the other classification criterion. in a similar manner, error values of cc2 and Occ2 are summed up to give us the overall error of classification criterion 2.
Overall error and performance values of our three basic analyses are given in table 5. Overall error of Mcc UtADiS model is obtained as 0.015 whereas overall errors of classificiation criterion 1 and 2 including the out-of-classification criterion tests as well are found as 0.017 and 0.018 respectively. When we combine these two errors by taking the average of them, we simply obtain the overall error of standard UtADiS including the training and holdout samples of both criteria and we will be able to compare the overall performance of standard UtADiS and Mcc UtADiS for the same data sets. Similar to prediction and explanation ability scores, error values also show that the overall error value of Mcc UtADiS approach is lower than those of classificiation criterion 1 and 2 including out-of-classification criterion tests. This means that Mcc UtADiS approach, which takes all classifications into consideration, yields lower error values and better explanation abilities.
Through UtADiS analysis, we also estimate the weights of the evaluation criteria, which indicate their contribution to the classification. table 6 shows the weights of the evaluation criteria in three models. The weights of the evaluation criteria exhibit a similar structure for all three models. The most important criterion in cc1 UtADiS, cc2 UtADiS and Mcc UtADiS models is criterion 4 (current Account Balance/GDP) with a weight of 49.91%, 47.19% and 49.26% respectively.  in table 8, the utility threshold values obtained through the three models are listed. As previously mentioned, data set 5 includes 4 and 3 predefined classes for classification criteria respectively. Therefore, for this data set, cc1 and cc2 have 3 and 2 thresholds respectively. According to our methodology, if the number of classes is not equal, our model uses the highest number of classes as a number of classes. Because of this approach, Mcc UtADiS comprises 3 threshold levels. When the prediction and explanation ability of models obtained in different data sets is examined, one encounters a similar situation with our original data set. in all data sets, Mcc UtADiS model yields close prediction ability to our separate standard models and better prediction ability scores than the explanation ability scores of out-of-classification criterion (Occ) tests. table 9 exhibits prediction and explanation ability scores for data sets in a detailed manner. The number of alternatives in each predefined class for each data set is also provided in this table.  Figure 7. in all data sets, overall error of the Mcc UtADiS model is significantly lower than the overall error of the standard UtADiS model. For the first data set, overall error of the standard UtADiS is 9.7% higher than the Mcc UtADiS. For the remaining four data sets, same situation repeats with the values of 16.3%, 8.9%, 46.7% and 129.2% error differences respectively. These results supports the findings of the previous section so that, Mcc UtADiS approach which takes all classifications into consideration simultaneously, yields lower error values and better explanation abilities.

An analysis of MCC utAdiS with varying correlations between two classification criteria
in this section, we test the applicability of Mcc UtADiS with varying correlations between two classification criteria. Our aim is to examine the performance of the Mcc UtADiS method as opposed to standard UtADiS in case of more than one classification criteria for varying consistency levels.We assume two decision makers' classifications on a hypothetical example with 12 alternatives and 10 evaluation criteria as shown in table 12. in that sense, the consideration of any additional information is only expected to change the classification results of the Mcc UtADiS and standard UtADiS in a similar manner, while there is no theoretical evidence that the incorporation of some specific additional variables will improve the performance of one method and decrease the performance of the other (Zopounidis, Doumpos 1999a).  in order to obtain a complete set of problem instances, we generate 7 different scenarios by changing the correlation coefficient between decision makers' predefined classifications such that, correlation coefficents are distributed between 1 and -1 as shown in table 13. in this table, DM1 and DM2 columns represent the predefined classifications of two decision makers for the abovementioned hypotetical example.
in table 14, the utility threshold values obtained through the three models for 7 scenarios are listed. The utility threshold values of cc1 models are the same for all scenarios due to the same classification scheme of the DM1. On the other hand, the utility threshold values of cc2 models, which represent the predefined classifications of the DM2, vary between 0.267 and 0.700. table 15 summarizes the overall prediction and explanation ability scores computed through weighted averages of performance scores for each classification. Here, the averages of the overall performances of the two standard UtADiS models including holdout and training samples are compared with the Mcc UtADiS model. As the inconsistency between two classification criteria increases, the overall prediction ability of both approaches decreases. However, in all scenarios, the performance of the Mcc UtADiS model is at least the same or better than the overall performance of the standard UtADiS.
Similar supportive results about the performance of Mcc UtADiS are obtained by evaluating the overall error values of the models, which are objective function values. Overall error values of all data sets are given in table 16. in all scenarios, overall error of the Mcc UtADiS model is significantly lower than the overall error of the standard UtADiS model. These results supports the findings of the previous two sections so that, Mcc UtADiS approach which takes all classifications into consideration simultaneously, yields significantly lower error values and better explanation abilities.

Conclusions
The primary aim of this study was to propose an extended version of UtADiS. This methodology handles more than one classification criteria, which possibly involve different predefined classes for alternatives. We call this extended version as Mcc UtADiS Model. Similar to the standard model, Mcc UtADiS model also performs the analysis with minimum total classification error. nevertheless, it takes more than one classifications into consideration in a simultaneous manner. Second aim of the study was to test the classification ability of UtADiS methodology through out-of-classification criterion approach, which is a new variant of the studies comprising out-of-time and out-of-sample test methodologies. Within the context of our study, results obtained in out-of-classification criterion testing are then compared with the prediction ability results of Mcc UtADiS.
in addition, various applications of Mcc UtADiS model were performed in country risk evaluation using two classification criteria. We tested Mcc UtADiS approach using real world data. Throughout our analysis, we used two classification criteria based on the credit risk ratings of two international agencies (Standard & Poor's and Moody's). We also worked with multiple data sets including different number of evaluation criteria, varying number of countries, different classification criteria and different number of predefined classes in order to test the robustness of the model. in general, explanation ability scores for the out-of-classification criterion tests were lower than the prediction ability scores of other models. This means that one model obtained through one classification criterion fails in explaining another model with same alternatives and evaluation criteria but with different classification criteria. However, combined use of multiple classification criteria, Mcc UtADiS approach obtained very close prediction ability scores with the standard separate models. These results emphasize both the sensitivity of UtADiS models to the classification criteria and the importance of using a multiple classification criteria approach. Similar supportive results are achieved by evaluating the overall error values of the models, which are objective function values. Eventually, we concluded that rather than explaining two models with different classification criteria using each others' thresholds and weights, Mcc UtADiS models deal with both classifications simultaneously and make more accurate classifications.
We also tested the applicability of Mcc UtADiS with varying correlations between two classification criteria. Our aim was to examine the applicability of the Mcc UtADiS method as opposed to standard UtADiS in case of more than one classification criteria for varying consistency levels. Results of this part showed that, as the inconsistency between two classification criteria increases, the overall prediction abilitiy of both the standard UtADiS and the Mcc UtADiS approaches decreases. However, in all data sets, the performance of the Mcc UtADiS model yielded significantly lower error values and better explanation abilities than standard UtADiS.
The current research may be extended towards various directions. First of all, our approach can be generalized for supporting sorting type of group decisions by treating the multiple assignments as a whole. Furthermore, additional applications of multiple classification criteria type problems are recommended to be analysed. Moreover, the country risk evaluation application may be extended towards the inclusion of additional evaluation criteria and/or classification criteria. Finally, comparative studies with similar techniques can be achieved, in order to validate the performance of Mcc UtADiS.