NEW MCEQLS FUZZY AHP METHODOLOGY FOR EVALUATING LEARNING REPOSITORIES: A TOOL FOR TECHNOLOGICAL DEVELOPMENT OF ECONOMY

. The paper aims to present a new methodology to evaluate the quality of features and functionality of learning object repositories (LORs). The quality of features and functionality of LORs is analysed in terms of engaging LOR users and content producers. Thus, it can be referred to as quality-in-use of LORs. This methodology consists of creation and consequent application of methods and the model for the quality-in-use of LORs. The model of the quality-in-use of LORs is presented in this paper. The methodology for evaluating the quality-in-use of LORs is based on the general MCEQLS (Multiple Criteria Evaluation of the Quality of Learning Software) approach to evaluate the quality of learning software. The essential part of the novel methodology is the application of improved Fuzzy AHP method to establish criteria weights of the quality-in-use of LORs. It is shown that the created methodology is suitable and stable for evaluating the quality of LOR features and its functionality. A more detail presentation is given on the results of the expert evaluation of the quality-in-use of three LORs that are most popular in Lithuania against the proposed methodology. The novelty of the presented research is achieved through the innovative instrument consisting of the model of the quality-in-use of LORs and the Fuzzy AHP method. The presented methodology could serve as a technological tool for decision making in education as well as in different areas of economy.


Introduction
Learning object repositories (LORs), or learning repositories, are referred to here as properly constituted systems, i.e. organised collections of learning objects (LOs) consisting of learning objects, their metadata and tools/services to manage them (Kurilovas 2013). An LO is referred to here as "any digital resource that can be reused to support learning" (Wiley 2000). Educational authorities and/or companies in many countries have launched web-based LORs in order to make it easier for teachers and students to find the most relevant LOs. An LOR is one of the main parts of an e-learning system and/or environment of any educational institution, and, therefore, the overall quality of learning services highly depends on the quality of LORs.
The main players in the education sector are the educational institutions themselves (schools, universities, etc.), education authorities (ministries of education, regional and other agencies, etc.) and policy makers, as well as providers of the educational software (LOs publishers, producers, etc.). Educational institutions are interested in using high-quality LOR software. Therefore, they need proper approaches, models, and methods for creating or choosing a high-quality LOR software in the market.
Therefore, the problem related to the evaluation of the quality of LORs is high on the agenda of the international research and education systems. On the other hand, the development of high-quality LORs should significantly improve the accessibility and quality of education, which is the main factor of economic development.
According to Zavadskas and Turskis (2008), each alternative in the multi-criteria decision-making problem can be described by a set of criteria. Criteria can be qualitative and quantitative. Usually, they have different units of measurement and a different optimisation direction. According to Gasperovic and Calpinskas (2006), from the technological point of view, we can divide the quality criteria of the learning software into criteria of "internal quality" and criteria of the "quality-in-use". According to Gasperovic and Calpinskas (2006), the "internal quality" is a descriptive characteristic that defines the quality of software independently from any particular context of its use, while the "quality-in-use" is an evaluative characteristic of software obtained by making a judgment based on the criteria that determine the worthiness of software for particular users.
The model of the "quality-in-use" of LORs and its evaluation methods are the main research topics of the paper. According to LOR experts of the largest EU Educational Repositories Network project (EdReNe 2014), it deals with LOR features and functionality.
The rest of the paper is organised as follows. MCEQLS approach is presented in Section 1. The model of the quality-in-use of LORs (criteria system) is presented in Section 2. The Fuzzy AHP (Analytic Hierarchy Process) and criteria weights of the quality in use of LORs are presented in Section 3. Ratings of the criteria and experimental evaluation results are presented in Section 4. Conclusions are provided in the last Section.

MCEQLS approach to evaluation of the quality of learning software
In their previous contributions, the authors (Kurilovas et al. 2011;Kurilovas, Zilinskiene 2013;Kurilovas, Serikoviene 2013) proposed to use the MCEQLS (Multiple Criteria Evaluation of Quality of the Learning Software) approach. The authors showed that the MCEQLS approach can significantly improve the quality of the expert evaluation of learning software. Besides, the MCEQLS approach can significantly reduce cost, time, and human resources necessary for the evaluation process.
While creating MCEQLS, the authors analysed the following scientific methods, requirements, and principles to minimise the aforementioned problems related to the evaluation of the quality of learning software: (1) principles of multiple criteria decision analysis for the identification of quality criteria based on Belton and Stewart (2002); (2) different techniques to identify the weights of criteria; (3) different decision making theories (e.g. Fuzzy and AHP) to obtain final evaluation measures; and, finally, (4) scalarisation, i.e. the experts' additive utility function (Kurilovas et al. 2011).
According to the general MCEQLS approach, in order to practically evaluate the quality-in-use of alternative learning repositories, the experts-evaluators should use several steps: (1) use the LOR quality model constructed with the help of principles of the multiple criteria decision analysis for the identification of quality criteria; (2) establish the weights of LOR quality-in-use criteria according to the normalisation requirement and the Fuzzy AHP method can be used for this purpose; (3) evaluate the alternatives against all the quality criteria; (4) calculate the numerical value of the quality-in-use of a particular LOR alternative using the obtained numerical values (ratings) and the weights of the quality criteria with the help of the experts' additive utility function (Kurilovas et al. 2011).
The higher is the numerical value of the quality of the particular LOR alternative, the higher is the quality of this alternative in comparison with other evaluated alternatives, and the educational institutions should decide on the purchase or creation of this LOR alternative for their educational needs.

Model of the quality-in-use of learning repositories
We performed a systematic review of scientific literature on the quality of LORs in ISI Web of Science database in May 2015. The topic selected for the systematic review was "TS = (quality AND learning object repositories)". 91 references were found incl. 29 Articles, 61 Proceedings papers, and 1 Review. The following research results seem to be most relevant to the topic of our paper. Palavitsinis et al. (2014) present the certification process of the metadata quality assessment for LORs. Clements and Pawlowsky (2012) analysed how teachers as users of open LORs act in the re-use process and how they perceive quality. Based on a quantitative empirical study, they also surveyed, which quality requirements users have and how they would contribute to the quality process. According to Clements and Pawlowsky (2014), resources, organisations, and technologies seem to be of particular importance when looking at quality.
The results obtained by Sanz-Rodriguez et al. (2011) suggest that their measurement of reusability of LOs could constitute an indicator of quality, which would allow searching the results to be ordered, with those with the greatest possibility of being reused taking the priority. Furthermore, the proposed reusability indicator could be calculated automatically or in an assisted way if metadata elements satisfy the identified minimum quality requisites. Koutoumanos et al. (2011) provide an insight into the approach for capitalising on the European and international demand for quality-controlled educational content, describing a framework and the associated quality issues for modelling the publication lifecycle of digital LOs within a federation of LORs. Lattanzio and Sandrini (2011) think that the coexistence of alternative paradigms that drive the evolution of LORs imposes a revision of the quality criteria taken into account, starting from necessities of users (students and teachers') rather than from the requirements typical of technologies and standards.
According to Zschocke and Beniest (2009), studies show that LORs have difficulties in obtaining good quality metadata from their contributors, especially when this process involves many different stakeholders. Sicilia et al. (2005) consider that Semantic Web ontologies can be used to improve the quality of LO metadata records, but they are not enough by themselves. In order to respond to requests by returning the adequate LOs, the repository is required to be aware of the amount, type and quality of metadata records it stores. According to Sicilia et al. (2005), the design of a LOR approach to "semantic lifecycle" is described and illustrated through the concrete architecture of the prototype of the repository.
In terms of the quality-in-use of LORs, these papers only address mainly the quality issues of metadata of LOs. In order to provide a more comprehensive model of the quality-in-use of LORs, partners of the Educational Repositories Network (EdReNe 2014) identified a number of quality criteria for evaluating LOR features and functionality. The main question here is how to engage users in using LORs. The following quality-in-use criteria of LORs are the result of the authors' analysis and aggregation of the criteria identified by EdReNe experts.
One more criterion -remixing content -was also identified by EdReNe experts, but it was noted that building your own content from other repository LOs is something not yet easily feasible in most LORs. Also, the question is whether the combination of LOs should really take place in LORs, but it does not happen in most LORs at the moment. This does not happen e.g. in repositories under consideration further in the experimental section. Therefore, the authors have not analysed this criterion in the current study.

Fuzzy AHP method and weights of the quality-in-use criteria of LORs
There are a number of multiple criteria decision-making (MCDM) methods that could be used in the evaluation of the quality of learning repositories (Zavadskas et al. 2007Zavadskas, Turskis 2010;Brauers, Zavadskas 2012;Razavi Hajiagha et al. 2013;Chakraborty, Zavadskas 2014). All of them have their pros and cons. Previously, we have analysed both MCEQLS Fuzzy (Kurilovas, Serikoviene 2013) and MCEQLS AHP (Kurilovas, Zilinskiene 2013) methods to evaluate the quality of learning objects and learning scenarios, respectively.
In the paper, we improve the Fuzzy AHP method and use it the in MCEQLS approach.

Improved Fuzzy AHP method
According to Saaty (1990), the Analytic Hierarchy Process (AHP) is a useful method for solving complex decision-making problems involving subjective judgment. In AHP, the multi-attribute weight measurement is calculated via pairwise comparison of the relative importance of two factors (Lin 2010). The design of the questionnaire incorporates pairwise comparisons of decision elements within the hierarchical framework. Each evaluator is asked to express the relative importance of two criteria in the same level by a nine-point rating scale. After that, we have to collect the scores of pairwise comparison and form pair-wise comparison matrices for each of the evaluators.
After that, we have to construct a set of pairwise comparison matrices (size n×n) for each of the lower levels with one matrix for each element in the level immediately above by using the relative scale measurement.
The pairwise comparisons are done in terms of which element dominates the other. There are n(n -1)/2 judgments required to develop the set of matrices in this step. Reciprocals are automatically assigned in each pairwise comparison.
Then hierarchical synthesis is used to weight the eigenvectors by the weights of the criteria, and the sum is taken over all weighted eigenvector entries corresponding to those in the next lower level of the hierarchy.
In AHP, the results expert evaluation are vague, i.e. they depend on the qualification, experience, and subjectivity level of experts. If we change the group of experts or try to perform the other evaluation with the same experts, we will often obtain different evaluation results. In the proposed method, the authors apply the Fuzzy sets theory (Kurilovas et al. 2011) oriented towards the rationalisation of uncertainty. The application of uncertainty lets the experts evaluate not only one point but an appropriate range of values. The influence of uncertainty could be evaluated in different ways by applying the theory of Fuzzy numbers or mathematical statistics methods. In MCDM methods, the quality criteria weights have a significant influence on the final result. The Fuzzy logic is widely applied to establish the quality criteria weights because Fuzzy numbers reflect the opinion of the group of experts in a more precise way. Therefore, we propose the Fuzzy AHP method instead of the simple AHP to conclude a group pairwise comparison matrix, using expressionless triangular numbers.
This proposed Fuzzy AHP method is suitable to set the weights of the quality criteria when the experts evaluate alternatives independently of the opinion of other experts. Each expert accomplishes the evaluation procedure applying the simple AHP method of pairwise comparison. The matrix of expert's pairwise comparison is verified if the expert has not conflicted with his/her own opinion. This allows obtaining the weights of the quality criteria in a more precise way.
On the other hand, the analysis of the use of MCDM methods has shown that the methods are more stable when applying the weights established by the Fuzzy AHP method in comparison with the weights established by the simple AHP method, i.e. the average weights of the group. An MCDM method is deemed stable if results of the method application change a little while results of the expert evaluation do not change significantly.
In more detail, the algorithm of the Fuzzy pairwise comparison matrix of the quality criteria of the group of independent experts, the algorithm to determine the stability of MCDM methods, regarding the uncertainty of expert grades and to choose the results of the most stable MCDM method, and analysis of establishing the quality criteria weights using different Fuzzy and AHP methods have been proposed by Vinogradova (2015).
Thus, due to the preciseness and stability of the Fuzzy AHP method, we propose to use it in establishing the quality criteria weights.
Triangular fuzzy numbers are three parameters (l, m, u), which define the quality between 0 and 1 within the membership function (Kurilovas et al. 2011).
The pairwise comparison matrix of expressionless parameters is set by a panel of experts from the individual agreement of experts on pairwise comparison matrices.
The concerted decision matrix is calculated by the experts. When j ≥ i, as the matrix is inverse. The expressionless number of parameters of a triangular matrix of the expert group is set as follows: Applying the advanced method of analysis of the expressionless fusion extension proposed by Chang (1996), in order to calculate the S i value of calculating the weighting set is as follows: The degree of possibility of M 2 = (l 2 , m 2 , u 2 ) ≥ M 1 = (l 1 , m 1 , u 1 ) is expressed as: Let d′(A i ) = min V(S i ≥ S k ), k = 1, 2, …, n; k ≠ n. Then the weight vector is given by W′ = (d′(A 1 ), (d′(A 2 ), …, (d′(A n ) T . The weight vector is normalised to get the normalised weights: W = (d(A 1 ), (d(A 2 ), …, (d(A n )) T .

Application of the Fuzzy AHP method to establish the weights of the quality-in-use criteria of LORs
Seven experienced experts have expressed their opinion on the importance (i.e., weights) of LOR features and functionality according to the proposed Fuzzy AHP method.
Every expert's consistency in the filled pairwise comparison matrix was checked by establishing their Consistency Index and Consistency Ratio (CR). Next, expert matrixes that had the CR < 0.1 were selected, and consistency of the expert group was checked by establishing the Coefficient of Concordance W and c 2 criteria. All expert weights of AHP matrixes were consistent since W = 0.85 and, accordingly, the value of c 2 = 47.66 is significantly higher than the critical value After application of the fundamental scale of absolute numbers for the estimation of the weights of the quality-in-use criteria of LORs presented in Section 3, one expert matrix is as follows: 1.00 2.00 1.00 1.00 2.00 2.00 5.00 2.00 2.00 0.50 1.00 0.50 0.50 1.00 1.00 4.00 1.00 1.00 1.00 2.00 1.00 1.00 0.50 0.50 5.00 2.00 2.00 1.00 2.00 1.00 1.00 2.00 2.00 5.00 2.00 2.00 0.50 1.00 2.00 0.50 1.00 1.00 4.00 1.00 1.00 0.50 1.00 2.00 0.50 1.00 1.00 4.00 2.00 1.00 0.20 0.25 0.20 0.20 0.25 0.25 1.00 1.00 1.00 0.50 1.00 0.50 0.50 1.00 0.50 1.00 1.00 3.00 0.50 1.00 0.50 0.50 1.00 1.00 1.00 0.33 1.00 The degree of possibility of criteria is presented in Table 1. The weight vector of criteria is presented in Table 2.

Ratings of the quality criteria of LORs
According to the MCEQLS approach, the following step after identifying the weights of the quality criteria is to identify ratings (values) of the quality-in-use criteria of the alternatives under consideration.
To present the experimental results of the practical (experimental) application on the novel Fuzzy AHP method, the authors selected three LOR alternatives that are mostly used by Lithuanian teachers.
Those alternatives are: (1) European Learning Resource Exchange (LRE 2014) service for schools; (2) Lithuanian Learning Object Metadata repository (emokykla 2014), which is the main part of the Lithuanian educational portal; (3) Lithuanian LOR the Learning Garden (Ugdymo sodas 2014) (in Lithuanian -"Ugdymo sodas"). Experimental results of setting the ratings (values) in triangular fuzzy numbers of the aforementioned alternatives performed by the authors are presented in Table 3.

Experimental evaluation results
According to the MCEQLS approach, the last step is to identify final evaluation results by using additive utility function presented in (Kurilovas et al. 2011). Here we have to add all the weights of the criteria presented in Table 2 multiplied by ratings (values) of the criteria presented in Table 3.
The final evaluation results are as follow: (0.6461 0.5009 0.4351). The obtained evaluation results mean that in terms of their features and functionality, the first LOR alternative (i.e. LRE) meets 64.61% of quality in comparison with the ideal, the second one (Lithuanian LOM repository) -50.09%, and the third one (the Learning Garden) -only 43.51%. Thus, LRE is a "good" LOR alternative, while the Lithuanian LOM repository is a "fair" LOR alternative, and the Learning Garden is an LOR alternative, which is between "poor" and "fair".
Therefore, according to the results of the application of the MCEQLS Fuzzy AHP methodology, LRE is a better alternative for users in comparison with the Lithuanian LOM repository and the Learning Garden. Table 3 While applying the MCEQLS approach for the evaluation of the quality of LORs, the similar evaluation results were obtained by using the MCEQLS Fuzzy TFN methodology (Kurilovas, Serikoviene 2013) and the MCEQLS AHP methodology (Kurilovas, Zilinskiene 2013). However, as it was mentioned above in Section 4.1, the MCEQLS Fuzzy AHP methodology is proposed because of its benefits in comparison with simple Fuzzy and AHP methods for establishing weights of the quality criteria in terms of stability and preciseness of the method.

Conclusions
In the paper, the authors propose the MCEQLS Fuzzy AHP methodology for the evaluation of the quality-in-use of learning repositories (i.e. its features and functionality). LOR features and functionality are the main characteristics in terms of engaging LOR users and content producers.
The methodology proposed is based on the MCEQLS approach to evaluate the quality of learning software, thus being a part of a more general decision support system for educational stakeholders to help them make well-founded decisions in the area. The essential part of the novel methodology presented is the application of the improved Fuzzy AHP method to establish the weights of the quality-in-use criteria of LORs. The presented research has shown that the created methodology is suitable and stable for evaluating the quality of LOR features and functionality.
The results of the expert evaluation of three real-life LORs showed that the methodology proposed is easy to use while applying it to solve real-life problems in the area. The experimental evaluation results showed that LRE is a better LOR alternative for users in comparison with the LOM repository of the Lithuanian portal and the LOR of the Learning Garden.
Finally, the proposed methodology provides experts and decisions makers with a clear instrumentality for a comparative analysis and choosing alternatives using numerical ratings (values) of quality criteria, and Fuzzy AHP-based weights.
In the paper, the new MCEQLS Fuzzy AHP methodology is proposed because of its benefits in comparison with simple Fuzzy and AHP methods for establishing weights of quality criteria in terms of the stability and preciseness of the method.
The main limitation of this work is that the paper does not include a detailed comparison new results obtained by applying the Fuzzy AHP with the results of other MCEQLS methods used by us previously in terms of the weights of LOR quality criteria. This detailed comparison concerning the establishment of weights of the quality criteria in terms of the stability and preciseness of the method needs to be studied separately in the future.
The main economic benefit of using the Fuzzy AHP to evaluate the quality-in-use of LORs is that the proposed instrument would help stakeholders to make proper decisions regarding the purchase or creation and the use of high-quality LORs. The proposed instrument would identify high-quality LORs in terms of their features and functionality and guarantee that users should have an easy and comfortable access to high-quality LOs in repositories. If an institution decides to use this instrument, this should be done by inviting a group of experienced experts to evaluate the quality of LORs available on the market or to create their own LOR based on the quality criteria model provided in the paper.
The authors believe that research results presented in the paper will be useful for all stakeholder groups interested in the development of high-quality LORs to engage its users and producers. The economic benefits of using the proposed instrument would be tangible because of proper decisions made by stakeholders based on a clear LOR quality model and precise and stable evaluation method proposed. A wide application of the proposed instrument should have a positive impact on the development of high-quality LORs, which in its turn should significantly improve the accessibility and quality of the educational content, thus being a useful tool for the technological development of economy.