EVALUATION OF USER EXPERIENCE IN AUGMENTED REALITY MOBILE APPLICATIONS

. Globalization, technological development and a dynamic business environment influence the change of customer information demands. It becomes vital for organizations to find out the customer demand change and discover technological solutions to satisfy these demands. One of these technologies is augmented reality, which connects real and digital environments by expanding it with digitally coded information which is decoded by using a specific device. As this type of technology enables the changing information needs of customers to be met faster, organizations are increasingly using these technological solutions to achieve a variety of purposes: to position products innovatively, increase product awareness, create added value for the customer, increase sales. However, organizations often face the challenge of evaluating commercial augmented reality mobile applications in user experience. A two-case study has been selected to evaluate the user experience of augmented reality commercial mobile applications and provide recommendations for their development to address this issue. In this research, such methods as scoping scientific literature review, expert evaluation, and user experience questionnaire method were used. The study has identified the main factors influencing the positive user experience: the explicit purpose of the application, easy to use and learn, smooth operation, imaginative information presentation, and interactivity.


Introduction
The rise of augmented reality technology is driven not only by globalization, technological development, and a dynamic business environment, but also by the changes of consumer information and knowledge needs (Raudeliūnienė et al., 2018;Raudeliūnienė & Davidavičius, 2017), organizations' efforts and investments to improve the quality of technology devices and software that allows using augmented reality and increase accessibility to the general public in a way that is intuitive and user-friendly (Dey et al., 2016;Javornik, 2016;Nam, 2015;Rese et al., 2017).Researchers and business practitioners predict that the value of the augmented reality market, consumer numbers, and ongoing scientific research interest will increase in recent years (Davidavičienė et al., 2019;Dey et al., 2016;Javornik, 2016;Laine & Suk, 2016;Liao, 2016;Rauschnabel et al., 2017).
While augmented reality technology solutions are gaining popularity in global markets, there is still a lack of comprehensive research to evaluate the user experience created by these technologies (Davidavičienė et al., 2019;Dey et al., 2016;Schrepp et al., 2014Schrepp et al., , 2017aSchrepp et al., , 2017b;;Sekhavat, 2016).Dey et al. (2016) systematically have analyzed augmented reality studies related to user perspective between 2005 and 2014 and identified main research limitations as a narrow spectrum of evaluation methods and subjectivity aspect in user's evaluation.Sekhavat (2016) has conducted a user study for evaluating such elements as "the sense of presence and the usability of the application" (Sekhavat, 2016).Schrepp et al. (2014Schrepp et al. ( , 2017aSchrepp et al. ( , 2017b) ) have proposed a user experience questionnaire method for evaluating a user's experience.Based on Davidavičienė et al. (2019) research results, some limitations of the user experience method were identified as a discrepancy of the evaluation scale according to the context of the research object and lack of data for identifying the causes of issues (Davidavičienė et al., 2019).This study aims to evaluate the user experience of augmented reality commercial mobile applications and provide recommendations for their development to address this type of problem.The research employed such methods as scoping scientific literature review, expert evaluation, and user experience questionnaire.The latter method is chosen because it enables to analyze the user experience in a more comprehensive way by using different scales of object attributes in terms of evaluation aspects such as attractiveness, perspicuity, efficiency, dependability, stimulation, and novelty (Davidavičienė et al., 2019;Santoso et al., 2016;Wang et al., 2015).Two augmented reality commercial applications that were active during the research were selected for the study: Inkhunter (a mobile application that tests virtual tattoos using augmented reality) and Arilyn (a mobile application that presents the visual story of a product by Kalnapilis).These applications are chosen because they reflect the purpose and substance of augmented reality and do not oblige long-term user engagement and ease of use.
The study structure consists of the theoretical aspects of a mobile augmented reality application evaluation, investigation cases of two mobile applications and their user experience, identification of their problem areas, and recommendations for the application development.

Literature review
Augmented reality technology began to gain popularity in the last decade, although the early products were applied in military aircraft.The concept of augmented reality technology has also evolved as this technology developed, and there are more cases of its applications (Davidavičienė et al., 2019).
Business organizations operating in a dynamic and uncertain environment are continually looking for new ways to efficiently communicate with consumers (Klepek & Starzyczná, 2018;Ahmed et al., 2017).Analyzing the communicative aspects of augmented reality technology, it can be seen that this technology creates a different aesthetic perception of information when information about objects is rendered more visually appealing, combining various content forms, designs, channels, creating preconditions for added value and more efficient communication (Davidavičienė et al., 2019;Huang & Liu, 2014;Oleksy & Wnuk, 2016;Yoon et al., 2012).Augmented reality technology is useful in an organization's marketing efforts when promoting their products (services) to create a more engaging and intuitive experience that is more attractive to consumers.Because augmented reality technology is highly interactive, most of the time, the focus is not only on the product (service) but also on the user experience (Bulearca & Tamarjan, 2010;Dacko, 2017).
In conclusion, augmented reality technology creates preconditions to develop promotional, experiential, relational, and entertainment value (Davidavičienė et al., 2019).However, scientists and business practitioners are still debating how to best assess the value of augmented reality to the consumer and their experience, what research methods and evaluation criteria to apply (Davidavičienė et al., 2019;Dey et al., 2016;Dhir et al., 2013;Dünser & Billinghurst, 2011;Santoso et al., 2016;Sekhavat, 2016).Researchers emphasize that augmented reality solutions are very diverse and heterogeneous and that there is no universal assessment tool that is appropriate for evaluation of all augmented reality platforms.There is also no consensus on the concept of user experience in the context of augmented reality (Dhir et al., 2013;Sekhavat, 2016).Scientists note that user experience is associated with subjective consumer feelings about the product (service) being used (Davidavičienė et al., 2019;Santoso et al., 2016).Thus, it is challenging to evaluate user experience expectations of augmented reality technology (Dhir et al., 2013) due to the change of experience over time.Scientists generally explore such main categories of augmented reality user experience evaluation as user perception and understanding, performance of user tasks, cooperation between users, system convenience and design (Davidavičienė et al., 2019;Dünser & Billinghurst, 2011).
When analyzing the end-user experience of augmented reality platforms, researchers typically use a mix of qualitative and quantitative research methods such as expert evaluation, surveys, structured interviews, focus groups (Davidavičienė et al., 2019;Georgiou & Kyza, 2017;Guimaraes & Martins, 2014;Ko et al., 2013;Laine & Suk, 2016;Majid et al., 2015;Mota et al., 2018;Rauschnabel et al., 2017), which present a combination of open and closed questions and assessment statements that are measured on Likert scales, usually of 5 or 7 levels.Multicriteria evaluation methods, which have a wide range of applicability in research, would also be useful in investigating the complex research objects of this type (Beskese et al., 2018;Delgado Méndez et al., 2019;Zavadskas, 1987;Zavadskas et al., 2019;Zeng & Xiao, 2018;Zhuang et al., 2019).
Time: the survey is not time-consuming; it takes 5-7 min.
The simplicity of questions: questions are simple and clear, but quite abstract to identify specific problem areas of the research object.

USE (Usefulness, Satisfaction, Easy to
Use and to Learn) (Albertazzi et al., 2012;Davidavičienė et al., 2019) This is a questionnaire based on criteria with a 7-level Likert scale.This assessment method consists of 30 questions analyzing a product in 4 dimensions -usefulness, easy to use, and to learn, and satisfaction.
Additionally, two open-ended questions are asked -the user is asked to identify the main negative and positive aspects of the evaluated object.The checklist of questions is based on evaluation criteria such as effectiveness, efficiency, satisfaction.
Purpose: the list of questions is tailored to evaluate augmented reality platforms.
Terminology: some of the terms in the questionnaire may not be clear to a user not aware of augmented reality principles.
TAM (Technology Acceptance Model) (Mota et al., 2018;Pantano et al., 2017;Rese et al., 2017) The questionnaire focuses on the user acceptance response to new technology when evaluated against such criteria as efficiency, ease of use, intention to use, satisfaction, informativeness, attitude to use, interactivity, information quality, response time, aesthetic qualities, and so on.
Content of the questions: multiple criteria object evaluation.
There is a lack of complexity in the questionnaire.

Complex questionnaire construct.
Additional open questions are missing to help understand the causes of the problems identified.
After a detailed examination of the augmented reality user experience tools and their advantages and disadvantages (Table 1), the User experience questionnaire evaluation tool was selected due to the complex questionnaire design, six evaluated dimensions: attractiveness (overall user impression of the object being evaluated); efficiency (is it possible to use the product quickly and efficiently, does the user interface look organized); perspicuity (is it easy to understand how to use the product); dependability (whether the user feels in control of the interaction or whether the interaction with the product is safe and predictable); stimulation (whether it is interesting and exciting to use the product or whether the consumer feels motivated to continue using the product); novelty (whether the product design is innovative and creative or whether the product attracts the consumer's attention).Previous studies show (Davidavičienė et al., 2019) that the research results obtained by this method only serve to identify the perceived problem areas of the object being evaluated, so it makes sense to use a combination of different evaluation tools to understand the overall user experience in augmented reality mobile applications.

Research methodology
In order to investigate the overall user experience in augmented reality mobile applications, the research process structure from three stages was created (Figure 1).In the first research process based on specific selection criteria as application purpose, target audience, user commitment, operating system, the popularity of applications, two different augmented reality mobile applications -Inkhunter and Arilyn -were chosen.These applications are selected because they have a different goal and target audience, do not require long-term user commitment, operate both on Android and iOS mobile operating systems, it is easy not only to find out the purpose, content, and features of the application in a short time but also to evaluate these applications characteristics.Besides, these applications are trendy among customers (Table 2).In the Inkhunter mobile application, the user can try various virtual tattoo designs using augmented reality technology when the application operates on a marked-based case.Photos can be saved, edited, and shared with others on social networks.Although this application does not have a direct purchase feature, it is classified as a commercial augmented reality mobile application as it promotes tattoo designs by various developers.
The Arilyn mobile application operates on a marked-based principle and is designed to commercialize the Kalnapilis brand and increase sales of its products.The user needs to point a specially designed Kalnapilis product at their smartphone with the Arilyn mobile application turned on, and they see a visual story of a product by Kalnapilis.This augmented reality solution uses visually appealing 3D graphics, animation, and sound.
In the second research process, the user experience tools were selected: user experience questionnaire tool (Schrepp et al., 2014(Schrepp et al., , 2017a(Schrepp et al., , 2017b) and expert evaluation.Data analysis tool for both data collection tools -user experience questionnaire and expert evaluationwas Microsoft Excel (Figure 1).
The user experience questionnaire tool was chosen for this study because it reflects the overall user experience of the object being evaluated.Summarizing other augmented reality assessment tools, it can be noted that most of the tools analyze more technical characteristics of the augmented platforms, and the respondents are asked specific questions about the augmented reality platform.
The authors of the User experience questionnaire method in order to interpret the user experience of the evaluated product, propose to compare the results obtained with the benchmark proposed in this method, which is based on the evaluation data of 246 other products.
The following interpretations of the evaluation results according to the established standard are proposed (Schrepp et al., 2014): -excellent (10% of the best results); -good (10% of the results in the benchmark data set are better than the result for an evaluated product, and 75% of the results are worse); -above average (25% of the results in the benchmark data set are better than the result for the evaluated product, 50% of the results are worse); -below average (50% of the results in the benchmark data set are better than the result for the evaluated product, 25% of the results are worse); -bad (25% of the worst results).The evaluation results were converted into numerical values from -3 to +3 on the opposition trait scales of this method, where "-3" represents the most negative value, "0" the neutral value, and "+3" the most positive value.On a scale with values higher than "+1" identifies positive, less than "-1" identifies negative meanings (Santoso et al., 2016).
Analyzing the positive and negative values found for each value scale, it has been observed that some values in the context of mobile augmented reality applications may be different from those found in the User experience questionnaire method.For this reason, it was decided to use an additional expert evaluation (a structured questionnaire) in the study to determine which features of the chosen approach can be considered positive and negative for mobile augmented reality commercial applications.7 Lithuanian experts with at least five years of experience in augmented reality participated in the expert evaluations.According to the expert evaluation results, the vast majority of the scales corresponded to the positive and negative values determined by the standard method.However, changes have been made to the "unpredictablepredictable" scale, with "unpredictable" becoming positive by experts and "predictable" being negative in the context of augmented reality mobile applications, as experts say one of the most significant benefits of this technology is its novelty, amazement, the factor of effect.
Previous studies have noted that the sample of respondents is not significant in terms of time, as each respondent has to familiarize himself/herself with the augmented reality platform before completing the questionnaire.In the user experience questionnaire method instruction is noted that a sample of 20-30 respondents when using this method creates prerequisites for obtaining reliable assessment results.In this study in 2018, 30 respondents were selected using a non-random sample based on the age criteria (18-34-year-olds, as previous studies have shown that the highest penetration of the augmented reality platform is in the 18-24 and 25-34 age group categories), 73% of which survey respondents were in the 18-24 age group and 27% -of 25-34 age group category.53% of the respondents were men, and 47% -female representatives.The majority of the respondents had a higher university education (43%) and secondary education (40%); 10% -higher non-university education, 7% -a science degree.
Because the device used to operate the applications also influences the user experience, all respondents were given the same device to use the applications, the LG G3 smartphone running on Android.This device boasts a high-quality camera (13Mbs), high image quality (4K), large screen (5.5"), and special sensors required for smooth operation.
Respondents were asked to complete specific tasks during the survey (on Inkhunter -try out some virtual tattoo designs, take pictures, share them on social media, Arilyn -view at least a few Kalnapilis campaign stories, test features, take pictures and save the content) to encourage them to get to know the mobile applications that they will evaluate.Respondents were familiarized with these applications for about 10 to 15 minutes, and immediately after testing each application, the respondent was asked to fill in a part of the questionnaire for evaluation.Respondents took about 10-15 minutes to complete the questionnaire.As the survey was conducted directly, respondents were given the opportunity to ask questions.

Research results and discussion
The evaluation results of the augmented reality application Inkhunter have shown that the overall user experience evaluation is positive.In all dimensions analyzed, the value obtained is more significant than 0.8 (in the user experience questionnaire method instructions, the value of 0.8 is the boundary between neutral and positive).The novelty (2.358) dimension received the highest rating, the efficiency (2.092), perspicuity (2.067), and attractiveness (2.028) aspects were similarly evaluated, and the lowest values were assigned to dependability (1.717) and stimulation (1.8) dimensions.The augmented reality Arilyn application is rated highest in dimensions such as novelty (2.2) and perspicuity (2.067), on average in attractiveness (1.456), and lowest in stimulation (1.142), dependability (1.1) and efficiency (0.967) (Figure 2).The user experience questionnaire method analysis results tool provides the ability to determine the internal consistency of individual dimensions, which is expressed as Cronbach's alpha coefficient.If Cronbach's alpha reaches 0.60, this means that there is sufficient scalability.Inkhunter application's internal compatibility results show that the internal compatibility of the dependability dimension is insufficient, which is related to the interpretation aspects of the "unpredictable-predictable" scale (Table 3).Analyzing the results of the Inkhunter evaluation, this application seems to be attractive to users.It has more hedonic (stimulation, novelty) compared to pragmatic (efficiency, perspicuity, dependability) features.Hedonic characteristics are not related to the achievement of purpose but reflect pleasure.Augmented reality application Arilyn is moderately attractive to consumers, with higher hedonic properties compared to the pragmatic ones associated with the goal-seeking and practical value of the product.This shows that the Arilyn application creates a rather enjoyable experience, is entertaining but not efficient and useful (Figure 3).
In the User experience questionnaire's analysis results tool, a T-test was performed to compare the differences between the two Inkhunter and Arilyn application dimensions, showing whether these differences are significant.According to this test, the difference in the dimension estimation result is considered significant if their confidence interval does not intersect.In the T-test with a 95% applied significance level (α = 0.05), the results show a considerable difference between evaluated application scores in dimensions such as attractiveness, efficiency, and dependability.This indicates that the Inkhunter application is more attractive, efficient, and reliable for users than the Arilyn application.The Arilyn application is considered by its users to be impractical, not generating benefits, insufficiently interactive, and malfunctioning (Table 4).Respondents were asked to evaluate the advantages and disadvantages of augmented reality applications Inkhunter and Arilyn for overall assessment results.The main benefits of the Inkhunter application are related to the smooth functioning of the application, the content of the application, and its novelty, and the advertisement inside the application is evaluated as a significant disadvantage.The main advantages of the Arilyn application are its attractive content, interesting advertising solution and high-quality graphics, and its main drawback is the uselessness of the application (Table 5).Comparing the efficiency of commercialization of these applications, the Inkhunter application is ranked better than the Arilyn application.74% of respondents agree that the experience created by the Inkhunter application would encourage them to consider using this platform.In contrast, in the case of the Arilyn application, only 34% of respondents agree to purchase the products.Summarizing the results of the study, it can be seen that the Inkhunter application creates a greater user experience compared to the Arilyn application.Significant differences in evaluation results were found in such dimensions as attractiveness, efficiency, and dependability.
Based on the conducted research results, it is recommended that augmented reality application developers pay attention to the fact that its success and efficiency are greatly influenced by the user experience being created, controlled by various factors.In this study, it was found that the main factors influencing the augmented reality application user experience are: the expediency and utility of the application to the end-user; easy to use, perspicuity, clarity; smooth operation of the application; picturesque and attractive graphics, information presentation; interactivity and additional functionality.The augmented reality application should be clear to the user and user-friendly; the user should achieve the application's purpose in as few steps as possible.Because augmented reality applications are not intuitively perceived for some users, short, clear user instructions should be provided at the start step of the application.The augmented reality application should allow the user to manage, modify, and create content that encourages user creativity and engagement.It is recommended to integrate advanced functions such as the ability to share content on social networks, which personalize it and increase awareness of the application itself.

Conclusions
After analyzing the augmented reality approaches of researchers and business practitioners, in this study, augmented reality is defined as a technology, a real-time direct or indirect representation of the physical real-world environment, which allows the user to see the real environment augmented with virtual objects and enhances the interaction with the real world by attempting to create a mixed reality in real-time to improve the user experience.
Because of the uniqueness of augmented reality technologies in meeting users' information and knowledge needs and creating interactive communication with end-users, these technologies have a wide variety of applications globally, from educational applications to reaching commercial goals of an organization.This study evaluated the features of augmented reality technology applications used to achieve commercial goals for a business organization.
Although over the last decade, there has been an increase in augmented reality user experience evaluation research using evaluation tools such as SUS (System Usability Scale), USE (Usefulness, Satisfaction, Easy to Use and Learn), HARUS (Handheld Augmented Reality Usability Scale), ISO (Usability, Effectiveness, Efficiency) and TAM (Technology Acceptance Model).However, the examined evaluation tools show a lack of a complex questionnaire construct and they are more focused on the technical characteristics of augmented reality platforms.
For this study, the user experience questionnaire method was selected, which evaluates the overall product user experience in 6 fundamental dimensions (attractiveness, perspicuity, efficiency, dependability, stimulation, and novelty) and consists of 26 oppositional semantic scales that evaluate product characteristics.To apply this approach to the context of augmented reality mobile applications, an expert evaluation was conducted to determine which features can be considered positive and negative for mobile augmented reality commercial applications.Experts agreed to change the "unpredictable-predictable" rating scales from "unpredictable" to "positive" and "predictable" to "negative" as these technologies are characterized by the novelty, surprise, amazement effect they create.
The survey included 30 respondents, with the typical survey respondent being a 23-yearold male or female graduate.The Inkhunter application has been found to be attractive, innovative, and useful, and creates a positive user experience: novelty is the highest evaluated, and dependability is the lowest evaluated.One of the problem areas of this application is advertising that interferes with the user experience and causes malfunctions.The Arilyn application was rated highest for novelty and perspicuity and lowest for efficiency.This application is considered innovative due to augmented reality function; visual graphics features are easy to understand for the user and have hedonic features.The main problem areas of the Arilyn application are related to the fact that the technical functioning of the application is not always smooth, and the interactivity is lacking.
In order to eliminate some limitations in the user experience questionnaire method, the expert evaluation method was integrated into this study.Based on the expert evaluation results, some changes have been made to the "unpredictable-predictable" scale, with "unpredictable" becoming positive by experts and "predictable" is a negative in the context of augmented reality mobile applications, which is a valuable practical implication for researchers.
From the results of this study, augmented reality application developers and other educational organizations and society would benefit via identified key factors influencing the augmented reality application user experience.The study identified the following key factors contributing to a positive user experience that should be addressed when developing new augmented reality mobile applications: the explicit purpose of the application, easy to use and learn, smooth operation, imaginative information presentation, and interactivity.
There are several limitations to this study.The first limitation is related to the small generalizability (30 respondents) of the findings.The second limitation of this study is connected to the augmented reality application's usage duration when the respondent's interaction with the augmented reality platforms under evaluation was relatively short.They could not assess long-term technicalities of the application (for example, the memory used on the phone), so the influence of these factors was not evaluated.Future research areas could be related to an increase in generalizability with different business segments and other regions.

Figure 1 .
Figure 1.Research process structure (source: created by the authors)

Figure 2 .
Figure 2. Evaluation of augmented reality applications Inkhunter and Arilyn according to the six dimensions (source: created by the authors)

Figure 3 .
Figure 3. Evaluation results of augmented reality applications Inkhunter and Arilyn according to hedonic and pragmatic quality (source: created by the authors)

Table 1 .
Augmented reality user experience evaluation tools (source: created by the authors)

Table 2 .
Summary of augmented reality applications Inkhunter and Arilyn (source: created by the authors)

Table 3 .
Inner compatibility results of augmented reality applications Inhkunter and Arilyn (source: created by the authors)

Table 4 .
T-test results (source: created by the authors)

Table 5 .
The advantages and disadvantages of augmented reality applications Inkhunter and Arilyn (source: created by the authors)