A FRAMEWORK FOR EVALUATING AN INTEGRATED BIM ROI BASED ON PREVENTING REWORK IN THE CONSTRUCTION PHASE

. Construction firms attempt to estimate building information modeling (BIM) return on investment (ROI) to confirm whether BIM effects are sufficiently positive to satisfy decision-makers. Previous studies have presented the ROI in various ways, but a more definitive answer is required to consider possible various effects. Therefore, this study proposes a framework for an integrated BIM ROI, a simple, easy-to-understand, and practical tool that is established from substantive requirements from experts in the construction field. The framework consists of a three-phase process including a total of 11 steps. These phases are assessment planning, primary BIM ROI based on preventing rework, and integrated BIM ROI. Based on the proposed framework, an actual effect analysis of BIM project was conducted and the suitability of the methodology was discussed. The results of applying the framework showed that the primary ROI based on prevented rework costs was about 167.8% and the integrated BIM ROI to consider the overall effect of applying BIM was about 476.72%. In addition, the expert’s discussion confirmed that the framework can be employed as a practical means to evaluate BIM performance. This framework can be provided as a guideline to present an integrated BIM effect and assist to efficiently BIM application.


Introduction
Estimating the monetary performance of building information modeling (BIM) is one of the most critical tasks at the end of a project (Lee et al., 2012). Because the application of BIM requires investment, decision-makers who decide to apply the technology need to verify whether its monetary performance exceeds the cost. Return on investment (ROI) is the most widely used methodology that can verify BIM performance (Azhar, 2011;Barlish & Sullivan, 2012;Bryde et al., 2013;Ghaffarianhoseini et al., 2017;Qian, 2012;Walasek & Barszcz, 2017;. As ROI is provided with quantitative values, the decisionmakers can clearly confirm success or failure of the BIM project and determine whether its outcome meets expectations (Anumba et al., 2010;Autodesk, 2007;Bernstein et al., 2014;Young Jr. et al., 2009).
However, there are several limitations to accurately translate the BIM effects into numerical values. The mea-surement of BIM effects is based on various suppositions compared with a situation where BIM is not applied (Azhar, 2011;Barlish & Sullivan, 2012;Bryde et al., 2013;Ghaffarianhoseini et al., 2017;Lee et al., 2012). Consequently, several previous studies provided numerical values of BIM ROI in their case project, but these studies reported these values without presenting details about specific descriptions of the supposition and calculation processes (Ghaffarianhoseini et al., 2017;Lee et al., 2012). These cases report a very wide range of ROI values (Giel et al., 2010;Lee et al., 2012); thus, these results question the reliability of BIM applications and cause a negative perception of them.
Another limitation of previous BIM ROI studies is that it is difficult to propose an integrated BIM ROI that considers all possible effects. BIM is applied to construction projects to create a variety of effects, and an integrated BIM ROI considers all possible effects (Anumba et al., 2010;Autodesk, 2007;Bernstein et al., 2014;Young Jr. et al., 2009). In order to calculate the BIM ROI, the effects need to be converted into costs. However, converting all of the various effects into costs is practically impossible. Thus, an ROI that considers only the effects that can be converted to costs means that the results include an unreasonable or limited ROI calculation. In particular, intangible effects, such as improving communication between participants, which are difficult to convert into costs, are important challenges for an integrated BIM ROI to calculate. Therefore, many previous studies focused on a tangible effect as preventing rework that is easy to convert into cost, in contrast with intangible effects by BIM adoption (Lee et al., 2012(Lee et al., , 2018Ham et al., 2018;. Lee et al. (2012) and  analyzed a BIM ROI based on the avoidance costs of rework because of design errors. Based on the probability of detecting design errors, these studies calculated the direct cost and predicted the BIM ROI by considering the effects on the schedule and quality. Although specific measurement procedures have been presented, including a probabilistic approach, there is a lack of explanation for the detailed calculation methods of direct cost, and there is a limit to presenting the integrated BIM ROI because of excluding intangible effects. Thus, although quantitative values were presented with details, the BIM ROI was presented without intangible effects, or other possible effects were in question. The clients and decision-makers, especially in Korea, prefer an integrated BIM ROI, although it is a rather simple method if it involves more definitive procedures. In other words, the integrated BIM ROI could be more realistic and reasonable than an ROI that has been meticulously analyzed based on some parts of all the effects.
The purpose of this study is to propose a simple, easyto-understand, and practical framework for assessing the integrated BIM ROI in construction projects. In this study, design review, which is one of the basic services of BIM, in the construction phase, was examined. The effect of preventing rework by the design review is converted into cost, and, based on the results, the framework for estimating an integrated BIM ROI is proposed. In addition, the proposed framework has the extensibility to apply additional services beyond the basic services, such as 4D simulation, 5D simulation, and others. The proposed framework can be defined as a practical methodology that can clearly represent the integrated BIM ROI based on the quantification of rework prevented effect by BIM adoption. Design consideration for the framework and the criteria for analyzing the effect of BIM were established by eliciting experts' knowledge and consultation. This paper is organized as follows. The next section reviews previous studies on BIM ROI. The third section, based on the requirements, proposes the framework and explains the details of each phase. In the fourth section, the framework was applied to a case project, and the results and limitations are discussed.

Literature review
Many studies have conducted BIM ROI evaluations to quantify the BIM effects based on design review (Giel et al., 2010;Khanzode et al., 2008;Lee et al., 2012Lee et al., , 2018Sacks et al., 2005;. These studies reported that the design review has effects on various tangible or intangible benefits, such as preventing rework, reduction of errors and omissions, construction cost overruns, schedule delays, safety, and quality. These benefits are converted into cost and then applied to output costs of calculating BIM ROI. In this work, the service fee for the design review in the ROI calculation was input cost. Khanzode et al. (2008) reported that the rework costs were reduced to 0.2% of the work costs by using BIM; as a result, the schedule of the project was reduced by 6 months and the costs were reduced by about $9M for the overall project. Sacks et al. (2005) applied BIM to 50 sites in North America and quantitatively analyzed the effects by comparing the proposed method to the existing two-dimensional (2D) method. The results indicated that applying BIM saved 2.3% of the total construction costs because of the reduced engineering costs caused by improved work productivity, as well as the prevention of rework achieved by the pre-detection of design errors through interference reviews. Lee et al. (2012) analyzed the BIM ROI of a project in which BIM was applied to the construction of a new domestic large-scale multiplex building. A BIM ROI of 22-97% was derived by converting 709 design errors detected by BIM into rework cost savings. Then, the ROI increased to 624-699% when the potential effects caused by preventing delays were added.  estimated the amount of construction waste prevented by a BIM-based design validation process based on the amount of construction waste that might be generated because of design errors. Giel et al. (2010) selected two similar projects and analyzed differences in performance based on whether BIM was applied. In the project that used BIM, direct and indirect costs at the site were saved because of a reduction in design modification and construction periods. As a result, the BIM ROI ranged from 16% to 1,654%. Lee et al. (2018) estimated the BIM impact on preventing rework in their case study and determined that approximately $314,000 was saved by BIM adoption.
Although these studies provided quantitative numerical values for BIM ROI, most of the analysis results and their methods have limitations in clearly describing the effects of the application. In several cases, ROI results have been proposed without details about data collection and analysis methods (Ghaffarianhoseini et al., 2017;Lee et al., 2012). These cases report very wide ranges of ROI values depending on the project characteristics, tasks to be performed, and analysis methods. Such results may cause confusion in the interpretation of the effects of BIM application and, thereby, lower their reliability. In particular, most studies do not provide a BIM ROI that incorporates all possible effects, such as type and intangible benefits. For example, tangible effects, such as direct costs reduc-tion and schedule saving, from preventing rework were converted into costs, but intangible effects, such as quality improvement, safety improvement, and communication improvement, were excluded or provided only as probabilities (Lee et al., 2012;. In other words, limited BIM ROIs are presented that do not include all the benefits because they exclude intangible effects that are difficult to translate into costs. In fact, decision-makers need simple and clear ROI results that include a process to compare the input and output costs for applying BIM. Consequentially, contractors or decision-makers need to clearly confirm whether the effects of BIM application would exceed the BIM input costs; however, previous analyses cases fail to provide reliable and reasonable results.
Therefore, the target and scope of the analysis must be clear, and the effects of BIM application must be provided through a method of analysis that can be understood by project participants. In particular, it is necessary to set the correct target and scope in estimating the effects of BIM application and to propose a framework that considers integrated BIM effects in the calculation method.

Requirements
To improve the reliability and usability of the results, the requirements of the framework must be specified. For this, a consultation meeting was conducted with the field experts of the BIM project. The experts were composed of five BIM coordinators, six construction engineers, and three project managers who were involved in a BIM project at least twice, and they had an average of more than 5.5 years of experience related to BIM application. In addition, they had experience in reviewing preliminary constructability, design change, and the request for information based on design review reports provided by an involved BIM service company. Therefore, their opinions were reliable because they had a high understanding of BIM and recognized the need for BIM ROI. As a result, four common requirements of the framework were summarized as follows.
First, the outcome of the framework should be proposed to include monetary value. This criterion was suggested as the most important because monetary values provide more intuitive and persuasive results than reference percentage values such as "00% improvement compared with the existing level". Second, results of the design review from BIM that can also be solved in the existing method (2D) must be excluded or considered BIM contribution from effect analysis. In other words, even if construction error or rework possibilities are found through a design review of BIM, they cannot be seen as the effects of BIM if they are items that can also be found through the existing 2D methods. As mentioned in the literature review, Lee et al. (2012) and  proposed the BIM ROI analysis method based on the probability of design error detection. As a result of discussions with the experts, their analysis methods were discussed as reasonable and useful approaches. These approaches could improve the accuracy of the effect analysis by providing more practical and reasonable BIM application effects. Third, the calculation process of BIM effects must be clearly provided. If the calculation process is clearly presented and subjected to bidirectional reviews, more reliable results with high usability are provided. Lastly, an integrated BIM ROI that considers both tangible and intangible effects should be presented. Previous studies have provided monetary values for tangible effects, but they provided this value either as a percentage or separately for intangible effects, causing confusion when calculating the total ROI. Thus, this requirement could complement the limitations of numerous previous studies.
BIM ROI analysis based on the effect of preventing rework by design review is one of the typical methods of measuring the impact of BIM. In other words, it is necessary to systematically analyze the effect of preventing rework first because it can provide reliable analysis results and satisfy the requirements of various and complicated BIM effects. In addition, based on these results, questions about the overall effects of BIM, i.e., an integrated BIM ROI, need to be answered. Thus, the framework proposed in this study presents detailed processes that satisfy the requirements and provides the calculation method for the integrated BIM ROI.

The proposed framework
The objective of this study is to propose a framework for measuring BIM ROI. This framework is intended to be a method and process that can be adapted to fit a variety of BIM effects and project situations based on the requirements. The framework consists of a three-phase process, including a total of 11 steps. These phases are assessment planning, primary BIM ROI based on preventing rework, and integrated BIM ROI, as shown in Figure 1. The detailed descriptions are introduced in the following sections.

Phase I: Assessment planning
The first phase of this framework is assessment planning. This preparatory phase defines the object and the scope of the assessment, collects related data, and prepares the survey. Determining the target and scope of the assessment is the first step that is one of the most basic and important parts of the framework. This step means to establish the construction type, location, and duration of the BIM application to be assessed. In addition, this step clarifies the scope of BIM uses applied to the project and identifies the expected effects through the BIM application. The effects include the purpose of BIM application and all possible effects. For example, if the design review was applied to the project, the expected effects could be the cost savings from preventing rework, schedule compliance, work efficiency improvements, quality improvements, and safety improvements. These decisions determine the target and scope of the data collection and decide who will be surveyed for ROI calculation. In other words, the decisions are important criteria in the evaluation process and support the comfortable operation of the framework and ensure more correct results are provided.
The second step of the assessment planning phase is the data collection required for the BIM ROI calculation process. If the design review is the BIM service of the project, documentation should be collected for the ROI calculation, i.e., a contract of BIM service containing the execution plans and the service cost. Design review reports should also be collected according to the object and scope established in the first step. The reports are some of the most important documents for analyzing the preventing rework, and, if necessary, these documents may need to be revised to ensure estimators clearly understand the effects. Furthermore, guidelines of estimates must collect documents required for detailed calculation for items of preventing rework, such as quantity per unit, material and labor unit price, and overhead cost standard.
Lastly, the third step in this phase is preparation for the survey. Based on the data collected in the second step, the third step selects experts within participants of the project who analyze the items of the report. The experts are classified as work type according to the contents of the items, and they evaluate the BIM contribution level and perform an estimate for each item in the second phase.

Phase II: Primary BIM ROI based on preventing rework
The second phase of the framework is calculating primary BIM ROI based on the preventing rework, which is a preanalysis phase for deriving the integrated BIM ROI. This phase is proposed by referring to the methods presented in previous studies of Lee et al. (2012) and . These studies calculated the direct cost for each item by considering the likelihood of identifying errors by BIM when analyzing the economic impact through preventing rework. However, these studies need to be complemented in terms of accuracy by categorizing the probability of error detection into three levels (Level 1: 25% or below, Level 2: 50%, Level 3: 75% or above). From a preliminary review of the case study in this study and previous research (Lee et al., 2018), about 27% or more of the total errors were found to be errors that could be detected in traditional 2D drawing-based processes or without using BIM. If their approach is applied, there is a limit to accuracy because it applies a weighting value of 0.25 to these errors. In this study, the contribution of BIM was evaluated by applying a 5-point Likert scale based on the case of no contribution of BIM to detecting errors. Another limitation of these studies is when suggesting the results of calculation of direct costs without detailed explanation. Because this study strives to propose a practical process for deriving BIM ROI, it is necessary to present a practical and specific calculation process. In this study, assuming that a case of each error in the design review report was constructed without being detected, the actual cost of the case was calculated by placing it into three cost categories: the cost of initial construction, the cost of demolition, and the cost of reconstruction. Then, the BIM contribution level of each case was applied as a weighting value to the results of the cost calculated. As a result, the method proposed in this study could provide a more improved outcome in terms of accuracy and be more realistic.
The first step of the second phase is conducting a survey on the BIM contribution level. In this study, a survey form using a Likert-type 5-point interval scale is inserted into each design review report case to evaluate the BIM contribution level. Figure 2 shows an example of a questionnaire that has been modified to evaluate the BIM contribution level.
Because the definition of each scale is an important element that ultimately determines the cost calculation, the scale was defined as follows through expert discussion and improvements in methods of the previous studies mentioned above. First, point 1 is defined as the items that can be detected in traditional 2D drawing-based processes or without the BIM contribution. In other words, this point corresponds to simple errors or omissions in drawing notation with almost no rework probability. Next, point 2 is defined as the items that can be easily detected using BIM compared with traditional 2D drawing, or items that are not critical enough to cause a rework. These items correspond to the partial interferences detected during the creation of the integrated BIM model. They can be found naturally during construction and can be solved by drawing modifications through simple discussion. Point 3 corresponds to the items for which the contribution of BIM is in the middle of the contribution scales. Based on point 3, the lower points (one and two points) represent items for which the corresponding error can be easily found using traditional 2D drawing without the BIM application, or the possibility of rework is rare, and the higher points (point 4 and point 5) represent items for which the review through BIM is relatively significant. Thus, point 4 corresponds to the items for which it is difficult to find the corresponding error without the BIM advance review and the probability of rework is very high. Lastly, point 5 is defined as a serious item where rework occurs in case of not having the BIM advance review.
In the second step, the reliability of the survey results is analyzed, and the statistical value of the overall BIM contribution level is calculated. The BIM contribution value, c r , is then calculated by analyzing the BIM contribution level of each item collected through the survey. The c r value is calculated using the average value of each contribution level based on the collected survey results. Because the 5-point interval scale was applied to the survey, the c r value was set for the average values of the contribution level at 0.8 intervals (Table 1). After calculating the rework cost of each item of the next step, the c r value of each item is multiplied by the result.
In the third step, assume that rework occurs for each item in the design review report and perform the cost estimation of rework for each item. The cost of rework is divided into the costs of initial construction, demolition, and reconstruction. For each, cost, material, labor, and overhead costs are calculated. This cost estimation work is performed by discussing the costs with the respondents and the quantity surveyor in their project. Table 2 shows an example of a rework cost estimation sheet by the quantity surveyor. This example sheet calculates an estimate based on one case of the design review report. Without the design review by BIM, the grand total cost of this item is 4,890 in USD, which is the sum of the three costs (the cost of initial construction, the cost of demolition, and the cost of reconstruction). However, if the item was detected by design review, only the cost of reconstruction would have been used. As a result, the preventing rework effect of BIM design review would be 3,090 in USD, i.e., the sum of the cost of initial construction and the cost of demolition.
In the last step of this phase, primary BIM ROI based on preventing rework is calculated. The ROI calculation that reflects the c r value and the results of rework cost estimation in the third step are formulized in Eqn (1) where the net profit on preventing rework variables are as follows: primary BIM ROI pr , BIM ROI based on preventing rework; r -item number of design review report; n -total items of design review report; m r -material costs; l r -labor costs; o r -overhead costs; c r -BIM contribution level. The investment variables are as follows: sf -BIM service fee paid by the construction firm; ms s -monthly salary of a BIM coordinator assigned from the construction firm; t -total number of BIM coordinators assigned from the construction firm; w s -total work months of a BIM coordinator assigned from the construction firm; s -serial number of a BIM coordinator assigned from the construction firm; d s -input work ratio of BIM coordinator assigned from the construction firm. Lee et al. (2012) proposed the BIM ROI calculation formula based on prevented rework costs. The Eqn (1) of this study was simply revised by referring to the formula as a net profit divided by investment cost. The Primary BIM ROI pr can be defined as a calculation formula that can convert the BIM benefits' contribution to preventing rework into a monetary value from the point of view of a construction firm in the construction phase. The targets of this calculation formula are the design review report created in the construction phase. Under the assumption that rework occurs for each item of the report, the formula calculates the rework costs by calculating the material costs (m r ), labor costs (l r ), and overhead costs (o r ) and reflects the BIM contribution level (c r ) to them. Because the BIM ROI of this study is from the perspective of the construction firm, the investment includes the BIM service fee and the cost that is related to the manpower assigned from the construction firm to support the service. The ROI derived from this phase is an important result for the calculation of the integrated BIM ROI in phase III and could be independently suggested at the same time.

Phase III: Integrated BIM ROI
The last phase of the framework is calculating an integrated BIM ROI. The ROI, proposed to consider the overall effect of applying BIM to the project, is analyzed based on the BIM ROI pr . The last phase consists of a total of four steps, as shown in Figure 3.
The first step of the last phase defines BIM effects from applying a BIM service. The application of BIM produces a variety of effects in the construction phase. Because the purpose of this framework is to predict the economic impact of applying BIM, a definition of all kinds of possible effects from applying BIM service is first required. The person in charge of ROI measurement needs to define the effects based on the BIM services applied to the project. For example, if the design review is a BIM service in project, the possible effects are preventing rework, preventing scheduling delays, improving work efficiency, and improving quality. In addition, if there is an effect aimed at applying BIM, this may be added. Importantly, the defined effects must include quantitative or tangible effects that can be converted into costs. The effects of preventing rework are proposed as tangible effects in this study.
The second step is to measure the weighting value for the defined effects. In this study, the weighting value of defined effects is estimated through the analytic hierarchy process (AHP) questionnaire. The AHP, which was developed by Saaty in the early 1970s (Saaty, 2005), is a structured tool to assist people in dealing with complicated decision-making problems by stratifying the problem into smaller issues and prioritization issues based on ex- pert knowledge (Saaty & Vargas, 2012). The AHP has been applied in various areas of construction as a systematic approach for solving decision-making problems. In addition, the AHP determines the relative importance weights of factors for goals using pair-wise comparisons (Saaty, 2005;Saaty & Vargas, 2012). To evaluate the weighting value of each effect, a questionnaire is developed. Pair-wise comparisons were used to evaluate the relative economic impact of effects. After consistency was checked, the value of each effect is then derived.
In the third step, the economic impact of each effect is calculated by considering the weighting value. This is a very critical step in this framework for presenting an integrated ROI. The weighting values from AHP are the criteria for converting each effect into an economic impact. In other words, because the preventing rework effect has been converted into economic impact from phase II, the other effects can be converted into economic impact using a proportional expression that applies the economic impact of preventing rework and its weighting value. For example, if the economic impact of the preventing rework effect is 100 and the weighting value is 0.2, then the other effect of a weighting value of 0.5 is calculated to have an economic impact of 250. The economic impact of each effect is calculated according to Eqn (2), which modifies a proportional expression such as: where: the variables are as follows: n -the defined effect number; EI n -economic impact of effect n; wv n -weighting value of effect n; EI pr -economic impact of the preventing rework; wv pr , weighting value of the preventing rework. The last step of the framework is calculating the integrated BIM ROI. ROI is calculated as net profit divided by investment cost (Botchkarev et al., 2011;Lynch et al., 2006). In this framework, the net profit is the sum of EI n , which is the total economic impact by BIM adoption, and the investment applies the calculated investment cost in phase II. The integrated BIM ROI is calculated according to Eqn (3) (3)

Overview
Based on the proposed framework, an actual effect analysis of a BIM project was conducted, and the suitability of the methodology was discussed. This case study was aimed at a BIM project applied to the construction of building A of a public sports facility project in Korea. BIM services were applied to this project to develop the BIM model of architecture and structure and to provide the design review report. The BIM model was completed in the level of development (LOD) 300 (American Institute of Architects [AIA], 2013) based on the drawings from an architectural firm before starting the construction, and the design review was performed three weeks before the start of construction of the review regions to be constructed. The overview and created BIM model of the case project are shown in Table 3 and Figure 4, respectively.

Phase I: Assessment planning
The first step is to define the analysis of the object and the scope. As described in Table 2, building A of the case project was selected for ROI analysis. The building was an appropriate target for ROI analysis because the design review was systematically performed according to the construction progress with active cooperation from participants on site. The main constructor required an analysis of the effects of BIM application and finally decided to conduct an analysis of the integrated ROI. The ROI analysis was aimed at BIM service that established a BIM model of architecture and structure and included a design review of them. In contrast, the BIM model of the mechanical, electrical, and plumbing (MEP) part was excluded from the analysis because it differs from the LOD and timing of establishing the BIM model of the architecture and structure. The second step is collecting the data required for the ROI calculation process. A total of 127 cases of design review related to architecture and structure were reported in the project. In addition, the documents, such as detail schedules and photos, needed to discuss these cases were collected, and data for cost estimation were prepared. Then, the cases were grouped as related cases. Table 4 shows the work type classifications. The work type categories were architecture (48%, 61 cases); structure (16%, 13 cases); between architecture and structure (38%, 30 cases); and other works related to civil engineering and landscaping (12%, 9 cases).
Lastly, the third step in this phase is preparation for the survey. Based on the data collected in the second step, this step selects experts within participants of the project who analyze the items and evaluate the BIM contribution level. The participants in the survey included five BIM coordinators, six construction engineers, three project managers, and two cost estimators; and these participants were involved in a BIM project at least twice and had an average of more than 4.2 years of experience related to BIM application.

Phase II: Primary BIM ROI based on preventing rework
In the first step, 127 items were provided to the survey participants selected in phase I, and a survey on the BIM contribution level evaluation was conducted (August to September 2017). On completion of the survey, in the second step, the BIM contribution levels submitted by the participants were averaged and classified as the c r value in Table 1. Table 5 shows the result of the classification of c r values for each work type. The 50% c r value was the largest with 36 cases and 0% (with no contribution of BIM) c r value was the second largest with 33 cases. In contrast, there were only 3 cases where it was certain rework occurred (100% c r ).
As a third step, the rework cost was calculated from the sum of the initial construction and demolition costs, assuming that each item was reworked. The calculation was performed by an estimation expert with more than 20 years of experience to ensure the reliability of the results.  Although the evaluation was performed by one person, consistency in the estimation was secured, and the accuracy of the results was ensured by repeated reviews. Table 6 shows the results of the estimation details.
In the last step in the second phase, according to Eqn (1), the primary BIM ROI pr was finally calculated. In the previous step, the rework cost for all items was calculated to be about USD 576,977 (Table 6). Through applying the c r value, the economic impact of preventing rework was calculated as USD 322,191. Table 7 shows the calculation details of the numerator of Eqn (1). Therefore, the net profit on preventing rework was USD 467,253. The investment value is the sum of the BIM service cost paid by the construction firm (sc) and the construction firm personnel support cost for BIM coordination. In this case, the BIM service cost was approximately USD 162,000 and five BIM coordinators (t) were deployed at 5% work ratio (d) over a period of about 28 months (w), which is approximately USD 30,000. Accordingly, the investment cost was approximately USD 192,000. Overall, the primary BIM ROI pr of the case study was calculated to be approximately 167.8% (= USD 322,191/USD 192,000).

Phase III: Integrated BIM ROI
In the first step, BIM effects from applying BIM service in the case project were discussed from which the criteria were defined such as: preventing rework, schedule compliance, improving work efficiency, safety improvement, quality improvement, and strengthening BIM capability. In the second step, the weighting values of these six defined effects were calculated based on an AHP question-naire submitted to the experts. The experts who participated in the first and second steps were the same as the experts defined in the first phase and, as mentioned, there were a total of 16 experts with an average of 4.2 years of experience working on BIM applied projects. All of the respondents showed a consistency ratio of less than 10%. According to Eqn (2), the economic impact of each effect was calculated, as shown in Table 8. The last step of the framework is calculating the integrated BIM ROI. According to Eqn (3), the net profit is the sum of EI n , which is the total economic impact by BIM adoption, and the investment is applying the calculated investment cost in phase II. Therefore, the total economic impact was USD 915,315, and the investment cost was USD 192,000. Lastly, the integrated BIM ROI of the case project was calculated to be approximately 476.72% (= USD 915,315/USD 192,000).

Discussion
To evaluate the applicability and reliability of the proposed framework, a questionnaire survey was distributed to construction engineers (four BIM managers, three engineers, and two decision-makers of BIM adoption) in a case study. The participants were asked to grade the applicability and reliability of the methodology and process on a 5-point Likert scale: "1 = Highly Ineffective, 2 = Useless, 3 = Moderately Useful, 4 = Highly Useful, and 5 = Highly Effective". The results of the questionnaire indicated an informative value in evaluating the BIM ROI by scoring an average of 4.7 points (the reliability scores had an average of 4.5 points, and the applicability scores had an average of 4.9 points). These results show that the framework that includes the methodology proposed in this study can be employed as a practical means to evaluate BIM performance. As a result, it was confirmed that the proposed methodology and calculation results were more definitive and provided practical answers from the perspective of the decision-makers. In addition, these results show that the framework that includes the methodology proposed in this study can be employed as a practical means to evaluate BIM performance. In general, at the end of a project to which BIM is applied, the question "What are the total benefits of BIM?" is asked. In previous studies, however, there were only tangible effects or integrated effects presented without any intangible effects being presented. The framework in this study may be useful for providing an answer to the total benefits of BIM. In this study, the results of a proportional expression were based on the preventing rework effect, but if other quantitative effect measurements are proposed, it is possible to modify the framework. As a result of discussions with the experts involved in the case study, limitations of the framework were the relatively simplistic method that did not consider the association of the defined effects where time and cost overlap in each effect. There is also a limit that does not consider the issues of MEP. Thus, it is necessary to consider more varied BIM effects and to compensate for the calculation method. Furthermore, further research is needed to propose an integrated performance measurement methodology capable of suggesting a comprehensive set of BIM application effects.

Conclusions
The application of BIM has various positive effects in the construction industry and is continuously expanding. However, a reliable and comprehensive effects analysis of BIM application has not yet been conducted. Therefore, this study proposed an analysis method of the BIM effect and framework, and its suitability was reviewed through a case study. The expert's verification results revealed that the effect cost calculations and detailed contents were successful in securing a level of conformity that could be accepted by contractors. In the future, further research is to be conducted to derive an integrated performance meas-urement methodology capable of presenting a comprehensive BIM application effect in the construction phase. In addition, the reliability of the results will be reinforced through continuous case studies, as only one project was used as a case study here and only a limited number of participants assisted with the survey. The results of this study can be used as a foundation that could assist further studies on BIM effect analysis.

Funding
This work was supported by the National Research Foundation of Korea (NRF) and funded by the Korea government (MSIT) under Grant 2017R1C1B5075498.

Author contributions
ML conceived and designed the research, performed data analysis, and wrote the manuscript. UKL contributed to the research procedure, developed a detailed idea of the research, and revised the manuscript.