SIMULATION-ENHANCED APPROACH FOR RANKING MAJOR TRANSPORT PROJECTS

Owing to financial constraints, it becomes imperative to rank major transport projects to determine implementation priorities and budget allocations. The central Government in Taiwan is using rankings derived from the Analytic Hierarchy Process (AHP) and direct subjective rankings to set funding priorities. The current approach does not account for the variations in rankings for setting these priorities. Nor does it adequately consider the compatibility with the proposed projects and the national policies in transport infrastructure development. To address these problems, the Central Government has revised the method for project ranking. The revised method expands the matrix of the attributes and impacts that are to be evaluated. It also uses a Monte Carlo simulation analysis to help in determining the rank orders. A pilot study was conducted to assess the revised method. The study uses 25 major rail projects proposed in 2002 as a test bed.


Introduction
The resources available in any country for transport infrastructure improvement rarely meet the needs. Taiwan's central government encounters this dilemma regularly. As an example, 25 rail projects were proposed to the central government in 2002 for a total budget request of US $ 2,2 billion but the funding level approved that year for the rail projects was only $ 0,7 billion. Under this type of severe fiscal constraints, it becomes imperative to employ a rational and structured process to determine funding priorities.
Major transport projects require large capital spending, and they invariably have a wide range of tangible and intangible impacts. To facilitate an efficient, equitable and environment-friendly allocation of limited resources, the impacts of a project should be weighed against those of other projects to determine funding priorities. This is a difficult task because of the lack of a single and objective measure that can be used to determine the net worth of each competing project to the society. In a democracy, this problem is compounded by the presence of many stakeholders whose vested interests often make the funding of a major transport project contentious. Under the circumstances, Taiwan's central government has been using the Analytical Hierarchy Process (AHP) and direct ranking to determine the funding priorities of major transport projects (Su et al 2002).
Since its introduction by Saaty (1980) more than two decades ago, the AHP has been used in many countries. In the US, for example, the AHP has been used for evaluating urban transit alternatives (Kaysi and Abdul-Malak, 2001). The Indiana Dept of Transportation (Kim and Bernardin, 2002) has also used AHP for prioritising major highway capital investments. And, because of its solid mathematical foundation, the AHP has been recommended to the Michigan Dept of Transport to develop a composite performance index for each transit service that receives state funding (Khasnabis et al, 2002). In Europe, the AHP has been used for resources allocation (Ramannathan and Ganesh 1995). AHP application software has also been developed (Ossadnik and Lange 1999). In Turkey, the AHP was the tool used for evaluating alternative rail transit networks for Istanbul (Gercek et al, 2004).
The AHP provides an analytical foundation to combine both tangible and intangible impacts into numerical scores for ranking alternatives. It requires evaluators to perform pair-wise comparisons of the relative importance of goals and objectives, as well as the relative desirability of competing projects. For evaluating major transport projects in Taiwan, however, the application of the AHP in the past has several major weaknesses. To address this problem, the Central Government has revised the current method for project ranking. As part of this effort, a pilot study was conducted to assess the revised method. The study uses 25 rail projects proposed in 2002 as the test bed. This paper describes the revised method and the findings of the pilot study.

Project funding process
In Taiwan, the funding of major highway or rail transport projects follows a process (Fig 1). First, various transport agencies of the local governments or the Central Government formulate projects based on projected needs for transport services. At present, this initial effort is often carried out without much coordination between agencies. Second, each proposing agency has to conduct a feasibility study. According to the current regulations, such a study has to cover economic feasibilities, financing issues, and socio-economic and environmental impacts. Third, each agency must submit a comprehensive study report to the Executive Yuan for review and approval. The Executive Yuan is the highest executive branch of Taiwan's Central Government. Table 1 shows an example of the key items and findings included in the feasibility study reports of two rail projects. Fourth, approved projects are then presented to a panel of evaluators for ranking. The panel is composed of 10-12 representatives from the Institute of Transportation, the Dept of Highways and Railroads, and the Dept of Accounting. Finally, the panel submits its recommendations to the Executive Yuan the authority to make final funding decisions. It is not bound by the recommendations made by the project evaluation panel. The budget allocation is done on an annual basis. This means that each proposing agency has to submit a budget request for the following fiscal year. It also means that an ongoing project has to compete with new or other ongoing projects for funding. As a result, whether a project has received funding in the past and to what extent a funded project has been completed are relevant concerns in project ranking. This annual budgeting cycle has been criticised for running a high risk of inefficient implementation of approved projects.

Ranking process
Taiwan's national polices for transport infrastructure development are to: (1) foster an integrated multi-modal transport network that links airports, harbours, highways, bus and rail transit systems, and regional railways; (2) improve transport safety, efficiency, and mobility; (3) ensure sustainable use of resources; and (4) preserve cultural heritage and improve living environment. Because of a very high population density (23 mil people in an area of 36 000 km 2 ), Taiwan's central government views rail transport as a key for promoting sustainable socioeconomic and land use development.
The Central Government has been using the AHP to evaluate major transport projects. For rail transport projects, the evaluation is based on the various items of project attributes and impacts, such as those shown in Table  1. This practice has 3 major drawbacks. First, the AHP requires pair-wise comparisons to determine: (1) the relative importance of each attribute or impact (eg, rate of return vs air pollution level); and (2) the relative desirability of each project as characterised by each attribute or impact. The number of attributes and impacts that need to be considered is large. This makes the evaluation process rather tedious and prone to evaluator fatigue and inconsistency in exercising judgment. This is compounded by the fact that the number of projects to be evaluated in a typical year is also large. Second, current regulations do not require a feasibility study to address the compatibility of a proposed project with the national policy of developing an integrated, multi-modal transport network. Furthermore, because projects are independently initiated by various agencies, feasibility studies do not address the issue of whether a proposed project would complement other projects. Therefore, there is a need to expand the evaluation criteria. And, finally, the ranking process as practiced in the past reduces various ratings into an average rating for each project according to the following formula: where i r -average rating of project i; j W -average weight of evaluation criterion j; and ij X -average rating of project i with respect to evaluation criterion j.
The average ratings of the competing projects become the basis for determining the rank orders of the pro-jects. It should be noted that j W and ij X are the averages of the respective weights and ratings assigned by the evaluators. The averaging process would invariably result in a loss of information concerning the true nature of the desirability of a project. To mitigate the drawbacks mentioned above, the Central Government has revised the ranking method. This revised method consists of 5 tasks as described below.

Task 1. Develop categories of evaluation criteria
This task is the responsibility of the evaluation panel that usually consists of 10 to 12 evaluators. The evaluators are representatives of 3 Central Government agencies: the Institute of Transportation, the Dept of Highways and Railroad, and the Dept of Accounting. As a pilot study of the revised ranking method, a panel of 12 evaluators was asked to evaluate 20 rail projects that were proposed in 2002. Five of the projects were pending approval by the Executive Yuan at the time of the study. The panel members were asked to treat these projects as approved projects.
A major concern in the process of developing evaluation categories is the ease in applying the AHP. After some discussions, the panel members agreed to classify project attributes and impacts detailed in feasibility studies, such as those shown in Table 1, into 6 evaluation categories. These 6 categories include: (1) project development stage; (2) degree of project completion; (3) safety, efficiency, and mobility; (4) economic and financial feasibility; (5) level of pollution; (6) social, cultural, and land-use impacts. Each category represents a group of similar project attributes or impacts. In addition, compatibility with the national transport policies makes up the 7th category of evaluation criteria. This evaluation category covers concerns about the energy policy, human resources development, cross-jurisdictional coordination of transport service, equity of mobility, and national transport development plan.

Task 2. Determine weights of evaluation categories
In this task the evaluators uses the AHP to determine the weight of each evaluation category. The resulting weight reflects the relative importance of that category as compared with all other categories. Because project impacts and attributes are grouped rather than being treated individually, the number of pair-wise comparisons that have to be performed is quite manageable. Table 2 shows the weights derived from the AHP in the pilot study.
It should be noted again that Taiwan's current regulations do not require a feasibility study to address the compatibility of a project with the national transport policies. Surprisingly, Table 2 reveals that the evaluators as a whole rate evaluation category 7 (compatibility with national transport policies) the most important by giving it a mean weight of 0,3. Therefore, there is a need to change the regulations to require that feasibility studies explicitly address the compatibility issue.
Table 2 also shows that, among the first 6 evaluation categories, category 1 (project development stage) and category 2 (degree of project completion) are deemed more important. This implies that the ranking process favours ongoing projects. It also implies that the evaluators' judgments are consistent with the common practice of not stopping an ongoing project.

Task 3. Rate project attributes and impacts
With respect to a given evaluation category, each evaluator would assign an integer desirability rating to each competing project. This rating is on a scale of 1 to 6, with 6 representing the most desirable. Evaluators are required to review the feasibility study reports before executing this task of direct rating.
At present, the feasibility study of a project does not have to provide an analysis of the compatibility with the national transport policies. Therefore, in terms of the compatibility issue, evaluators have to assign a ranking for each project without input from a proposing agency. The execution of this task is not a significant problem. This is because the members of the evaluation panel are seasoned experts on the national transport policies and needs, and they are familiar with the potential impacts of the competing projects. Table 3 shows the ratings of one of the projects included in the pilot study.

Task 4. Determine the distributions of aggregated ratings of competing projects
Instead of using average weights and ratings in Eq 1 to determine a single rating for each project, the revised method uses the weights and ratings assigned by individual evaluators to determine the probability distribution of the final rating of each project. This task requires the use of Monte Carlo simulation. The project on rehabilitation of rail line structure is used as an example to illustra-te the simulation process. The needed data are shown in Tables 2, 3. Refer to Table 2. The weights assigned to each evaluation category may differ from one evaluator to another. Therefore, the weight of each evaluation category is not a constant and thus would be best represented by a probability distribution. To facilitate simulation, the weights for each category should be transformed into a cumulative distribution. For evaluation category 1, which has weights ranging from 0,17 to 0,28, the cumulative distribution is as shown in Fig 2. Given this distribution, a random number R with a value uniformly distributed between 0 and 1 can be generated to represent the cumulative proportion shown in Fig 2. Based on this random number and the cumulative distribution, a corresponding weight W 1 can be determined for evaluation category 1. A large number of weights can be generated in this manner in a simulation process. These weights will form a distribution that is statistically identical to the original distribution. Similarly, the ratings, shown in Table 3 with respect to each evaluation category can be represented by a cumulative distribution for simulation analysis.
The simulation analysis requires a large number of simulation runs in order to simulate the actual distribution of each weight or rating. Each run would proceed in several steps. First, a weight is generated randomly for each evaluation category, subject to the constraint that the sum of the weights of all evaluation categories must equal 1,0. Next, for each project, a rating is generated randomly with respect to each evaluation category. And, finally, the generated weights and ratings are used in the following equation to determine the aggregated rating of each project: where i R -aggregated desirability rating of a project based on simulation run i; ji W -weight generated for evaluation category j in simulation run i; and ji K -rating generated for the project with respect to evaluation category j in simulation run i. The aggregated rating obtained from Eq 2 represents one possible aggregated rating of a project. Based on the results of a large number of simulation runs (eg, 5 000), the frequency distribution of the aggregated ratings of each project can be identified. Fig 3 shows the resulting frequency distribution for the project on rehabilitation of rail line structures. This distribution has a mean rating of 4,15 and a standard deviation (SD) of 0,34. Fig 3 shows that the distribution is skewed to the right, with a probability of less than 20 % that an aggregated rating would exceed 4,5. If the mean weights and ratings shown respectively in Tables 2 and 3 were used in Eq 1, the result would not be able to reveal that the rating of the project can vary between 3,0 and 5,5.

Task 5. Determine rank orders of competing projects
The rank order of a project is based primarily on the mean aggregated rating of that project. It may not be adequate, however, to determine the final rank order of a project solely on the basis of the mean aggregated rating. This is because when the aggregated ratings of a project in different simulation runs spread over a wider range, it becomes less certain that the mean rating represents the true desirability of the project. A project that has a wider spread of ratings is also one that is more controversial and should be given a lower priority. For this reason, the revised method for project ranking requires the frequency distribution of the aggregated ratings of each project be analysed to provide an additional information.
Based on the results of 500 000 simulation runs for all the projects proposed in 2002, Fig 4 shows that, on average, there is only a 5 % probability that the aggregated ratings of a project would exceed its mean by 0,74. Therefore, if a project has a much greater probability that its aggregated ratings will deviate from the mean rating by 0,74, the project can be reasonably judged as controversial. It is recommended that the threshold probability be set at 9 % (ie, 1,8 times 5 %). This threshold leads to 4 of the 25 competing projects (or 16 %) being classified as controversial. This information is to be included in future panel's recommendation to the Executive Yuan. The results of the pilot study are summarised in Table 4 along with the percentage of budget request actually granted for each project. Fig 5 shows that the funding level, expressed as percent of requested budget, tends to increase with the mean aggregated rating of a project. This implies that the mean aggregated ratings derived from the revised method can be meaningfully used for budget allocation.

Conclusions
Taiwan's Central Government uses the AHP and direct rankings to determine the relative desirability of major transport projects. To address several weaknesses in the past practice of project ranking, the Central Government has conducted a pilot study to assess a revised method for project ranking. The pilot study findings show that the revised method is easy to apply and can generate meaningful information to help in budget allocation. It is expected that the revised method implementation will enhance the ability of the Executive Yuan to effectively and equitably utilise limited resources.
The revised method of project ranking is essentially a methodological framework in which details of project evaluation can evolve over time in response to changing regulations and goals of transport infrastructure development. One area that the pilot study did not address is the practical maximum number of evaluation categories that evaluators can effectively handle in applying the AHP. The study shows that the evaluators can easily carry out paired comparisons of 7 evaluation categories. Increasing this number of evaluation categories would allow project attributes and impacts to be stratified into more groups. This would minimise the possibility that evaluators may ignore some project attributes or impacts in passing judgment.