AUTOMATED SHAPE-BASED PAVEMENT CRACK DETECTION APPROACH

Pavements are critical man-made infrastructure systems that undergo repeated traffic and environmental loadings. Consequently, they deteriorate with time and manifest certain distresses. To ensure long-lasting performance and appropriate level of service, they need to be preserved and maintained. Highway agencies routinely employ semi-automated and automated image-based methods for network-level pavement-cracking data collection, and there are different types of pavement-cracking data collected by highway agencies for reporting and management purposes. We design a shape-based crack detection approach for pavement health monitoring, which takes advantage of spatial distribution of potential cracks. To achieve this, we first extract Potential Crack Components (PCrCs) from pavement images. Next, we employ polynomial curve to fit all pixels within these components. Finally, we define a Shape Metric (SM) to distinguish crack blocks from background. We experiment the shape-based crack detection approach on different datasets, and compare detection results with an alternate method that is based on Support Vector Machines (SVM) classifier. Experimental results prove that our approach has the capability to produce higher detections and fewer false alarms. Additional research is needed to improve the robustness and accuracy of the developed approach in the presence of anomalies and other surface irregularities.


Introduction
Pavement management can be traced as early as the ancient Roman Empire, but pavement management using computer systems began during the 1970s. Advances in pavement health monitoring technologies and Pavement Management Systems (PMSs) have helped transportation agencies make discoveries about the best practices for preventive maintenance and pavement management (Vaitkus et al. 2016).
The 2017 American Society of Civil Engineers (ASCE) Infrastructure Report Card assigned a 'D' grade for United States road infrastructure, given that 1 out of every 5 miles of highway pavement is in poor condition (ASCE 2017). Efficient condition monitoring strategies for bridges and pavements can aid engineers in identifying developing distresses and scheduling maintenance early.
Highway and pavement condition monitoring techniques can broadly be classified under four major categories: deflection-based, image-based, wave propagationbased, and in situ sensing-based (Gopalakrishnan 2016). Each one addresses the health-monitoring objective from a different perspective and foundation.
Image-based pavement condition monitoring methods have a history of more than 30 years, and they have primarily been focused on pavement surface cracking, because that is one of the pavement distresses that can be easily captured through imaging. What began as windshield or manual surveys evolved into capturing analog photographs or videotapes, which were then processed to extract pavement-cracking information (McGhee 2004).
Although the current state-of-the-practice is to acquire 2D digital images of pavements using high-speed cameras mounted on a specialized data-collection van moving at highway traffic speeds, many State Highway Agencies (SHAs) have now moved towards pavement condition data collection using the so-called 3D systems that contain more information (elevation and intensity) than 2D images. Once the high-resolution digital images of the pavement surfaces are obtained, they are processed through a compression subsystem to achieve size reduction without loss of quality before they are stored. The images are then processed using various algorithms to ex-*Corresponding author. E-mail: rangan@iastate.edu tract cracking information and summary statistics, which are then recorded in the surface distress database (and can be linked to a PMS). In the commercial systems, the proprietary crack detection software algorithms and data formats are highly dependent on the vendors' data acquisition systems. For instance, the toolbox for automatic crack detection included in the RoadInspect software developed by Pavemetrics (Laurent et al. 2008) is entirely based on the proprietary Laser Crack Measurement System (LCMS) and uses a fis-data format that requires proprietary software library to read.
Advancements are still being made in the development of accurate and reliable image-based pavementcrack-detection and classification algorithms, including recent attempts to employ big data-driven Deep Learning (DL) approach (Some 2016;Zhang et al. 2016;Chen et al. 2009;Gopalakrishnan et al. 2017). There is a need for the development of automated, low-cost crack detection algorithms that can be implemented by highway agencies for cost-effective and continuous roadway condition monitoring and management. A brief review of the current state-of-the-practice in image-based highway pavement cracking data collection, processing, and reporting in the United States is summarized in Section 1. In Section 2, a detailed description of each stage of our vision-based crack detection approach is given. We present experimental evaluation of our approach in Section 3. The main conclusions are summarized in the last section.
1. Image-based pavement cracking data collection, processing and reporting: a brief review 1.1. Existing pavement cracking data collection practices Many state and local agencies in the United States employ highway-speed data-collection vehicles to collect pavement images, which are then processed using proprietary image processing algorithms to classify cracking type, extent, and severity. The Federal Highway Administration (FHWA) Long-Term Pavement Performance Program (LTPP) developed the Distress Identification Manual, which provides a consistent and uniform method to collect and report pavement distress data for the LTPP (Miller, Bellinger 2003). Recently, American Association of State Highway and Transportation Officials (AASHTO) provisional standards AASHTO PP 67:2016 and AASHTO PP 68:2014 have been developed to help 'achieve a significant level of standardization that will contribute to the production of consistent pavement condition estimates' . These two standards define the terminology and outline the procedures for collecting images of pavement surfaces and specifically quantifying crack distresses using automated methods.
Most SHAs have their own distress identification/ survey manuals that have been modified from the LTPP distress identification manual to fit each agency's data collection needs for pavement management and design.
According to National Cooperative Highway Research Program (NCHRP) Synthesis 401, Quality Management of Pavement Condition Data Collection: a Synthesis of Highway Practice, transverse cracking and fatigue cracking are among the distresses for which data are most commonly collected by highway agencies (Flintsch, McGhee 2009 (Miller, Bellinger 2003), while longitudinal cracking, thermal cracking, and reflective cracking data collection procedures for AC-surfaced pavements were often not consistent with the LTPP data collection procedures (Pierce, McGovern 2014).
As of 2012, more than 35 SHAs employed semiautomated and automated image-based methods for network-level pavement cracking data collection (Vavrik et al. 2013). The various sources of variability in pavement cracking data collection and processing for automated, semi-automated, and manual methods are summarized in Figure 1. Studies have highlighted that a noticeable bias exists in automated crack detection methods toward detecting high-severity cracking than low-severity cracking because high-severity cracking is in general, more readily identifiable than low-or medium-severity cracking (McQueen, Timm 2005;Flintsch, McGhee 2009).
NCHRP Synthesis 334, Automated Pavement Distress Collection Techniques: a Synthesis of Highway Practice, documents highway agency practices with regard to the automated collection and processing of pavement condition data techniques typically used in network-level pavement management. Factors that could potentially contribute to variability in automated pavement cracking data collection and processing practices adopted by various highway agencies (based on a survey conducted in 2003) are summarized below (McGhee 2004): -automated cracking data collection: agency, co ntract; -automated cracking data processing: agency, co ntract; -image capture: analog, digital, laser; -protocol use: AASHTO, LTPP, other; -monitoring frequency (years): 1, 2, 3; -reporting intervals: 100-300 m, 10-50 m, segment, other; -linear reference methods: mile post, latitude-longitude, link-node, log mile, other. A variety of pavement cracking data is desired by the SHAs, not only for their asset/pavement management activities, but also for FHWA's Highway Performance Monitoring System (HPMS) reporting requirements and for evaluating and calibrating the AASHTOWare Pavement ME Design software (http://me-design.com, currently being implemented by several SHAs). Recent changes in HPMS requirements demand that the state DOTs collect the following detailed cracking data (Vavrik et al. 2013;Zimmerman et al. 2013

Pavement crack detection and classification
Transportation Research Circular No E-C156, Automated Imaging Technologies for Pavement Distress Surveys, summarized the current state-of-the-art in the acquisition and processing of pavement surface images (Wang, Smadi 2011). In recent years, several advances have been made in image collection technology, equipment hardware and software, decoding and extraction methods, etc. A number of projects sponsored by SHAs, the NCHRP, and the FHWA have been initiated and completed with the objective of automating and improving image-based pavement distress detection and classification. Under High-Speed Rail IDEA Project 49, Ahuja and Barkan (2007) employed machine vision analysis by imaging both visible and infrared spectra of railroad equipment undercarriage for addressing incipient failure detection. A prototype of the machine vision inspection system was developed and tested at a passenger car service and inspection facility. Elkrry and Anderson (2014) provided a comprehensive summary of the network-level and project-level non-invasive imaging technologies applicable to pavement assessment. An Iowa DOT project (Neubauer, Todsen 2014) is investigating the use of acoustic imaging equipment to inspect bridge substructural elements.
More recently, studies have been exploring the potential for using 3D laser imaging technology for pavement distress surveys. Wang and Li (2014) proposed the use of 3D laser imaging for pavement surface data collection on the Oklahoma DOT Interstate network, including longitudinal profile, transverse profile, macro-texture, cracking, and various surface defects. Under a project sponsored by the Southern Plains Transportation Center, Wang (2016) is investigating the use of 1 mm 3D laser imaging (PaveVision3D system) for pavement surface characterization (mean texture depth, mean profile depth, etc.) related to pavement safety. An ongoing Florida DOT-sponsored research project (Roque 2014) is investigating the application of imaging techniques to evaluate the polishing characteristics of aggregates. An Ohio DOT/FHWA-sponsored research project (Wei et al. 2015) is currently investigating the use of a nonintrusive side-of-the-road camera to develop a Rapid Video-Based Vehicle Identification (RVIS) system.
A summary of selected studies in recent years that have focused on improving image-based pavement distress detection methods is provided in Table 1.

Shape-based crack detection
A crack is a thin and long road distress, characterized by its darker visual appearance. There exist several types of cracks, with different severity levels as discussed in Section 1. In this paper, we mainly focus on longitudinal and transverse cracks on concrete as well as asphalt pavement surfaces. An example of transverse crack on PCC pavement surface is given in Figure 2. To facilitate observation, the crack is surrounded by a block rectangle.
We propose a shape-based pavement crack detection approach, taking advantage of the spatial distribution of crack pixels. Our algorithm consists of Potential Crack Component (PCrC) extraction and shaping. It operates on a 75-by-75 pixel block. For each block, it first detects PCrCs, Then, it identifies their shapes and matches them against expected shapes. An explanation of notations that will be used throughout the rest of the paper is summarized in Table 2.

PCrC extraction
We design a framework for crack detection. It includes local filtering, Minor Component Removal (MCR) and Maximum Component Extraction (MCE), to extract PCrC from pavement blocks.

Local filtering
Typically, crack pixels are relatively lower in intensity values compared to non-crack pixels. Thus, pixel intensity value in a crack block follows bimodal distribution. Typi-cal example of bimodal histogram from a crack block is presented in Figure 3. In this figure, the left minor mode is for crack pixels and the right major mode is for noncrack pixels. Red circle represents the least frequent value between the two modes, known as the antimode. We make the observation that an efficient way for crack segmentation is to filter out pixels whose intensity values are above antimode.
Antimode is not easy to compute, especially for those blocks containing thin cracks. However, mean intensity value µ is a more accessible metric. We manually compute antimode values over a set of crack blocks, and formulate the relationship between µ and antimode. Experimental  results prove that the antimode value is very close to 0.8·µ for a crack block. Using this fact, we design a filter as given in Eq.
(1) to remove non-crack pixels from each pavement block: where: RawBlock (i) (x) and FliterBlock (i) (x) represent the image intensity at position x in input pavement block RawBlock (i) and output pavement block FliterBlock (i) , respectively; (i) is the mean intensity value of input pavement block RawBlock (i) ; f is parameter, which empirically set as 0.8 based on histogram analysis. Example in Figure 4 demonstrates the performance of local filtering. By comparing Figure 4a with Figure 4b, we make the observation that the local filter has the capability to extract whole crack from the concrete crack block. The filter also extracts some non-crack components, most of which are from groove segments and are generally in small sizes.

Minor component removal
We employ MCR operation to remove small-size, noncrack components in block FilterBlock. Any connected component whose size is smaller than T pixels is removed. Based on experimental observations, we set T as 10 for purpose of removing minor groove components while keeping cracks at low severity level. In our work, MCR operation is achieved by the MATALB function bwareaopen(). We experiment with MCR operation on crack block as shown in Figure 4b, and display its output in Figure 4c. By comparing these two images, we notice that the MCR operation removed most of non-crack components, leaving cracks and groove segments in relatively larger size.

Maximum component extraction
Crack segments in high and medium severity are in larger size compared to groove segments in concrete pavement surfaces. That means maximum component in MajorBlock corresponds to a crack when the block contains cracks at high and medium severity. We employ MCE operation to extract the maximum component and treat the component as PCrC. In our work, MCE operation is achieved by MATLAB function bwconncomp(). We experiment with MCE operation on crack block as shown in Figure 4c, and present its maximum component in Figure 4d. By comparing Figure 4d with Figure 4a, we make the observation that extraction process successfully extracted the whole high-severity crack.

Experimental observations
We experiment with PCrC extraction process on a set of pavement blocks, and make the following observations.
-Case 1: Continuous-Crack Block (CCB). PCrC extraction process has the capability to extract the whole crack. For such a crack block, size of its PCrC is higher than 150 pixels with probability as high as 99.99%. This can be explained by the fact that width of cracks in medium and high severity is larger than 2 pixels. An example of CCB is given in Figure 5. We present its PCrC in Figure 5 (fourth image from left to right), whose size is equal to 196 pixels. By comparing Figure 5d with Figure 5a, we notice that the PCrC overlaps with the crack very well. -Case 2: Noncontinuous-Crack Block (NCCB). PCrC extraction process fails to extract the whole crack. Output PCrC corresponds to the largest crack segment. Its size depends on crack width as well as interruption locations. For a NCCB containing cracks in low or medium severity, size of its PCrC is between 50 and 150 pixels in high probability. Fi gure 6a shows an example of NCCB, where one crack in medium severity is present. The crack consists of four segments. We show PCrC of the block in Figure 6d, whose size is equal to 121 pixels. By comparing Figure 6d with Figure 6a, we notice that the PCrC is from the largest, top segment of the crack. -Case 3: Non-Crack Block in Strong Tined Texture (NCBSTT). PCrC extraction process considers the largest groove segment as PCrC. For a NCBSTT, size of its PCrC lies between 50 and 150 pixels with probability as high as 90%. Figure 7a shows an example of NCBSTT. Its PCrC is presented in Figure 7d, with size equal to 130 pixels. -Case 4: Non-Crack Block in Light Tined Texture (NCBLTT). PCrC extraction process extracts the largest and darkest pavement patch as PCrC. For a NCBLTT, size of its PCrC is smaller than 50 pixels with probability as high as 99%. Figure 8a shows an example of NCBLTT. PCrC of the non-crack block is presented in Figure 8d. Its size is equal to 18 pixels. Based on the above discussion, we conclude that it is not proper to detect pavement cracks based on the area of PCrC alone. We need to develop a new metric to distinguish crack blocks from non-crack blocks.
By comparing Figure 6 with Figure 7, we notice that one major difference between NCCB and NCBSTT is the spatial distribution of potential crack pixels (i.e., white pixels) in block MajorIm. However, NCBSTTs differ from crack blocks mainly in the size of their PCrCs. To generalize the difference between crack and noncrack blocks, we make a few modification to the definition of PCrC. Details are given in Algorithm 1 (Figure 9).  Figure 9. PCrC modification algorithm In this algorithm, we replace MaxBlock with MajorBlock and FilterBlock under scenarios of NCCB, NCBSTT and NCBLTT, respectively, and define PCrC as the set of white pixels in binary block, MaxBlock. T1 and T2 are empirically set to 150 and 50 pixels, respectively. The difference between crack and non-crack blocks can be simplified as spatial distribution of pixels in PCrC. In the following section, we aim to develop a metric to measure the closeness of pixels in PCrC.

PCrC shaping
For a crack block, pixels in its PCrC are organized into a curve. However, for a non-crack block, pixels in its PCrC are typically distributed over whole block. Based on this fact, we decide to employ polynomial curve to fit all PCrC pixels, and then employ the average curve fitting error to measure closeness of PCrC pixels.

Polynomial curve fitting
As we do not know cracks' orientation in advance, we first fit crack pixels in both X and Y directions to handle scenarios of straight horizontal and vertical cracks. We then select the one producing smallest curve fitting error as the final polynomial fitting curve. In our work, polynomial curve fitting is achieved by MATLAB function polyfit(). Details are given as follows: where: { } 1 , , K CX cx cx = … and { } 1 , , K CY cy cy = … are vectors of X and Y coordinates of PCrC pixels; fpolyH and fpolyV are the returned polynomial fitting functions in the X and Y directions, respectively; n is the degree of curve fitting polynomials. We set n as 3 considering that cracks are generally smooth curves.
With fpolyH and fpolyV available, we compute their average curve fitting errors over all PCrC pixels. We employ MATLAB function polyval() to evaluate values of fitting polynomials at PCrC pixels. Details are given as follows: where: AveErr H and AveErr V are average curve fitting errors of fpolyH and fpolyV, respectively. Then average curve fitting error AveErr is defined as follows: An example of polynomial curve fitting is shown in Figure 10. We experiment with both vertical and horizontal polynomial curve fitting on the PCrC as shown in Figure 10. Curve fitting results are presented in Figure 10b and Figure 10c, where red lines represent fitting curves. Average curve fitting errors in Y and X direction are equal to 3.6303 and 120.8035, respectively. Based on curve fitting results, we conclude that the crack is in vertical direction, and the final average curve fitting error of the crack block is set as 3.6303.

Threshold filtering
With average polynomial curve fitting error AveErr available, we define a Shape Metric (SM) measuring closeness of PCrC pixels. Details are given as follows: It is worth mentioning that SM has the following features: -the SM value of a solid PCrC increases with its width, as shown in Figure 11b. Details to develop the chart are as follows. For each width value W, we create a vertical crack running across the block center. Horizontal width for each y follows the uniform distribution between 0.5·W and W in order to mimic true shapes of cracks. An example of a 25-pixel wide crack is shown in Figure 11a.
As the block size is set to be 75-by-75 pixels, the maximum SM value of a crack block is equal to about 0.007; -for distributed PCC from PCC modification operation, its SM value will be relatively much higher due to the fact, that pixels in PCC are widely distributed. Therefore, it is reasonable to detect cracks based on SM. A pavement block is considered as crack block if and only if its SM value is smaller than 0.08 (Detection Criterion). We execute PCrC modification operation on pavement blocks as shown in Figures 4-7, and compute their SM values. Computation results show that SM values of the four blocks are equal to 0.0112, 0.0591, 0.6235, and 1.2322, respectively. Based on the detection criterion, the first two and last two blocks are classified as crack and non-crack blocks respectively, which satisfy our expectation.

Baseline
Marques (2012) did a survey on previous works on crack detection, analyzing strengths and weakness of each approach. Based on the survey findings, they proposed one SVM based crack detection approach. The main idea of the approach is as follows: -create training set consisting of crack and noncrack samples; -extract statistic features, including minimum and mean intensity values, variance, and higher order moments (3rd and 4th), from each sample; -employ these statistics to train a SVM classifier. Experiment results proved that the SVM based approach performs best in comparison to previous approaches (Oliveria, Correia 2009 for some pavement datasets. Therefore, we choose it as the baseline for our approach to compare with.

Performance comparison
For each pavement image, we move the detection window from left to right, and then to the next row. We overlap detection windows in both horizontal and vertical directions so that each crack will be covered by more different windows. The stride (window overlap) is fixed to 25 pixels. Typically, a crack is long enough to be covered by tens of detection windows. We need to define proper metrics to measure performance of block-based crack detection approaches: -Complete Detection (CD). It occurs when a crack is completely covered by detection windows; -Partial Detection (PD). It occurs when a crack is partially covered by detection windows; -Misses (M). A miss appears when a crack is covered by no detection windows; -False Alarm (FA). Any connected component in non-crack areas is considered as a false alarm. It is worth mentioning that both shape and SVM based approaches fail to distinguish pavement joints from cracks as these blocks are similar to each other in terms of both statistic and shape features. One difference between them is that the joints are in straight lines. Using this fact, we first employ Hough transform to detect pavement joints. We then execute crack detection approach and remove any crack detections that intersect with pavement dividers. In the following figures, black lines represent detected longitudinal pavement joints.

High and medium severity pavement cracks
We experiment with both crack detection approaches on a set of pavement images, where cracks in high and medium severity are present. This dataset consists of 21 concrete and 24 asphalt pavement images. Total number of concrete and asphalt cracks are equal to 21 and 39, respectively.
An example of high severity concrete crack is presented in Figure 12a. Crack detection results of shape and SVM based approaches are shown in Figure 12b and Figure 12c, respectively. We make the following observations. Shape-based approach successfully detect the whole crack while producing 1 false alarms. The false alarm can be explained by the fact that shape of the black patch looks like a minor crack segment. However, SVM based approach partially detect the crack segment and produce 4 false alarms. The partial miss is due to thinness of the crack segment. Another example of asphalt pavement image is shown in Figure 13a, where three cracks in high severity are present. Crack detection results of shape and SVM based approaches are given in Figure 13b and Figure 13c, respectively. Shape-based approach completely detects cracks at middle and bottom, and partially detects the top crack. Note that the crack segment between the joints are detected but removed as detection windows intersect with joints. The partial miss of the top crack is due to the low contrast between the crack segment and background. However, SVM based approach partially detects cracks at top and bottom, and misses the crack at middle. The misses are due to high intensity values in pavement background around cracks.
Detection results on the whole dataset is summarized in Table 3. We make the observation that our shape-based approach achieves fewer misses as well as false alarms compared to SVM based approach. Besides, for partial detections, the average detection percentage of shape-based approach is higher compared to SVM based approach.

Low severity pavement cracks
We experiment with both approaches on a set of concrete pavement images where cracks at low severity are present. For this dataset, we only consider partial detections, as cracks at low severity are too thin to be detected complete-   ly. An example of concrete pavement image is presented in Figure 14a. This image contains one crack at high severity, and two cracks at low severity. Cracks at low severity are enclosed by black rectangles to facilitate observation. Crack detection results of shape and SVM based approaches are presented in Figure 14b and Figure 14c, respectively. The shape-based approach successfully detects both cracks at low severity while producing 1 false alarm. The false alarm is introduced due to presence of the small black patch. However, SVM based approach misses one crack at low severity and produces 2 false alarms.
Crack detection results on the whole dataset is summarized in Table 4. We make the observation that shapebased approach gives more detections as well as fewer false alarms.

Conclusions
The main objective of this research was to develop a shapebased pavement-crack-detection approach for the reliable detection and classification of cracks from acquired 2D concrete and asphalt pavement images. Concrete and asphalt pavement JPEG images acquired through the 2Darea-scanning digital-imaging method (dimensions of 3072×2048 pixels) were used for the analysis.
Our approach takes advantage of spatial distribution of potential pavement crack pixels. To achieve this, we design a framework, including local filtering, MCR and MCE, to extract PCrCs. We then define a SM based on curve fitting error to refine cracks from pavement background. We experiment with our crack detection approach on several datasets. Experimental results prove that our approach achieve higher detections as well as fewer false alarms compared to statistical learning based approaches like SVM. It is acknowledged that the test dataset used in this research for assessing the performance of the proposed approach did not include many of the problems one finds in real world applications, such as pavement markings, potholes, surface deterioration, skewed images/joints, shadows, bleeding, pumping etc. Additional research is needed to improve the speed, robustness and accuracy of the developed approach in the presence of anomalies and other surface irregularities.