All Issue

2023 Vol.13, Issue 1 Preview Page

Scientific Paper

Automated pavement patches detection using low cost vision system and semantic image segmentation

저비용 비전 시스템과 의미론적 이미지분할을 이용한 포장 패치 자동 탐지 시스템

Jung Hee Lee1*
,
Heungbae Gil2
이 정희1*
,
길 흥배2
1Deputy General Manager, ICT Convergence Research Division, Korea Expressway Corporation Research Institute
2Research Director (Ph.D.), ICT Convergence Research Division, Korea Expressway Corporation Research Institute
1한국도로공사 도로교통연구원 ICT융합연구실 차장
2한국도로공사 도로교통연구원 ICT융합연구실 연구위원
*Corresponding Author
11 July 2023. pp. 98-113
PDF XML Citation
Abstract

Local road management agencies, which require frequent road surface inspections to maintain a safe driving environment, face difficulties in manually assessing road conditions. Moreover, stopping on the shoulder of a high-speed highway to inspect the road surface is extremely dangerous for inspectors. To overcome these challenges, there have been many efforts to automate road surface image acquisition, assessment, and analysis. However, existing methods mainly focus on detecting damage such as potholes and cracks, and have limitations in monitoring the condition of road surfaces, such as the need for repairs or the status of maintenance. This study proposed an automatic pavement patch detection method incorporating image processing and deep neural network image segmentation to provide a global view of the road surface. The panoramic image of the road is obtained by converting the video images captured by the omnidirectional camera of vehicles into top-view images using inverse perspective and affixing the images together. Next, the U-Net-architecture-based deep neural network image segmentation technique is applied to detect patches. Testing using actual highway driving videos revealed performances of 0.827 and 0.821 in the U-Net model based on ResNet152 and EfficientNetb7 backbones.

Keywords
Pavement patch detection
U-net
Automated pavement monitoring system
Inverse perspective mapping
References
1
Abbas, S.A. and Zisserman, A. (2019). “A geometric approach to obtain a bird’s eye view from an image”, Proceedings - 2019 International Conference on Computer Vision Workshop, ICCVW. 10.1109/ICCVW.2019.00504
2
Alam, M.Y., Nandi, A., Kumar, A., Saha, S., Saha, M., Nandi, S. and Chakraborty, S. (2020). “Crowdsourcing from the true crowd: device, vehicle, road-surface and driving independent road profiling from smartphone sensors”, Pervasive Mob. Comput., 61. 10.1016/j.pmcj.2019.101103
3
Cao, M.T., Tran, Q.V., Nguyen, N.M. and Chang, K.T. (2020). “Survey on performance of deep learning models for detecting road damages using multiple dashcam image resources”, Advanced Engineering Informatics, 46. 10.1016/j.aei.2020.101182
4
Chang, K.T., Chang, J.R. and Liu, J.K. (2005). “Detection of pavement distresses using 3D laser scanning technology”, Proceedings of the 2005 ASCE International Conference on Computing in Civil Engineering, Computing in Civil Engineering, pp. 1085-1095. 10.1061/40794(179)103
5
Chen, T., Cai, Z., Zhao, X., Chen, C., Liang, X., Wang, P. and Zou, T. (2020). “Pavement crack detection and recognition using the architecture of segNet”, J. Ind. Inf. Integr., 18. 10.1016/j.jii.2020.100144
6
Chun, C. and Ryu, S.K. (2019). “Road surface damage detection using fully convolutional neural networks and semi-supervised learning”, Sensors (Switzerland), 19(24). 10.3390/s1924550131842513PMC6961057
7
Coenen, T.B.J. and Golroo, A. (2017). “A review on automated pavement distress detection methods”, Cogent Engineering, 4(1). 10.1080/23311916.2017.1374822
8
Dharneeshkar, J.S., Soban Dhakshana, V., Aniruthan, S.A., Karthika, R. and Parameswaran, L. (2020). “Deep learning based detection of potholes in Indian roads using YOLO”, Proceedings of the 5th International Conference on Inventive Computation Technologies, ICICT, pp. 381-385.
9
Gopalakrishnan, K. (2018). “Deep learning in data-driven pavement image analysis and automated distress detection: A review”, Data, 3(3), p. 28. 10.3390/data3030028
10
Hadjidemetriou, G.M., Christodoulou, S.E. and Vela, P.A. (2016). “Automated detection of pavement patches utilizing support vector machine classification”, Proceedings of the 18th Mediterranean Electrotechnical Conference: Intelligent and Efficient Technologies and Services for the Citizen, MELECON. 10.1109/MELCON.2016.7495460
11
Hadjidemetriou, G., Serrano, M.M., Vela, P.A. and Christodoulou, S. (2015). “Patch defects in images using support vector machines”, Civil-Comp Proceedings CIVIL-COMP 2015 - 15th International Conference on Civil, Structural and Environmental Engineering Computing, Civil-Comp Press.
12
Hartley, R. and Zisserman, A. (2004). Multiple view geometry in computer vision, Cambridge University Press. 10.1017/CBO9780511811685
13
Inzerillo, L., Mino, G. di and Roberts, R. (2018). “Image-based 3D reconstruction using traditional and UAV datasets for analysis of road pavement distress”, Automation in construction, 96, pp. 381-385. 10.1016/j.autcon.2018.10.010
14
Jo, Y. and Ryu, S. (2015). “Pothole detection system using a black-box camera”, Sensors (Switzerland), 15(11), pp. 29316-29331. 10.3390/s15112931626610498PMC4701334
15
Kang, B.H. and Choi, S.I. (2017). “Pothole detection system using 2D LiDAR and camera”, International Conference on Ubiquitous and Future Networks, ICUFN. 10.1109/ICUFN.2017.7993890
16
Kingma, D. P. and Ba, J. L. (2015). “Adam: A method for stochastic optimization”, 3rd International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings.
17
Laraqui, A., Baataoui, A., Saaidi, A., Jarrar, A., Masrar, M. and Satori, K. (2017). “Image mosaicing using voronoi diagram”, Multimed Tools Appl., 76(6), pp. 8803-8829. 10.1007/s11042-016-3478-z
18
Lei, X., Liu, C., Li, L. and Wang, G. (2020). “Automated pavement distress detection and deterioration analysis using street view map”, IEEE Access, 8, pp. 76163-76172. 10.1109/ACCESS.2020.2989028
19
Lin, T.Y., Goyal, P., Girshick, R., He, K. and Dollar, P. (2020). “Focal loss for dense object detection”, IEEE Trans Pattern Anal Mach Intell, 42(2), pp. 318-327. 10.1109/TPAMI.2018.285882630040631
20
Liu, Z., Cao, Y., Wang, Y. and Wang, W. (2019). “Computer vision-based concrete crack detection using U-net fully convolutional networks”, Automation in Construction, 104, pp. 129-139. 10.1016/j.autcon.2019.04.005
21
Maeda, H., Sekimoto, Y., Seto, T., Kashiyama, T. and Omata, H. (2018). “Road damage detection and classification using deep neural networks with smartphone images”, Computer-Aided Civil and Infrastructure Engineering, 33(12), pp. 1127-1141. 10.1111/mice.12387
22
Mei, Q. and Gül, M. (2020). “A cost effective solution for pavement crack inspection using cameras and deep neural networks”, Constr. Build. Mater., 256. 10.1016/j.conbuildmat.2020.119397
23
Meocci, M., Branzi, V. and Sangiovanni, A. (2021). “An innovative approach for high-performance road pavement monitoring using black box”, J. Civ. Struct. Health Monit., 11(2), pp. 485-506. 10.1007/s13349-020-00463-8
24
Pan, Y., Zhang, X., Sun, M. and Zhao, Q. (2017). “Object-based and supervised detection of potholes and cracks from the pavement images acquired by UAV”, International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences - ISPRS Archives, 42(4W4), pp. 209-217. 10.5194/isprs-archives-XLII-4-W4-209-2017
25
Park, J., Lee, J.H., and Bang, J. (2021) “Potholeeye+: Deep-learning based pavement distress detection system toward smart maintenance”, CMES - Computer Modeling in Engineering and Sciences, 127(3), pp. 965-976. 10.32604/cmes.2021.014669
26
Patra, S., Middya, A.I. and Roy, S. (2021). “PotSpot: Participatory sensing based monitoring system for pothole detection using deep learning”, Multimed Tools Appl., 80(16), pp. 25171-25195. 10.1007/s11042-021-10874-4
27
Peraka, N.S.P. and Biligiri, K.P. (2020). “Pavement asset management systems and technologies: A review”, Automation in Construction, 119. 10.1016/j.autcon.2020.103336
28
Quintana, M., Torres, J. and Menéndez, J.M. (2016). “A simplified computer vision system for road surface inspection and maintenance”, IEEE Transactions on Intelligent Transportation Systems, 17(3), pp. 608-619. 10.1109/TITS.2015.2482222
29
Radopoulou, S.C. and Brilakis, I. (2014). “Improving patch defect detection using vision tracking on video data”, EG-ICE 2014, European Group for Intelligent Computing in Engineering - 21st International Workshop: Intelligent Computing in Engineering.
30
Radopoulou, S.C. and Brilakis, I. (2015). “Patch detection for pavement assessment”, Automation in Construction, 53, pp. 95-104. 10.1016/j.autcon.2015.03.010
31
Radopoulou, S.C. and Brilakis, I. (2017). “Automated detection of multiple pavement defects”, Journal of Computing in Civil Engineering, 31(2). 10.1061/(ASCE)CP.1943-5487.0000623
32
Radopoulou, S.C., Jog, G.M. and Brilakis, I. (2013). “Patch distress detection in asphalt pavement images”, ISARC 2013 - 30th International Symposium on Automation and Robotics in Construction and Mining, Held in Conjunction with the 23rd World Mining Congress. 10.22260/ISARC2013/0176
33
Ragnoli, A., de Blasiis, M.R. and di Benedetto, A. (2018). “Pavement distress detection methods: A review”, Infrastructures, 3(4), p. 58. 10.3390/infrastructures3040058
34
Ronneberger, O., Fischer, P. and Brox, T. (2015). “U-net: Convolutional networks for biomedical image segmentation”, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 9351. 10.1007/978-3-319-24574-4_28
35
Sarsam, S.I. (2016). “Pavement maintenance management system : A review”, Trends in Transport Engineering and Applications, 3(2), pp. 19-30.
36
Silva, L.A., Blas, H.S.S., García, D.P., Mendes, A.S. and González, G.V. (2020). “An architectural multi-agent system for a pavement monitoring system with pothole recognition in uav images”, Sensors (Switzerland), 20(21), pp. 1-23. 10.3390/s2021620533143311PMC7663208
37
Ukhwah, E.N., Yuniarno, E.M. and Suprapto, Y.K. (2019). “Asphalt pavement pothole detection using deep learning method based on YOLO neural network”, Proceedings - 2019 International Seminar on Intelligent Technology and Its Application, ISITIA. 10.1109/ISITIA.2019.8937176
38
Wang, Z. and Yang, Z. (2020). “Review on image-stitching techniques”, Multimedia Systems, 26(4), pp. 413-430. 10.1007/s00530-020-00651-y
39
Wu, H., Yao, L., Xu, Z., Li, Y., Ao, X., Chen, Q., Li, Z. and Meng, B. (2019). “Road pothole extraction and safety evaluation by integration of point cloud and images derived from mobile mapping sensors”, Advanced Engineering Informatics, 42. 10.1016/j.aei.2019.100936
40
Yang, Q. and Ji, X. (2021). “Automatic pixel-level crack detection for civil infrastructure using Unet++ and deep transfer learning”, IEEE Sens J., 21(17), pp. 19165-19175. 10.1109/JSEN.2021.3089718
41
Zakeri, H., Nejad, F.M. and Fahimifar, A. (2017). “Image based techniques for crack detection, classification and quantification in asphalt pavement: A review”, Archives of Computational Methods in Engineering, 24(4), pp. 935-967. 10.1007/s11831-016-9194-z
42
Zhang, L., Wen, T. and Shi, J. (2020). “Deep image blending”, Proceedings - 2020 IEEE Winter Conference on Applications of Computer Vision, WACV. 10.1109/WACV45572.2020.9093632
43
Zhu, J., Zhong, J., Ma, T., Huang, X., Zhang, W. and Zhou, Y. (2022). “Pavement distress detection using convolutional neural networks with images captured via UAV”, Automation in Construction, 133. 10.1016/j.autcon.2021.103991
Information
  • Publisher :KOREAN ASPHALT INSTITUTE
  • Publisher(Ko) :한국아스팔트학회
  • Journal Title :Journal of the Korean Asphalt Institute
  • Journal Title(Ko) :한국아스팔트학회지
  • Volume : 13
  • No :1
  • Pages :98-113
  • Received Date : 2023-06-15
  • Revised Date : 2023-06-24
  • Accepted Date : 2023-06-29
  • DOI :https://doi.org/10.22702/jkai.2023.13.1.9
Journal Informaiton Journal of the Korean Asphalt Institute Journal of the Korean Asphalt Institute
Online submission
  • KISTI Current Status
  • KISTI Cited-by
  • KCI 문헌 유사도 검사
  • orcid
  • open access
  • ccl
Journal Informaiton Journal Informaiton - close