NCKU MMDB

https://youtu.be/Q4XOQRdLVO0

https://github.com/love3499/NASAHackathon-2021-NCKU-MMDB

Active Inspection Platform for Abnormal Road Area

Fig 1. shows the system architecture diagram of "Active Inspection Platform for Abnormal Road Area".

This system includes three modules: Hidden Abnormal Road Inspection Module, UAV Road Inspection Module, and Open Inspection Platform for Abnormal Road Area. The hidden danger area inspection module is designed to quickly locate hidden dangerous bridges and roads based on satellite data and assist the government to quickly formulate management and maintenance strategies. UAV Road Inspection Module completes the automated execution of the bridge defect inspection process, and outputs a complete inspection report for inspection workers to confirm the damage status of the bridge. Finally, the Open Inspection Platform for Abnormal Road Area integrates the functions of the above two modules, redefines the maintenance process of bridges and roads, and presents alarm information on the platform. By doing so, the efficiency of the emergency response functions of regional natural disaster and road maintenance for the government can have a great improvement via this platform, and the cost gap of road maintenance between mountainous areas and urban can be reduced.

Hidden Abnormal Road Inspection Module

Fig. 2 shows the structure of Hidden Abnormal Road Inspection Module.

1. Step 1. Satellite imagery tagging module or natural disasters integrates the earthquake data, mudslide data, and precipitation data provided by NASA and plots these information in the cloud image. The shades of color are used to indicate the range and level of the above open data.
2. Step 2. Input the satellite image with natural disaster information in Step 1 into the inspection module via CNN for hidden natural disaster positioning module for image feature extraction.
3. Step 3. Enter the feature extraction results in step 2 into unsupervised learning image classification for area classification, and finally output three categories: damaged area, potential abnormal area, and safe area, to assist the government, inspection workers, and residents to quickly locate the risk of damage Road or bridge.

UAV Road Inspection Module

Fig. 3 shows the structure of UAV Road Inspection Module.

1. Step 1. Inspection workers use UAV to collect bridge image data and sensor data and upload them to the data analysis module.
2. Step 2. Input the bridge image into the defect type and location recognition module (YOLOv4), and obtain the bridge location category information through model recognition.
3. Step 3. According to the bridge location and image information obtained in Step 2., the scope marking module (U-Net) performs defect detection and confirms the type of defect. Finally, it can colorize the defect feature in the image.
4. Step 4 The defect area and location calculation module calculates the degree of damage of the bridge defect based on the image information of the bridge defect obtained in Step 3.
5. Step 5. The inspection report generation module generates a bridge inspection report according to the D.E.R.U evaluation method for inspection workers to evaluate and confirm the damage condition of the bridge.

Open Inspection Platform for Abnormal Road Area

Our website relies on obtaining satellite cloud images from the public API of NASA, and returns the target satellite cloud images based on the date and landmark selected by the user. Satellite cloud images can be used for more in-depth analysis, allowing users to know the current status of each landmark in real time. We use NodeJS to set up the website server. In order to speed up the development, we also use jQuery for partial writing. The map part is constructed with Leaflet. Leaflet is used because of the cross-platform, secondary development threshold and lightness. 3D modeling uses Three.js, the reason is that Three.js encapsulates WebGL well, which greatly reduces learning costs and development complexity. With real-time detailed information and 3D modeling, users can quickly know the road sections to pay attention to. The government can also be notified when the construction is damaged and know the damaged part, reducing a large number of inspection and maintenance costs and effectively increasing the life of the building.

Sustainable Development Indicators

• Sustainable
• Traditionally, people and bridge inspection vehicles are regularly dispatched in according to the law. Relying on professional inspection experience and the cost is quite high. Our platform is low in maintenance costs, and the accuracy of AI defect detection technology is increasing year by year, which will definitely become a future trend.
• Disaster-risk resilient
• Nowadays, repairs are often carried out after a disaster or damage occurs, and lack the ability to warn local residents. Our platform can integrate satellite information in real time through AI, actively identify hazardous areas, and warn the government and drivers.
• Inclusive
• Nowadays, Road hazards in urban areas are maintained more often than in remote areas. Our platform combines information of satellite and UAG to find hazardous areas everywhere, all people can report the road hazards and obtain road safety information immediately.

Expected effect

1. Tradition relies on manpower. → Active prediction of abnormal road damage location.
2. Old closed road maintenance procedures. → The open platform is available to residents, governments, and maintenance personnel.
3. Difficulty in overhaul and maintenance in remote areas. → Integrate satellite and UAV information to achieve regional equality.

The satellite cloud images and disaster information provided on the NASA Worldview and NASA SEDAC websites inspired us to innovate the bridge and road maintenance procedures. As a result, a platform was established to improve the efficiency of the emergency response functions of regional natural disaster and road maintenance for the public agency. At the same time, the cost gap of road maintenance between mountainous areas and urban can be reduced. In this study, these satellite cloud images and the information from various disasters were adopted to draw our own satellite cloud images with the marks of several information including earthquakes, mudslides, and rainfall. The image will be input to the unsupervised learning and active detection model, enabling the model to analyze the risk in each area. Besides, the Data Mining technology was applied to construct a model in warning the exceeding value, and output the easy-to-be-understood warning information (eg, the exceeding information of regional rainfall). Finally, the information would be presented on our platform.

As of the UAV Road Inspection Module, our team collected aerial images of roads and bridges with UAV and input the images into the model for structural defect analysis of bridges. Eventually, the bridge structure analysis report and 3D model of the bridge were output, and the defects in the bridge were marked out in the modeling image. This bridge image collected by our team was also sent to the 2020 UAV Innovation Application Project and Hackathon Competition, and finally won the first place in the competition.

In the future, the data collected in this project will also be published on the platform as a reference for other inspection teams with the hope of providing other UAV inspection workers with standardized procedures to create an open data of Active Inspection Platform for Abnormal Road Area.

The team has been committed to research and development combination of UAV and AI image recognition technology for high-altitude inspection and other related applications. In the past, the bridge and road inspection system we developed can only assess the degree of damage to bridges and roads through regular inspections. It is difficult to find the abnormal structure of the bridges and roads early.

Inspired by the NASA hackathon competition and NASA Open data, our team focused more on helping to improve the sustainable development and disaster resilience of urban infrastructure.

Our team proposes a new type of bridge and road inspection method, hoping to quickly locate bridges and roads with safety concerns after a disaster, and provide an active bridge defect inspection platform to contribute to the sustainable development of the city.

[1] AlexeyAB/darknetPaper Yolo v4: https://arxiv.org/abs/2004.10934 More details: medium link Discussion: Reddit About Darknet framework…github.com

[2] https://heartbeat.comet.ml/deep-learning-for-image-segmentation-u-net-architecture-ff17f6e4c1cf

[3] https://worldview.earthdata.nasa.gov/

[4] https://data.nasa.gov/Earth-Science/Global-Landslide-Catalog/h9d8-neg4

[5] https://sedac.ciesin.columbia.edu/data/set/ndh-earthquake-frequency-distribution/maps/services

[6] D. Cavaliere, V. Loia, A. Saggese, S. Senatore and M. Vento, "Semantically Enhanced UAVs to Increase the Aerial Scene Understanding," in IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 49, no. 3, pp. 555-567, March 2019, doi: 10.1109/TSMC.2017.2757462.

[7] Y. Qiming, Z. Jiandong and S. Guoqing, "Modeling of UAV path planning based on IMM under POMDP framework," in Journal of Systems Engineering and Electronics, vol. 30, no. 3, pp. 545-554, June 2019, doi: 10.21629/JSEE.2019.03.12.

#Road Inspection, #Deep Learning, #Unsupervised Learning, #UAV

This project has been submitted for consideration during the Judging process.