https://drive.google.com/file/d/1uNQB2sqd4gj0v8UPJb-H3EvbhVNmQ4iv/view?usp=sharing

https://appchallenge2021-slope.cerexio.com/login?email=cobb@cerexio.com&pwd=123456

Satellite remote sensing is one of the predominant methods of acquiring terrestrial as well as extraterrestrial spatial data which measures the electromagnetic radiation that interacts with the surface of the earth, its core, the atmosphere, and the objects on it. Since the launch of Landsat-1 in 1972, satellite platforms have dramatically evolved along with their sensory capabilities. A typical present-day satellite is equipped with a variety of optical imaging sensors, which includes visible, reflective IR, multispectral and hyper spectral sensors, thermal IR imaging sensors, and radar imaging sensors [1, 2]. In addition, a number of non-imaging sensors are also included ranging from radiometers, altimeters, spectrometers, spectroradiometers, and LIDAR [1]. This myriad of sensory observations paves the way to the utilisation of Machine Learning (ML) techniques to process this multitude of information. Despite the abundance of applied machine learning (especially Deep Learning (DL)) in other domains, the techniques for soil condition monitoring and moisture level estimation is still in its infancy and the full potential of satellite remote sensing in this domain is yet to be uncovered [3, 4, 5, 6]. For instance, the authors illustrate the existing challenges associated with stand alone L-band and P-band based Interferometric SAR observations (eg: shorter soil penetration depths, higher surface scattering losses and greater variability of sensitivity across different vegetation types) as well as the challenges associated with the data, including different sampling resolutions, missing data, noise, low-quality data and the scarcity of the annotated data. We believe the full potential of DL can be uncovered by re-inventing the learning pipeline and better utilising the recent advances in the areas of self-supervised learning, long-term temporal pattern mining, and multi-sensor fusion.

System

Existing application framework will be used to process both satellite images, and external asset information along with ground environmental sensor data. The application will evaluate current slope safety index with confidence probability, and also forecast expected index using weather information using predefined learning models built around history data. 

All the information used to build the model can be transformed into simulation parameters, to identify the risk by changing values. The resulting information from simulation can be used as feedback for engineering teams and designing teams to validate settlement integrity. 

Features :

• Current/Predictive analytics of Slope Stability :

Identify the risk of existing slopes using history data, and predict based on the operational environment. Though the existing model is trained using different operational environment, different bands of satellite images, it needs to be incorporated based on Singapore's environments to get improved performance.  

• Prescriptive analytics using simulation model :

Recommend specific design parameters of slope construction based on history data, and movements. Understanding would happen scenarios using simplified interfaces. 

• Heat map representations/ GIS integrations: 

Visualise output using heat maps (possible failure regions, safety regions), integrate GIS with detailed asset information to easily explore, and navigate. 

• Slope risk cost performance modelling :

The outcome of the risk assessment informs decision-making regarding the requirement for risk mitigation and control strategies such as protection, stabilisation, early warning, as well as the need for increased knowledge of the slope physical characteristics (e.g., geology, groundwater), failure mechanisms and potential triggers; and monitoring requirements for managing residual risks. The calculated safety index will also be passed for financial elements to correctly determine future investments, and budget forecasting.

The data gathering from satellites will differ based on availability, and regional interest. Currently, Singapore operating TeLEOS-1 optical 1m resolution satellite since 2015, and going for TeLEOS-2 SAR based satellite with 1m Resolution, and full polarimetry (With a polarimetric SAR, we can obtain much more information than conventional SAR systems) to be launched at 2023. In addition to that another NeuSAR is expected to launch in 2022, which is a small SAR satellite with 2m resolution full polarimetry. Another type DS-SAR also to be launched in 2022 with 1m resolution capacity. 

We can obtain SAR images with different bands from TERRASAR-X/PAZ, KOMPSAT5, COSMOSKYMED or X Constellation with various SPOT resolution, and scene size. Once the images are gathered, system is incorporated with Persistent Scatterers SAR

Interferometry techniques with higher timer series of SAR images combined to identify point-like stable reflector which can overcome the problems with traditional Differential SAR interferometry method to improve the accuracy and reliability of deformation calculations. Our method has successfully worked in urban man made slopes with millimeter-level deformation measurement accuracy by overcoming the limitations of geometrical and temporal decorrelation as well as atmospheric disturbance. The coherent measurements on persistent scatterers allow us a terrain movement estimation with sub-millimeter accuracy, which allows us to predict the stability index with other soil correlation parameters based on the operational environment. 

Multi-Sensor Self-Supervised Learning

The proposed data fusion and analytic framework. We will be utilising sub-networks for each input modality which will allow us to learn individual salient factors of each modality irrespective of its density. Furthermore, we propose to employ memory buffers to store historical observations. In order to fully utilise this additional capacity and effectively map their long term dependencies, we will be employing Neural Memory Networks (NMN). Details regarding the specific architectural designs are provided in Stream 2. In this section, we illustrate how we address the data scarcity-related challenges identified in [7] and efficiently train the proposed framework.

The self-supervised learning paradigm is receiving more interest in the remote sensing literature as a mechanism to alleviate the overwhelming sizes of labeled data that are required to train present-day deep learning architectures. However, the pretext tasks (self supervised learning objectives) are limited to image inpainting [8] or change detection [9, 10, 11]. To the best of our knowledge, none of the existing methods have investigated the utility of cross-modal pretext tasks, where a particular sub-network receives a particular modality and learns to transfer this input to a secondary modality. 

For instance, the RGB sub-network will learn to transfer the input to thermal or IR image-based as its pretext task. Such cross-domain transfer has shown promising results in GAN-based learning frameworks [12] as the network gains the ability to compare and contrast the similarities and differences between the modalities while learning with noisy, incomplete, and heterogeneous data. In addition, we will leverage the temporal nature of the observations and will predict the observation of a particular modality at tth frame using past observations. Hence, our framework will utilise a combination of pretext tasks, the first work to explore such a learning paradigm, and will facilitate better capture of the semantics of the training data. After pre-training our framework with self-supervised learning, which will dramatically reduce the amount of annotated examples, we will fine-tune it for the end-user (downstream) tasks.

Multi-modal Temporal Analytic Intelligence

Apart from the direct outputs that are predicted from the model, we can leverage the long-term dependencies that are modelled in our NMN-based framework and regress temporal outputs. This is a prominent characteristic required for change detection in remote sensing. For instance, in order to better estimate hazardous regions for risk management, or to monitor the changes in soil quality, our model can seamlessly compare the prior templates that are stored in individual memories and estimate these measures of interest. Furthermore, actionable intelligence can be derived by evaluating the derived map changes and investigating the influential factors, which will be monitored by our multi-modal framework (eg: temperature, precipitation, human activities, etc).Moreover, this multi-granularity temporal framework can handle the temporal nonuniformity of the sampled data. As the memory buffer is preserving the historical data it can sample information from different temporal resolutions while making a particular instance. Therefore, there is no need for the frequency of the data sampling to be uniform making this ideal for SAR based applications.

Multi-modal Spatial Analytic Intelligence

The proposed GCNN outputs multiple analytics. For instance, in the slope stability estimation task, the framework will generate multiple analyses such as soil moisture maps and hazard maps indicating the areas that could be affected in the future. While it is popular to design and train separate networks for separate tasks, we employ multi-task learning using one representation network with multiple outputs to take advantage of the relationships among related tasks. This multi-task model will be trained using the Gradient blending algorithm proposed in [16]. By sharing representations between related tasks, our platform will focus its attention on those features that actually matter; reduce the risk of overfitting by an order of N, where N is the number of tasks; and lessen the impact of task-specific noises and representation bias, enabling the model to generalise better. 

[1] L. Zhu, J. Suomalainen, J. Liu, J. Hyyppä, H. Kaartinen, H. Haggren et al., “A review: Remote sensing sensors,” Multi-purposeful application of geospatial data, pp. 19–42, 2018.

[2] S. P. Mertikas, P. Partsinevelos, C. Mavrocordatos, and N. A. Maximenko, “Environmental applications of remote sensing,” in Pollution Assessment for Sustainable Practices in Applied Sciences and Engineering. Elsevier, 2021, pp. 107–163.

[3] M. Ottinger and C. Kuenzer, “Spaceborne l-band synthetic aperture radar data for geoscientific analyses in coastal land applications: A review,” Remote Sensing, vol. 12, no. 14, p. 2228, 2020.

[4] S. H. Alemohammad, T. Jagdhuber, M. Moghaddam, and D. Entekhabi, “Soil and vegetation scattering contributions in l-band and p-band polarimetric sar observations,” IEEE Transactions on Geoscience and Remote Sensing, vol. 57, no. 11, pp. 8417–8429, 2019.

[5] M. U. Müller, J. D. Shepherd, and J. R. Dymond, “Support vector machine classification of woody patches in New Zealand from synthetic aperture radar and optical data, with lidar training,” Journal of Applied Remote Sensing, vol. 9, no. 1, p. 095984, 2015.

[6] B. Chen, X. Xiao, H. Ye, J. Ma, R. Doughty, X. Li, B. Zhao, Z. Wu, R. Sun, J. Dong et al., “Mapping forest and their spatial–temporal changes from 2007 to 2015 in tropical hainan island by integrating alos/alos-2 l-band sar and landsat optical images,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 11, no. 3, pp. 852–867, 2018.

[7] A. Karpatne, I. Ebert-Uphoff, S. Ravela, H. A. Babaie, and V. Kumar, “Machine learning for the geosciences: Challenges and opportunities,” IEEE Transactions on Knowledge and Data Engineering, vol. 31, no. 8, pp. 1544–1554, 2018.

[8] C. Tao, J. Qi, W. Lu, H. Wang, and H. Li, “Remote sensing image scene classification with self-supervised paradigm under limited labeled samples,” IEEE Geoscience and Remote Sensing Letters, 2020.

[9] X. Niu, M. Gong, T. Zhan, and Y. Yang, “A conditional adversarial network for change detection in heterogeneous images,” IEEE Geoscience and Remote Sensing Letters, vol. 16, no. 1, pp. 45–49, 2018.[10] B. Hou, Q. Liu, H. Wang, and Y. Wang, “From w-net to cdgan: Bitemporal change detection via deep learning techniques,” IEEE Transactions on Geoscience and Remote Sensing, vol. 58, no. 3, pp. 1790–1802, 2019.

[11] H. Dong, W. Ma, Y. Wu, J. Zhang, and L. Jiao, “Self-supervised representation learning for remote sensing image change detection based on temporal prediction,” Remote Sensing, vol. 12, no. 11, p. 1868, 2020.

[12] A. Almahairi, S. Rajeshwar, A. Sordoni, P. Bachman, and A. Courville, “Augmented cyclegan: Learning many-to-many mappings from unpaired data,” in International Conference on Machine Learning. PMLR, 2018, pp. 195–204.

[13] T. Fernando, S. Denman, A. McFadyen, S. Sridharan, and C. Fookes, “Tree memory networks for modelling long-term temporal dependencies,” Neurocomputing, vol. 304, pp. 64–81, 2018.

[14] T. Fernando, S. Denman, S. Sridharan, and C. Fookes, “Going deeper: Autonomous steering with neural memory networks,” in Proceedings of the IEEE International Conference on Computer Vision Workshops, 2017, pp. 214–221.

[15] D. Priyasad, T. Fernando, S. Denman, S. Sridharan, and C. Fookes, “Memory based fusion for multi-modal deep learning,” Information Fusion, vol. 67, pp. 136–146, 2021.

[16] W. Wang, D. Tran, and M. Feiszli, “What makes training multimodal classification networks hard?” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 12 695–12 705.

#savetheplanet #landsat #landsat9 #predictiveanalytics

This project has been submitted for consideration during the Judging process.