OREO

Universal Event | Measuring the Value of Earth Observations

Rainfall Pattern Prediction

High-Level Project Summary

Our aim is to investigate predictions based on the rainfall data and understand how we can improve on current methods to better locate natural disasters like rain induced flooding and landslides or better predict rain patterns for rainfed agriculture using Machine Learning algorithms for the predictions. Our model is trained by combining several neural networks which are trained seperately for seperate task.

Link to Project "Demo"

https://youtu.be/A2BslVs0QtA

Link to Final Project

https://github.com/RainfallPatternPrediction/

Detailed Project Description

Intro:

There is a lot of data present and a lot of our daily lives depend on it from checking the weather to finding the fastest route to ur destination. In this project we looking into a project on which our team has created using data provided by NASA and we investigate how this project would effect people and how data collection can be improved for better usage in the future.
Our aim with this project is to gain insight in how rainfall data is used to make predictions (more that just predicting when the next rain fall) like prediciting probability of landslides and also for arrival and retreat of monsoon and so on.
We have made our own model to explore the data and make meaningful predictions.

Project:

Obtaining and Parsing Data:

We started our plan by first obtaining the data(refrence 1 and 2). We proceeded by creating a simple chrome extension which will automatically download the data(code in refrence 3).
Now that we had our data we needed to process it and extract the information we needed namely the calculated precipitation data w.r.t the spatial positions and time. The following code extracted the desired data and color coded them based on rain intensity. Since all places on earth get rain on a daily basis we made a cutoff at 5mm/day. This cutoff was chosen as 5mm/day (about 0.2mm/hour) as it is considered light/moderate-light rain. A lower cutoff would have resulted in a cluttered view and a harder time for the Neural Network to make useful predictions(code in refrence 3).
*( shows cluttering as cutoff is decreased from 5mm/day -> 1mm/day -> 0.5 mm/day)*
we now have our data separated and filtered and in a convenient format with a 99.781% compression (30.5mb/file -> 66.8kb/image *on avg*). We now create a simple website to visualize the data(refrence 3) and also made a few saved animations (refrence 4).

Data Analysis :

From these visualizations we can see that when we increase the fps there is a slight bobbing of the range of the rainfall pattern along the longitudes. This is caused by the rotation of the earth on a slightly skewed axis. This is the same as how seasons arise and we can also see how the monsoon season approach and retreat which is a really useful way to make more informed decisions for rainfed agricultural needs.
We can also see how the rain pattern propagates longitudinally from left to right along the top and bottom and from right to left in the middle. This effect is actually the Coriolis effect in action.

Prediction of Data:

Now we continue with our plans for prediction of future rainfall data. We begin with creating a layout for how we would set up the neural networks and after lots of trial and error we ended up with this plan (code in refrence 3).

From our chrome extension we had downloaded about 6700 files amassing 250GB in storage and about 480MB when converted into the image. The first step of the plan was to convolve the images into an even smaller size with each convolved image being 4.32kb and a dimension of (120 x 60)px.
This convolution was necessary to decrease the dimensionality of the images for the prediction neural network. A Convolutional Neural Network was not used as it didn’t product as clear and accurate images and same for LSTM and CLSTM (Convolutional Long Short Term Memory ) models.
After the predictions were made in the convolved from , we denoised it by another neural network so that it may smoothen the images when they are eventually deconvolved by yet another neural network.
After all these individual models had trained we join them into a singular bigger model. This process is often used in Auto Encoders and Generative Adversarial Networks (GANs).
Here were the results for each step in the plan:
sept 1st 2021
convolution
prediction
denoising
deconvolution
actual rainfall on 2nd sept
convolution of 2nd sept

(comparison of convolved prediction after denoise of 2nd sept and actual convolution of 2nd sept )

Network Analysis:

We can see that the deconvolution and the correct result are definately not the same, but taking into concideration that the model was only train on data only till 23 May 2017 (from 1st july 2000) and still made an somewhat accurate prediction 4 years into the future is a good sign. Also when comparing the denoised with the convolution of 2nd sept we find that it managed to capture a lot of the key aspects such as the various branching at Asia and at the Atlantic Ocean so this narrows down to the effectiveness of the convolution-deconvolution pair.
Our main reason for the low training of the model is due to hardware limitations and limited time.
Our result can be applied with previous mentioned uses like seasonal rainfall pattern prediction and storm warnings.
This project was a large scale visualization and understanding of rain patterns but the same idea is applied on a singular region or city where the data is more accurate and much smaller, giving rise to more complex analysis and accurate predictions. This is also how your phone knows when there is going to a bad weather soon.

Code

The code for each part of the plan is here : Convolution, Predict, Denoise, Decode.

Space Agency Data

All the data was taken from NASA more specifically from the GES DISC section ( refrence 1 and 2 ). Nasa Earth Data Search was also used to see how the data mapped and acted as in inspiration for our visuals.

Hackathon Journey

The Main reason we took this specific challenge was based on the fact that looking at news channels and finding that a late monsoon has caused massive agricultural problems or reading about how prolonged rain has cause flooding and caused damage worth billions and we wanted to do our best to explain and contribute to the prevention of these problems.

Our working on the problem actually begain with researching about rainfall and flooding and how they are correlated. We also looked at what are the current measures taken to stop and predict these issues were and saw just a few and vague explaniation of predictions based on neural networks. Thats when we knew that we had to do something about it.

We had prior experience with Machine Learning modules like TF (tensorflow) and we started to work on it, we tried lots and lots of models and small tweaks and it still didnt seem to work. It was a real blockade in front of us and our destination. But with consistant efforts we chipped away at the problems one part at a time. Our main issue was the deconvolution didnt work out properly but after changing hyperparameters and retraining it for hours, we got it to something decent.

We understood that a perfect prediction is not possible as rainfall prediction isn’t just looking at the current patter and working from there but its dependent on a lot of other factors which change millions of times in day and their average can differ greatly from day to day such as the wind direction, pressure, humidity, temperature, etc. Knowing all this we still continued as a model doesn’t need to be 100% accurate to give meaningful insights on topics and our model would be great to understand how the general pattern changes over time

We were very nerve racked as the training for each individual model took about 4-7h and we had no idea if it would work after so many failed attempts. But when the dust settled we were happy to see that it had worked! There were some minor prediction errors but those were expected by such a simple and naive model.

References

Global Judging

This project has been submitted for consideration during the Judging process.