High-Level Project Summary
This challenge consists of establishing a risk scale of space-weather caused satellite resets, which is the risk of satellite computers shutting down because of strong radiation in space. Data on single-event upsets taken directly from the Canadian satellite CASSIOPE was meshed with open space weather datasets to determine what factors cause the upsets, and devise a way to estimate the risk that the satellite would be affected by space radiation, enabling better satellite operations.
Link to Project "Demo"
Link to Final Project
Detailed Project Description
When I started my first idea was to compute a likelihood that would provide a probability of a single event upset happening based on orbital parameters and space weather forecasts. Such a likelihood model would have been a poisson distribution that included shape effects from the location data from CASSIOPE and space weather variables. The expected SEU rate would have been estimated as the product of cross section, particle flux, normalization factor integrated over energy.
However, the CSA noted that this was not a random data-set, which makes developing a probabilistic model more difficult. Based on the challenge description it seemed to imply that a machine learning model was preferred. This was exciting because it gave me a chance to try ML for the first time.
Data included the F10.7 data, K-planetary index, and DST index which are proxies for space weather activity.
Following a ML tutorial in python I was able to get a ML program to work with a 98% rate of successfully classifying SEU vs non-SEU events! This seemed really good at first glance, however, due to the disproportionate ratio of non-SEU events to SEU events (1249:22), this 98% success rate also had a 100% false negative rate for SEU events. Essentially, my algorithms would classify everything as a non-SEU. In fact, for the test set it returned all 'Non-SEU' : ['Non' 'Non' 'Non' 'Non' 'Non' 'Non' 'Non' 'Non'],. While, this means that my program won't solve the problem, in the course of this weekend I was able to identify parameters that are markers for higher SEU risk.
In the last minute when the machine learning algorithm kept giving false negative rate, I wanted to provide some sort of numerical value indicating the likelihood of an event in the test set being an SEU. This was done by simply computing the probability density functions of all the orbital and space weather parameters and taking their product. By doing this the probability of the events in the test set being an SEU are: ['5.6', '13.5', '1.2', '4.5', '5.1', '7.8', '0.0', '0.0'] %. While this is misleading because the dataset that the probabilities are based off was not a random set, it provides a cheap estimate.
Future ideas
- Spend more time on the write-up/presentation. Don't run out of time!!
- From a numerical probability standpoint. The simplest way to determine the probability of an SEU would be with a binomial, or poisson distribution where the SEU rate is based off the past data. In the case of CASSIOPE this rate would be 22/1249. This would be agnostic of any orbital and space weather variables.
Space Agency Data
The data was provided by the CSA on their challenge website. The data included GPS time and location for SEUs, non-SEUs and a test set. The data is all on the github repository.
Hackathon Journey
This was an experience marked by Firsts: first hackathon, first time trying machine learning, first time using a number of python packages.
This challenge was one of the suggestions from the Canadian Space Agency.
Despite not being able to fully develop an algorithm to assign risk probabilities, I am still proud of creating a framework that can be refined to solve the problem. Given the alloted time and the fact that such a problem could easily take years to study and solve (see theses in references), I am amazed with how much I could accomplish in just 2 days! It has been a great learning experience and it has inspired me to sign up for online machine learning courses, so that I will have the tools to better solve these types of problems in the future.
I hope to be able to try an in person hackathon eventually!
References
Tools that were used:
https://machinelearningmastery.com/machine-learning-in-python-step-by-step/
https://www.asc-csa.gc.ca/eng/events/2021/space-apps-challenge-2021.asp#signup
https://towardsdatascience.com/classifying-rare-events-using-five-machine-learning-techniques-fab464573233
https://en.wikipedia.org/wiki/Kernel_density_estimation
There were a number of papers/thesis that have considered aspects similar to this project:
https://arc.aiaa.org/doi/pdf/10.2514/6.2014-1722
https://repository.tudelft.nl/islandora/object/uuid%3Ad5655997-8c6d-4149-8464-4adf9d492221
Tags
#SEU, #CSA, #single event upset, #space weather
Global Judging
This project has been submitted for consideration during the Judging process.