AstroUnify

https://docs.google.com/presentation/d/16rLebenJl89-1-joij8YVRA132BqnrEJtsnQyo869S0/edit?usp=sharing

https://github.com/danieltyukov/marine-debris-ml-model

Project Summary:

Floating marine debris is a global pollution problem which threatens marine and human life and leads to the loss of biodiversity. Large swaths of marine debris are also navigational hazards to vessels. Artificial intelligence, specifically deep learning, can be used to detect floating marine debris in satellite imagery. In this project, we seek to demonstrate the strong potential of using commercial small satellite imagery for detecting marine debris pollution and strengthening current and future efforts to clean the oceans. We present an application of a deep learning model designed for object detection in the TensorFlow framework for observing marine debris floating on the surface of the ocean.

The model was trained on our custom-labeled dataset of 1370 polygons containing marine debris as observed in Planetscope optical imagery. An overall precision score of 0.78 and recall score of 0.70 were obtained on the test dataset.

Modal Performance on Test Images:

Paper and Dataset Forthcoming.

Overview:

Data:

Planet small satellite imagery is utilized in this project. Specifically, the 3-meter imagery product called Planetscope. This imagery has four bands namely red, green, blue, and near infrared. The combination of high spatial resolution, high temporal resolution, availability of a near-infrared channel and global coverage of coastlines made this imagery quite advantageous for the purposes of this project. With these imagery specifications as well as plastic size and ghost fishing net size categories, we anticipated our model would be capable of detecting aggregated debris flotsam as well as some mega plastics including medium to large size ghost fishing nets.

Using the Planet Explorer, specific image scenes consisting of visible marine debris patches were selected for our training dataset. This step involved manually exploring Planetscope scenes and verifying the presence of marine debris. For this initially, we decided to focus our efforts on detecting marine debris from optical (red, green, blue) channel imagery. Initial investigation into the utility of the Planetscope near-infrared channel was conducted, and future work will integrate the near-infrared channel.

We used Nasa Image Labeler to manually digitize bounding box annotations for observable debris on Planetscope optical imagery. A total of 1370 bounding boxes were labeled on the image scenes. This constituted the initial training, testing and validation dataset for object detection modeling.

The next task was to prepare the dataset in model-ready format, which entailed tiling the image scenes into smaller frames and encoding the bounding boxes into coordinate arrays with numerical class ids. The need for tiling the imagery stems from computational efficiency at model runtime. To accomplish these tasks, we used Label Maker (LM). We used zoom level 16 as it most closely approximates the native spatial resolution of Planetscope imagery. An example configuration file for use with LM is located at data_utils/config.json. Finally, the dataset in compressed array format (.npz) was used to create binary TensorFlow Records datasets.

Tiled image with labels.npz entry. On the right are the bounding box annotation coordinates [xmin, ymin, xmax, ymax] and class ID 1, with the image array on the bottom:

Tiled Images with Plotted Annotations:

Model:

Our architecture of choice for this project is SSD Resnet 101 Feature Pyramid Network (FPN), which we've implemented with the Tensorflow Object Detection API. We employed a weighted sigmoid focal loss and transfer learning for our baseline model from a pre-trained resnet 101 checkpoint hosted on TensorFlow model zoo. Our best model currently performs with a test F1 score of 0.74.

After training is complete, we export the best model to TensorFlow serving format, package the trained model weights and inference code into a Docker image and deploy at scale through our inference pipeline (shown below).

For inference, we use the Planet tile endpoint to request a list of XYZ tiles for a given area of interest and time range. We send that list of tiles via SQS to our inference endpoint, and once deployed, we can inference at a rate of 3000 tiles of size 256x256 pixels per minute. The results written to the database include, for each XYZ tile, the original Planet image scene ID and XYZ tile name (containing the x coordinate, y coordinate and zoom level) and one or more bounding box coordinates, class values and confidence scores. We use the python utility, Mercantile, to translate the XYZ coordinates to latitude and longitude coordinates and finally, export the final predictions with a minimum confidence threshold to GeoJSON format. The GeoJSON files are used for display in an online dashboard.

Scaled Model Inference Pipeline:

Visualizing The Model:

After you run the script, it will read all your test images in path/to/test/image/tiles folder and output the final prediction into the same folder. You will find new images in test_image_path with the _test suffixed to the end of the file basenames. The images with the predicted bounding boxes and confidence scores plotted on top. As well, you will find a multipolygon geojson of predicted bounding boxes in the test_image_path.

Detections Geo-Registered and Vectorized to GeoJSON Format:

The model was trained on our custom-labeled dataset of 1370 polygons containing marine debris as observed in Planetscope optical imagery. An overall precision score of 0.78 and recall score of 0.70 were obtained on the test dataset.

We used Nasa Image Labeler to manually digitize bounding box annotations for observable debris on Planetscope optical imagery. A total of 1370 bounding boxes were labeled on the image scenes. This constituted the initial training, testing and validation dataset for object detection modeling.

The next task was to prepare the dataset in model-ready format, which entailed tiling the image scenes into smaller frames and encoding the bounding boxes into coordinate arrays with numerical class ids. The need for tiling the imagery stems from computational efficiency at model runtime. To accomplish these tasks, we used Label Maker (LM).

Our architecture of choice for this project is SSD Resnet 101 Feature Pyramid Network (FPN), which we've implemented with the Tensorflow Object Detection API. We employed a weighted sigmoid focal loss and transfer learning for our baseline model from a pre-trained resnet 101 checkpoint hosted on Tensorflow model zoo. Our best model currently performs with a test F1 score of 0.74.

For inference, we use the Planet tile endpoint to request a list of XYZ tiles for a given area of interest and time range. We send that list of tiles via SQS to our inference endpoint, and once deployed, we can inference at a rate of 3000 tiles of size 256x256 pixels per minute.

Marine Debris is a problem that is close to home for all of us on the AstroUnify team and has been the pivot of discussion and development between us for quite a while. With the opportunity given by the community who organized Space Apps and the local hosts, we as high school students finally found the medium where we could share this incredibly urgent issue with others and propose our step towards battling it, in order to leap towards saving our oceanic biodiversity and avoiding hazardous events caused by this problem that we as humans, have escalated to be a concern of global matter.

AstroUnify is based in Cyprus, a small rocky island in the Mediterranean surrounded by the marine blue blanket covering the scenery from anywhere you look. And what a shame it is to look on a bright Sunday morning during a scorching summer day when going to the beach and swimming around the coast, where plastic bags, and other trash-related material spans across the whole area.

This inspired us to start looking for publicized tools where we could see such trash piles from above, to avoid them. But when we realized that the whole technological field in analyzing marine debris from satellite images is at the very primitive stage with only some papers being published on how it could be done, we as a team have decided to begin building.

Fast forward to the space apps hackathon, with a tool still in development and no information written to describe it to the end-user, we have taken this challenge to stand for our small island in the battle against marine debris. By using existing satellite imagery, we have built a Machine Learning model.

The Space Apps Hackathon affected us in a way that made us realize that this problem that we faced personally, is an issue faced globally and encouraged us to work even harder as a team and research on how to take our current progress, complete the bigger picture, finish the first version, and study a way to summarize our progress to others in an understandable way.

As a team, we have faced setbacks where we had trouble organizing ourselves at first to complete this project in time. But even though we had this initial struggle, we managed to resolve it, and progress exponentially to complete our project and present it to the local and global judges.

And at the end of the day, we could not have done it without the other open-source contributors, who we thank sincerely. We at AstroUnify stand strongly for the meaning of open-source and believe that it is what drives innovation. Where people no matter where from or from what background can contribute immensely to make the world a better place without the main goal being to gain profit.

Nasa Earth Data: https://earthdata.nasa.gov

EPA: https://www.epa.gov/trash-free-waters

DOI: https://www.doi.gov/ocl/marine-debris

Planet Scope Imagery: https://www.planet.com/products/planet-imagery

Nasa Image Labeler: https://impact.earthdata.nasa.gov/labeler

Label Maker: https://github.com/developmentseed/label-maker

FPN: https://arxiv.org/abs/1708.02002

TensorFlow: https://www.tensorflow.org

Docker: https://www.docker.com

Planet Dev Tools: https://developers.planet.com

AWS Tools: https://aws.amazon.com

Coordinate Utilities: https://github.com/mapbox/mercantile

#marine_debris, #software, #marinedebris, #machine_learning, #artifical_intelligence, #ml, #ai, #geospatial_technology, #satellite_images, #python, #tensorflow, #data, #deep_learning, #nasa

This project has been submitted for consideration during the Judging process.