The Cyprus Garbage Project
In spring 2023, my MSc thesis explored "The identification of garbage dumps in the rural areas of Cyprus through the application of deep learning to satellite imagery". The results earned me a "distinction" and kind remarks from the chief assessor.
This is a very original project idea that utilises novel training data in your machine learning. Your execution is strong and you have the basis for future work in a larger study if you wish. Overall, your ideas are well thought out and considered at each stage, which produces a very focused piece of research. It was a great read. Well Done!
Tony Knowles, University of York
During my research, I realized the need for a system that could:
- Collect and organize image data
- Augment labeled images
- Run various deep learning algorithms for comparison
- Create test and prediction runs using resulting models
From this need, I developed deepTerra™. Originally designed for garbage dump identification, deepTerra™ simplifies the image classification lifecycle: collect, augment, train, test, predict.
Garbage dump recognition is an interesting machine learning topic, and presents various challenges, including:
- High dimensionality of data
- Utilizing high resolution satellite image sources
- Irregular, difficult-to-categorize garbage targets
- Obstructions by trees and vegetation
- Rapidly evolving heat characteristics
- Variation according to historical snapshots
deepTerra™ provides practical support and solutions in addressing issues such as those above. Using the system several datasets of verified labeled data have been produced (some of which are available on the resources part of the website). An abridged version of my MSc thesis has been made available on ResearchGate and arXiv. The full version can be found here.
The following illustrates what an image patch collected typically looks like:
Three images containing garbage
The accuracy of the models when applied to test datasets varies with the actual training and test datasets, but the following stats are fairly typical(trained on a set of 340 images using the ResNet50 model): :
Confusion Matrix
| Predicted Positive | Predicted Negative | |
|---|---|---|
| Actual Positive | True Positive: 163 | False Negative: 14 |
| Actual Negative | False Positive: 27 | True Negative: 136 |
Machine Learning Statistics
| Accuracy | 0.88 |
|---|---|
| Precision | 0.86 |
| Recall | 0.92 |
| F1 Score | 0.89 |
The models developed have been applied to several 1km square areas in the Pitsilia region of Cyprus, using deepTerra, the results of which are shown on this zoomable google map overlay. This was automatically generated by the tool and shows the results of applying the model to seven areas, each of the grey overlays corresponding to a single region. For the purposes of machine learning each of these 1km square areas was divided by the tool into approximately 25x25 metre patches, giving 1296 patches per square kilometre. Within the map numerous red squares can be seen. Each of these red patches represents one for which a significant quantity of garbage was detected by the model. The map can be zoomed in to examine more closely the location of each area containing garbage. Clicking on a red patch brings up a small box with more details about that patch. The seven areas mapped comprise an analysis of just over 9000 separate image patches, of which 214 were detected to contain significant garbage (correspoding to the 214 red squares).
As training areas are expanded, models will improve. Once analysis and tuning based on the current sets of data (images) has been completed, then it will be possible to start applying the tool to automatically analyse larger areas. The next challenge is to do 10km by 10km areas (i.e. 100 km squared), which would represent an analysis of 129,600 image patches. For context, the entire island covers approximately 9250 square kilometres, which maps onto a minimum of 12 million image patches.
There are a number of technical issues that will prove interesting as points of study, including:
I aspire to pursue a research doctorate in these areas. The unique challenge of Cyprus garbage detection lies in smaller, ad hoc dumps when compared to previous studies from other areas in the world, which have tended to focus on relatively large scale land-fill sites.
The Private Pools Project
The aim of this case study was to:- verify the application on a dataset different to garbage dumps
- evaluate a different machine learning algorithm to that used on the garbage project
- see how quickly a small but useful image recognition could project could be conducted from start to end
- see how image augmentation can improve the quality of the model
The first aims was satisfied by making the focus of the study the identification of private swimming pools, ie typically small pools roughly 8x4m in size.
The Cyprus garbage project used the ResNet50 maching learning architecture. For the private pools I the VGG16 model was used, which is simpler than ResNet50, but which would might well on swimming pools, which are more regularly shaped than garbage dumps.
The approach taken was to choose a promising area of Cyprus which has a reasonable population of swimming pools (in this case the city of Larnaca) and to use the tool to (1) collect and label image patches containing and not containing pools, (2) train and test a model using the dataset. (3) the training dataset was then expanded in size using the image augmentation facility and the model retrained and tested, with a comparision made with the non-augmented model.
The whole process took just 3 hours and resulted in a model that had reasonable predictive ability despite the small amount of time devoted to it and the limited dataset collected. This illustrates the value of the tool as a means of "rapid prototyping" such machine learning tasks.
The results for the non-augmented dataset of pools consisting of 106 training samples applied to the test dataset of 78 test samples was as follows:
Confusion Matrix
| Predicted Positive | Predicted Negative | |
|---|---|---|
| Actual Positive | True Positive: 44 | False Negative: 14 |
| Actual Negative | False Positive: 3 | True Negative: 17 |
Machine Learning Statistics
| Accuracy | 0.78 |
|---|---|
| Precision | 0.76 |
| Recall | 0.94 |
| F1 Score | 0.84 |
Confusion Matrix
| Predicted Positive | Predicted Negative | |
|---|---|---|
| Actual Positive | True Positive: 42 | False Negative: 5 |
| Actual Negative | False Positive: 10 | True Negative: 21 |
Machine Learning Statistics
| Accuracy | 0.81 |
|---|---|
| Precision | 0.81 |
| Recall | 0.89 |
| F1 Score | 0.85 |
This shows a modest improvement in predictive accuracy of the model and supports the value of image augmentation for small datasets.
Cats vs Dogs
deepTerra was primarily designed to support land based classification projects using satellite imagery. However, it works equally well as a general image classifier. To illustrate this a labeled cats/dogs image dataset consisting of 25,000 images was used to train a model to recognize these categories. After training, it was run against a test dataset of 5000 images. As the following results show, the model was able to correctly identfy pictures of dogs or cats with an impressive 97% of reliability.Confusion Matrix
| Predicted Positive | Predicted Negative | |
|---|---|---|
| Actual Positive | True Positive: 2505 | False Negative: 52 |
| Actual Negative | False Positive: 73 | True Negative: 2370 |
Machine Learning Statistics
| Accuracy | 0.97 |
|---|---|
| Precision | 0.97 |
| Recall | 0.98 |
| F1 Score | 0.98 |
In addition to the above, deepTerra has been applied to identifying vehicles and buildings, with similar positive results. The details, together with dataset links, will be posted in due course. One intersting application that I'm investigating is its use in classifying rock climbing sectors. Rock climbing is something dear to my heart, and using the characteristic land terrain markers visible to train a model which could be used to discover currently unknown climbing locations feels like an interesting, novel, and challenging machine learning application.
Haemotological
deepTerra was applied to the task of distinguishing cancerous blood cells from healthy ones using blood smear imagery. Two datasets were used:
- A collection of 10,000 AML-positive images sourced from the Cancer Imaging Archive AML dataset , and
- A subset of the Blood Cell Count and Detection (BCCD) Dataset , comprising several hundred normal blood smear images that include neutrophils, lymphocytes, monocytes, eosinophils, erythrocytes, and platelets.
The following images show a few samples from each of the two datasets.
Three images representative of healthy blood cells
Three images representative of abnormal (AML) blood cells
Several convolutional neural network (CNN) models were trained on a relatively small dataset comprising 100 normal and 100 AML blood smear images. These models were then evaluated on a separate test set of 720 images. Despite the limited size of the training data, each model achieved strong performance. The metrics reported below, obtained using a VGG19 architecture, are representative of the overall results.
Confusion Matrix
| Predicted Positive | Predicted Negative | |
|---|---|---|
| Actual Positive | True Positive: 362 | False Negative: 2 |
| Actual Negative | False Positive: 1 | True Negative: 363 |
Machine Learning Statistics (to 2 dp)
| Accuracy | 1.00 |
|---|---|
| Precision | 1.00 |
| Recall | 0.98 |
| F1 Score | 0.99 |
| MCC | 0.99 |
Sourcing and preparing the data for this study, as well as training and evaluating the models, took less than two hours—most of which was spent locating suitable datasets. This speed highlights the efficiency and practical utility of the deepTerra toolset.
Work is currently underway to extend deepTerra into the domain of semantic segmentation, with a focus on classifying individual cells within blood smear samples.
Cyprus Quarries
A study has been rapid prototyped into the proliferation of quarries in Cyprus. The method used was to use local knowledge and expert judgement (plus the deepTerra toolset) to build a training dataset of around 200 quarry patches and 300 not quarry patches. A test dataset of 115 quarry patches, and 125 not quarry patches was produced seperately for validation purposes. A ResNet50 model was trainined and tuned, with testing indicating an accuracy above 90%. The deepTerra satellite image data collection utility was then used to download high resolution imagery for more than 90% of the island. These were subdivided into more than 185,000 patches, each covering around 220x220m.
The following images show a few samples of quarry image patches, used for training.
Three images representative quarries
When the model was run against this prediction dataset, some 3000+ images patches showed potential quarry locations. Closer examination identified quite a few false positives that could be picked out as isolated single patch matches. An additional feature was built into the tool to give the option of only accepting a patch as "positive" if it had neighbouring patches that were also positive. This reduced the number of probable quarry patches, to around 2000. It should be noted that in most cases a single quarry is encompassed by several patches. Methods are being investigated into how to group patches into a single quarry. The overall results were formulated in csv and json format, and an interactive google maps overlay constructed that shows the determined quarry locations.The following image shows an example from the google maps overlay.
The full map can be viewed at this location. The datasets used to train and test the model are available on the resources page.