Downloading Sentinel-2 remote sensing images and Analysis with Large language models: A tutorial

Every day, a large number of satellites are orbiting around Planet Earth with different distances from the surface. Many of these flying objects are used to get information about the area on which they are passing. A whole field called Geographic Information Systems (GIS) is dedicated to the study and analysis of this sensed information. It is worth noting that not only satellites are used to produce this kind of information but also unmanned aerial vehicles (drones) and planes.

This process of gathering information from above, regardless of the platform, falls under a specific scientific discipline and we learn from this resource that it is called remote sensing, it is done by leveraging instruments that have the ability to detect reflected or emitted energy. The latter energy is called Electromagnetic energy that has the form of waves, it is characterised by a spectrum that classifies it into different types based on the wavelength of  the wave. Some of the types include: radio, x-ray and what the human eye can see (visible light).

Among the diverse applications of remote sensing, this tutorial will specifically focus on one common and easily interpretable output: True Color Images of areas on Earth, which are the typical Red Green Blue images we are used to. These images are constructed by combining different bands together and each band is defined by the wavelength that it captures.

Numerous satellite missions are dedicated to collecting this type of remote sensing data, and for the purpose of this tutorial, we will utilize the data provided by the Sentinel-2 mission. Operated by the copernicus project This mission has free access to the remotely sensed data and has 13 bands with three main resolutions: 10m, 20m and 60m. 

Nowadays, researchers in approximately all the fields are studying how they can use Artificial intelligence to address their research objectives. The field of GIS is no exception and there is a significant body of research that uses different machine learning and deep learning techniques to process remote sensing data for different objectives. 

Within the rapidly evolving field of AI, Large Language Models have seen breakthroughs in the last few years and have become powerful tools, especially with integration of vision into these models instead of just text based models. These models can now understand and process visual information and follow instructions based on images’ content. This advancement motivated the focus of this tutorial: Analyse the remotely sensed images using Large Language Models.

Given the immense information these LLMs are trained on, they already have a nice understanding of the field of remote sensing and geographic information systems. That is why these models can understand and answer with high precision instructions like for example : “describe the land cover in this image.”

To assess this capability, in this tutorial we will see how we can use the different APIs provided by the Copernicus data space in order to: 

1. Search for available products (data sensed by Sentinel-2 satellites) based on geographical coordinates of the area of interest and the time we are interested in. 
2. Download the True Color Image of our area of with a specific resolution

After that, we will leverage the google generative ai library to do the following:

1. Generate analysis of the remote sensing image and making structured output using a dedicated prompt and the structured output technique.
2. Feeding the model with multiple images with their corresponding dates and prompting it to track and describe the changes that occur over time.

These steps will be detailed in the subsequent sections of this tutorial, which are organized as follows: first we will go through accessing Sentinel-2 data in section 1, demonstrate how to leverage Large Language Models for analysis in section 2 and finally a walk through how can this be extended to a web app in section 3.