tutorials/workshops/philab24/05_stac.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Curation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we have seen before, datasets and models in EOTDL are categorized into different quality levels:\n",
    "\n",
    "- **Q0**: datasets in the form of an archive with arbitary files without curation. This level is ideal for easy and fast upload/download of small datasets.\n",
    "- **Q1**: datasets with STAC metadata but no QA. These datasets can leverage a limited set of EOTDL features.\n",
    "- **Q2**: datasets with STAC metadata with the EOTDL custom extensions and automated QA. These datasets can leverage the full potential of the EOTDL.\n",
    "- **Q3**: Q2 datasets that are manually curated. These datasets are the most reliable and can be used as benchmark datasets.\n",
    "\n",
    "Up until now we have been focused on Q0 datasets and models, but in this notebook we are going to explore higher quality level datasets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## STAC metadata"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The [STAC](https://stacspec.org/en) specification is a common language to describe geospatial information, so it can more easily be worked with, indexed, and discovered. Is a standarized way to expose, host, ingest and access geospatial collections that has been adopted as the EOTDL standard metadata format. For further information, check their [website](https://stacspec.org/en).\n",
    "\n",
    "In order to facilitate the STAC generation, which can be painful and time-consuming, the EOTDL offers some tools tto streamline this process. Let's dive into them!\n",
    "\n",
    "The first thing we have to understand is that the process starts with a `STACDataFrame`. This `STACDataFrame` is an interface between the datasets and STAC (catalogs, collections and items), with some variables that we can define and customise to ensure that the STAC metadata has the information we want, like `extensions`, which defines the [STAC extensions](https://stac-extensions.github.io/) that the image must have, or `bands`, with the bands we want to get from the image, if any.\n",
    "\n",
    "Let's see the example below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>image</th>\n",
       "      <th>label</th>\n",
       "      <th>ix</th>\n",
       "      <th>collection</th>\n",
       "      <th>extensions</th>\n",
       "      <th>bands</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>data/sentinel_2/Boadella_2.tif</td>\n",
       "      <td>Boadella</td>\n",
       "      <td>0</td>\n",
       "      <td>data/sentinel_2/source</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>data/sentinel_2/Boadella_3.tif</td>\n",
       "      <td>Boadella</td>\n",
       "      <td>0</td>\n",
       "      <td>data/sentinel_2/source</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>data/sentinel_2/Boadella_1.tif</td>\n",
       "      <td>Boadella</td>\n",
       "      <td>0</td>\n",
       "      <td>data/sentinel_2/source</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>data/sentinel_2/Boadella_4.tif</td>\n",
       "      <td>Boadella</td>\n",
       "      <td>0</td>\n",
       "      <td>data/sentinel_2/source</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>data/sentinel_2/Boadella_5.tif</td>\n",
       "      <td>Boadella</td>\n",
       "      <td>0</td>\n",
       "      <td>data/sentinel_2/source</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                            image     label  ix              collection  \\\n",
       "0  data/sentinel_2/Boadella_2.tif  Boadella   0  data/sentinel_2/source   \n",
       "1  data/sentinel_2/Boadella_3.tif  Boadella   0  data/sentinel_2/source   \n",
       "2  data/sentinel_2/Boadella_1.tif  Boadella   0  data/sentinel_2/source   \n",
       "3  data/sentinel_2/Boadella_4.tif  Boadella   0  data/sentinel_2/source   \n",
       "4  data/sentinel_2/Boadella_5.tif  Boadella   0  data/sentinel_2/source   \n",
       "\n",
       "   extensions  bands  \n",
       "0         NaN    NaN  \n",
       "1         NaN    NaN  \n",
       "2         NaN    NaN  \n",
       "3         NaN    NaN  \n",
       "4         NaN    NaN  "
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "sample_df = pd.read_csv('workshop_data/sample_stacdataframe.csv')\n",
    "sample_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a sample `STACDataFrame` already generated for our workshop. Here we can see a lot of information:\n",
    "- image: the path to every image.\n",
    "- label: the label assigned to every image. \n",
    "- ix: the index of the label.\n",
    "- collection: the collection which the image belongs to. \n",
    "- extensions: a list with the STAC extensions we want the image STAC item to have.\n",
    "- bands: a list of band we want the image STAC item to have.\n",
    "\n",
    "Now we have seen this, let's generate the STAC metadata for our dataset. Don't worry, we are going to explain it step by step!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First of all, we need to import the `STACGenerator` class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from eotdl.curation.stac.stac import STACGenerator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `STACGenerator` class is the entry point and the STAC generation class, where magic happens. Before we declare it, we need to understand the parameters we can give to it:\n",
    "- `image_format`: the extension of the images. Could be `png`, `jpg` and so on. By default is `tiff`.\n",
    "- `catalog_type`: the STAC Catalog type. It is a specification defined [here](https://pystac.readthedocs.io/en/0.4/concepts.html#catalog-types). By default is `SELF_CONTAINED`. \n",
    "- `item_parser`: the item_parser defines the strategy that must be followed to search for satellite images within the folder. We have defined 2 item_parser strategies, and new ones can be added as needed. The strategies that are implemented right now are the following.\n",
    "    - `StructuredParser`: this strategy is used when the images are each contained within a folder, so that the name of the item will be the name of the folder.\n",
    "    \n",
    "    <p align=\"center\">\n",
    "        <img src=\"images/structured_parser.png\" alt=\"Structured parser typical folder structure\" style=\"height:170px; width:200px;\"/>\n",
    "    </p>\n",
    "    \n",
    "    - `UnestructuredParser`: this strategy is used when there are multiple images contained in the same folder. We will use this strategy when using the EOTDL to download the dataset images, as it will always format the folder structure the same way. As this is what we have done, it is the strategy that we will use for the use case of this workshop, as all the images are in the same folder.\n",
    "\n",
    "    <p align=\"center\">\n",
    "        <img src=\"images/unestructured_parser.png\" alt=\"Structured parser typical folder structure\" style=\"height:200px; width:200px;\"/>\n",
    "    </p>\n",
    "    \n",
    "- `assets_generator`: the assets_generator parameter defines the strategy to follow with the generation of assets from each image. In this way, it could be the case that from a Sentinel-2 image we want to extract all its bands as assets, or simply extract the RGB bands, or not extract any as assets. By default, three strategies have been established, which can be expanded according to needs.\n",
    "\n",
    "    - `STACAssetGenerator`: does not extract new assets from the image bands, so a single asset is generated for the image.\n",
    "    - `BandsAssetGenerator`: from the original image it creates a new file for each band established in the 'bands` column, deleting the original file. An asset is added to the STAC item for each band.\n",
    "    \n",
    "    \n",
    "\n",
    "- `labeling_strategy`: the `labeling_strategy` parameter defines the strategy to extract a label from the filename of an image, to assign a label to it. By default, we have implemented 2 strategies:\n",
    "\n",
    "    - `UnlabeledStrategy`: we will use it when the images do not have a label that identifies them or that has been placed on purpose. It is the one we will use in our case, since the filename is simply the name of the constellation, and it is the default option.\n",
    "    - `LabeledStrategy`: we will use it when the images are labeled with labels in their filenames. An example would be that in a folder the images were called, for example, River_1.png, River_2.png, River_3.png, and so on. The file name must be the pattern <label>_<number>. This is the option we are going to use in the workshop, as all the images filenames are `boadella_<id>`.\n",
    "\n",
    "    > If you plan to use SCANEO to label a Q1+ dataset, we suggest you first convert the dataset to Q1+ (generating STAC metadata) and then label the dataset with SCANEO. This way, SCANEO will work on STAC mode and the process will be simpler. However, it is still possilbe to upgrade a Q0 dataset with labels as we will later in this section.\n",
    "\n",
    "\n",
    "For the specific case of our workshop, we will take into account the following:\n",
    "- The images have been downloaded using the EOTDL and are each in the same folder, so as `item_parser` we will use `UnestructuredParser`.\n",
    "- In this case we do not want to extract bands from the image as assets, so as `asset_generator` we will use `STACAssetGenerator`, which is the default option.\n",
    "- As said, although the images do not have labels in their filenames we can use the `LabeledStrategy`, as all the images filenames are `boadella_<id>`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from eotdl.curation.stac.stac import STACGenerator\n",
    "from eotdl.curation.stac.assets import STACAssetGenerator\n",
    "from eotdl.curation.stac.parsers import UnestructuredParser\n",
    "from eotdl.curation.stac.dataframe_labeling import LabeledStrategy\n",
    "\n",
    "stac_generator = STACGenerator(item_parser=UnestructuredParser, \n",
    "                               assets_generator=STACAssetGenerator, \n",
    "                               labeling_strategy=LabeledStrategy,\n",
    "                               image_format='tif'\n",
    "                               )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we now decide to generate a `STACDataFrame` from the folder with the images it will do fine, but it will be incomplete."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>image</th>\n",
       "      <th>label</th>\n",
       "      <th>ix</th>\n",
       "      <th>collection</th>\n",
       "      <th>extensions</th>\n",
       "      <th>bands</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>data/sentinel_2/Boadella_2021-07-16.tif</td>\n",
       "      <td>Boadella</td>\n",
       "      <td>0</td>\n",
       "      <td>data/sentinel_2/source</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>data/sentinel_2/Boadella_2020-01-18.tif</td>\n",
       "      <td>Boadella</td>\n",
       "      <td>0</td>\n",
       "      <td>data/sentinel_2/source</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>data/sentinel_2/Boadella_2020-09-14.tif</td>\n",
       "      <td>Boadella</td>\n",
       "      <td>0</td>\n",
       "      <td>data/sentinel_2/source</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>data/sentinel_2/Boadella_2022-03-08.tif</td>\n",
       "      <td>Boadella</td>\n",
       "      <td>0</td>\n",
       "      <td>data/sentinel_2/source</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>data/sentinel_2/Boadella_2020-01-28.tif</td>\n",
       "      <td>Boadella</td>\n",
       "      <td>0</td>\n",
       "      <td>data/sentinel_2/source</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     image     label  ix  \\\n",
       "0  data/sentinel_2/Boadella_2021-07-16.tif  Boadella   0   \n",
       "1  data/sentinel_2/Boadella_2020-01-18.tif  Boadella   0   \n",
       "2  data/sentinel_2/Boadella_2020-09-14.tif  Boadella   0   \n",
       "3  data/sentinel_2/Boadella_2022-03-08.tif  Boadella   0   \n",
       "4  data/sentinel_2/Boadella_2020-01-28.tif  Boadella   0   \n",
       "\n",
       "               collection extensions bands  \n",
       "0  data/sentinel_2/source       None  None  \n",
       "1  data/sentinel_2/source       None  None  \n",
       "2  data/sentinel_2/source       None  None  \n",
       "3  data/sentinel_2/source       None  None  \n",
       "4  data/sentinel_2/source       None  None  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = stac_generator.get_stac_dataframe('data/sentinel_2')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A key feature is the `label` column. Using the label of every image we are going to assign parameters like the STAC extensions that this image's item is going to have, or the bands we want to extract using the `BandsAssetGenerator`. We can obtain the existing labels in the STACDataFrame before adding new information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Boadella']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "labels = df.label.unique().tolist()\n",
    "labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Starting from the `Boadella` label we are going to define the STAC extensions. As STAC extensions we are going to implement the [proj](https://github.com/stac-extensions/projection), [raster](https://github.com/stac-extensions/raster) and [eo](https://github.com/stac-extensions/eo) STAC extensions. \n",
    "\n",
    "> Note: the supported extensiones are `('eo', 'sar', 'proj', 'raster')`.\n",
    "\n",
    "On the other hand, although we don't want to extract the image bands, we can define them to see their metadata using the `eo` STAC extension. To simplify, let's only define the bands `B04`, `B03` and `B02`, which are the RGB bands.\n",
    "\n",
    "To define these parameters for each label, we simply have to declare a dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "extensions = {'Boadella': ('proj', 'raster', 'eo')}\n",
    "bands = {'Boadella': ('B02', 'B03', 'B04')}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Note: if we wanted to extract the bands as assets, we should create a dict following the same convention, just as follows. As said it is not required for this workshop, but could be useful for other use cases.\n",
    "\n",
    "```\n",
    "bands = {'sentinel-2-l2a': ('B01', 'B02', 'B03', 'B04', 'B05', 'B06', 'B07', 'B08', 'B09', 'B11', 'B12')}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we are ready to generate a `STACDataFrame` with relevant information. Some extra parameters to take into account:\n",
    "- `path`: is the root path where the images are located at. In our case is `data/sentinel_2`.\n",
    "- `collections`: we can use this parameter to define the STAC collection to which we want each item with a specific label to go. There are several options:\n",
    "    - The default option puts all the STAC items in a single collection called `source`.\n",
    "    - The `*` option will consider folders located directly under the root folder as collections, so it will create a collection for each of them.\n",
    "\n",
    "    <p align=\"center\">\n",
    "        <img src=\"images/collection.png\" alt=\"* collection\" style=\"height:170px; width:200px;\"/>\n",
    "    </p>\n",
    "\n",
    "    - You can decide the collection you want an image to go to through its label, as we have seen in the case of extensions and bands. To give an example, we are going to define it like this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "collection = {'Boadella': 'boadella-sentinel-2'}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's generate the complete STACDataFrame!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>image</th>\n",
       "      <th>label</th>\n",
       "      <th>ix</th>\n",
       "      <th>collection</th>\n",
       "      <th>extensions</th>\n",
       "      <th>bands</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>data/sentinel_2/Boadella_2021-07-16.tif</td>\n",
       "      <td>Boadella</td>\n",
       "      <td>0</td>\n",
       "      <td>data/sentinel_2/boadella-sentinel-2</td>\n",
       "      <td>(proj, raster, eo)</td>\n",
       "      <td>(B02, B03, B04)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>data/sentinel_2/Boadella_2020-01-18.tif</td>\n",
       "      <td>Boadella</td>\n",
       "      <td>0</td>\n",
       "      <td>data/sentinel_2/boadella-sentinel-2</td>\n",
       "      <td>(proj, raster, eo)</td>\n",
       "      <td>(B02, B03, B04)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>data/sentinel_2/Boadella_2020-09-14.tif</td>\n",
       "      <td>Boadella</td>\n",
       "      <td>0</td>\n",
       "      <td>data/sentinel_2/boadella-sentinel-2</td>\n",
       "      <td>(proj, raster, eo)</td>\n",
       "      <td>(B02, B03, B04)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>data/sentinel_2/Boadella_2022-03-08.tif</td>\n",
       "      <td>Boadella</td>\n",
       "      <td>0</td>\n",
       "      <td>data/sentinel_2/boadella-sentinel-2</td>\n",
       "      <td>(proj, raster, eo)</td>\n",
       "      <td>(B02, B03, B04)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>data/sentinel_2/Boadella_2020-01-28.tif</td>\n",
       "      <td>Boadella</td>\n",
       "      <td>0</td>\n",
       "      <td>data/sentinel_2/boadella-sentinel-2</td>\n",
       "      <td>(proj, raster, eo)</td>\n",
       "      <td>(B02, B03, B04)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     image     label  ix  \\\n",
       "0  data/sentinel_2/Boadella_2021-07-16.tif  Boadella   0   \n",
       "1  data/sentinel_2/Boadella_2020-01-18.tif  Boadella   0   \n",
       "2  data/sentinel_2/Boadella_2020-09-14.tif  Boadella   0   \n",
       "3  data/sentinel_2/Boadella_2022-03-08.tif  Boadella   0   \n",
       "4  data/sentinel_2/Boadella_2020-01-28.tif  Boadella   0   \n",
       "\n",
       "                            collection          extensions            bands  \n",
       "0  data/sentinel_2/boadella-sentinel-2  (proj, raster, eo)  (B02, B03, B04)  \n",
       "1  data/sentinel_2/boadella-sentinel-2  (proj, raster, eo)  (B02, B03, B04)  \n",
       "2  data/sentinel_2/boadella-sentinel-2  (proj, raster, eo)  (B02, B03, B04)  \n",
       "3  data/sentinel_2/boadella-sentinel-2  (proj, raster, eo)  (B02, B03, B04)  \n",
       "4  data/sentinel_2/boadella-sentinel-2  (proj, raster, eo)  (B02, B03, B04)  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = stac_generator.get_stac_dataframe('data/sentinel_2', collections=collection, extensions=extensions, bands=bands)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It looks good! We have all we need to generate the STAC metadata. We only have to give the catalog an `id`, a `description` and an `output_folder`!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating boadella-sentinel-2 collection...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 12/12 [00:00<00:00, 345.98it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validating and saving catalog...\n",
      "Success!\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "stac_generator.generate_stac_metadata(stac_id='boadella-dataset',\n",
    "                                      description='Boadella dataset',\n",
    "                                      output_folder='data/sentinel_2_stac')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Voilà! We have generated the STAC metadata from our Q0 dataset, converting it into a Q1 dataset! But we are not done yet, as we have to generate the STAC labels item of every image, using the GeoJSON files that we have generated when labeling our dataset using SCANEO.\n",
    "\n",
    "In order to generate the labels collection of a source collection (understanding 'source' collection as the source where are the STAC items belonging to the images) we have implementated a customizable class named `LabelExtensionObject`. With this class you can decide how to create the labels of your dataset, wether you want to develop your own implementation or use the implementations we have already developed. Let's explain them!\n",
    "\n",
    "- `ImageNameLabeler`: this implementation should be used when the images of the dataset are named with the corresponding labels, such as `River_1`, `Forest_1`, and so on. We are not going to use this implementation, as it's not our use case.\n",
    "- `ScaneoLabeler`: this implementation should be used when the labels have been generated using SCANEO, so we have a folder with the `geoJSON` label files and their corresponding images. As seen, this is the implementation we are going to use. Let's check the parameters we should use:\n",
    "    - `catalog`: the path to the STAC catalog, or the pystac Catalog itself, we want to add the labels collection. In our case, `data/sentinel_2_stac/catalog.json`.\n",
    "    - `root_folder`: the path to the folder containing the `geoJSON` files.\n",
    "    - `collection`: the STAC collection we want to add the labels to. By default is `source`, but in our case is `boadella-sentinel-2`.\n",
    "    - Extra properties can be added using `kwargs`, such as `label:methods` or `label:overviews`. You can check them [here](https://github.com/stac-extensions/label#item-properties). We are going to add `label_methods` as `manual`.\n",
    "\n",
    "Knowing this, we can generate our labels collection!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Generating labels collection...: 12it [00:00, 1660.12it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Success on labels generation!\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "from eotdl.curation.stac.extensions import ScaneoLabeler\n",
    "\n",
    "labeler = ScaneoLabeler()\n",
    "\n",
    "catalog = 'data/sentinel_2_stac/catalog.json'\n",
    "labels_extra_properties = {'label_methods': [\"manual\"]}\n",
    "labeler.generate_stac_labels(\n",
    "    catalog=catalog,\n",
    "    root_folder='data/sentinel_2',\n",
    "    collection='boadella-sentinel-2',\n",
    "    **labels_extra_properties\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the STAC metadata is generated and we have our Q1 dataset, we can ingest the dataset into EOTDL as seen before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading STAC catalog...\n",
      "New version created, version: 3\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 24/24 [00:05<00:00,  4.53it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ingesting STAC catalog...\n",
      "Done\n"
     ]
    }
   ],
   "source": [
    "from eotdl.datasets import ingest_dataset\n",
    "\n",
    "ingest_dataset('data/sentinel_2_stac')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Q2 Datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section we will learn about Q2 datsets, datasets with STAC metadata and EOTDL's custom STAC extensions. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The ML-Dataset Extension"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The main extension used by EOTDL for Q2 datasets is the ML-Dataset extension. It enhances the STAC metadata of a dataset including information such as data splits (train, validation, test), quality metrics, etc.\n",
    "\n",
    "Let's see how to generate a Q2 dataset for our data. Q2 datasets are generated from Q1 datasets, datasets with STAC metadata. We already showed how to generate a Q1 dataset in the previous section.\n",
    "\n",
    "The addition of the `ml-dataset` STAC extension to a STAC catalog is pretty straightforward, so it can be done with a simple function called `add_ml_extension`. Sounds easy, right? Let's see what we need.\n",
    "- `catalog`: the path to the STAC catalog, or the pystac Catalog itself, we want to add the extension. In our case, `data/sentinel_2_stac/catalog.json`.\n",
    "- `destination`: if we want we can define an output folder to save the catalog, but by default the function generates it in the same folder of the given catalog. In our case, `data/sentinel_2_q2`.\n",
    "- `splits`: we should put is as `True` if we want to split the labels. By default is `False`, and the default values for the splits are `Train`, `Test` and `Validation` in a `80, 10, 10` proportion. In our case is `True`, and we are fine with the default proportions.\n",
    "- `splits_collection_id`: the id of the collection we want to make the splits to. In our case, `labels`, which is the default option.\n",
    "- `name`: the name of the dataset. In our case, `Boadella Q2 Dataset`, but feel free to customize it at your own.\n",
    "- `tasks`: the tasks of the dataset. In our case, `[segmentation]`.\n",
    "- `inputs_type`: the type of the dataset inputs. In our case, `[satellite imagery]`.\n",
    "- `annotations_type`: the type of the annotations. In our case, `raster`.\n",
    "- `version`: the version of our dataset. In our case, `0.1.0`.\n",
    "\n",
    "Let's add the extension!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating splits...\n",
      "Total size: 12\n",
      "Train size: 9\n",
      "Test size: 1\n",
      "Validation size: 1\n",
      "Generating Training split...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 9/9 [00:00<00:00, 25592.36it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating Validation split...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1/1 [00:00<00:00, 10433.59it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating Test split...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1/1 [00:00<00:00, 15363.75it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Success on splits generation!\n",
      "Validating and saving...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Success!\n"
     ]
    }
   ],
   "source": [
    "from eotdl.curation.stac.extensions import add_ml_extension\n",
    "\n",
    "catalog = 'data/sentinel_2_stac/catalog.json'\n",
    "\n",
    "add_ml_extension(\n",
    "\tcatalog,\n",
    "\tdestination='data/sentinel_2_q2',\n",
    "\tsplits=True,\n",
    "\tsplits_collection_id=\"labels\",\n",
    "\tname='Boadella Q2 Dataset',\n",
    "\ttasks=['segmentation'],\n",
    "\tinputs_type=['satellite imagery'],\n",
    "\tannotations_type='raster',\n",
    "\tversion='0.1.0'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When ingesting a Q2 dataset, EOTDL will automatically compute quality metrics on your dataset, that will be reported in the metadata. Optionally, you can compute them to analyse your dataset before ingesting it with the `MLDatasetQualityMetrics` class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Looking for spatial duplicates...: 48it [00:00, 3981.93it/s]\n",
      "Calculating classes balance...: 48it [00:00, 156918.62it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validating and saving...\n",
      "Success!\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "from eotdl.curation.stac.extensions import MLDatasetQualityMetrics\n",
    "\n",
    "catalog = 'data/sentinel_2_q2/catalog.json'\n",
    "\n",
    "MLDatasetQualityMetrics.calculate(catalog)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember, however, that the metrics will be computed automatically when ingesting the dataset, so you don't need to do it yourself. These metrics incude aspects such as the number of samples, duplicates, missing values, class imbalance, etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, ingest our Q2 dataset!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading STAC catalog...\n",
      "New version created, version: 2\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 40/40 [00:13<00:00,  3.06it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ingesting STAC catalog...\n",
      "Done\n"
     ]
    }
   ],
   "source": [
    "from eotdl.datasets import ingest_dataset\n",
    "\n",
    "ingest_dataset('data/sentinel_2_q2')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An that's all! You have successfully created your very own dataset for AI4EO techniques! From downloading the images and labeling them to creating the STAC metadata and ingesting the catalog, we have covered the entire dataset creation process. We hope you have enjoyed and it have been profitable!\n",
    "\n",
    "Don't forget to check the [EOTDL](https://www.eotdl.com/) website for further news and improvements, and the [EarthPulse](https://earthpulse.ai/) website to stay updated about our adventures!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Discussion and Contribution opportunities"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Feel free to ask questions now (live or through Discord) and make suggestions for future improvements.\n",
    "\n",
    "- What features would you like to see in EOTDL concerning dataset curation?\n",
    "- What metadata is important for you when working with datasets?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "sen12floods",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}