tutorials/notebooks/23_stac_item_parsers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# STAC items parsers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we want to generate STAC metadata from a imagery dataset through EOTDL, we must generate a STACDataFrame, as seen in this [notebook](20_stac.ipynb). When generating the STACDataFrame, there is a needed parameter called `item_parser`. In this notebook we are going to dive in it.\n",
    "\n",
    "Uncomment the following line to install eotdl if needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install eotdl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `item_parser` defines the strategy that must be followed to search for satellite images within the folder and **create an ID for every STAC item**. We have defined 2 item_parser strategies, and new ones can be added as needed. The strategies that are implemented right now are the following.\n",
    "- `StructuredParser`: this strategy is used when the images are each contained within a folder, so that the name of the item will be the name of the folder.\n",
    "\n",
    "<p align=\"center\">\n",
    "    <img src=\"assets/structured_parser.png\" alt=\"Structured parser typical folder structure\" style=\"height:170px; width:200px;\"/>\n",
    "</p>\n",
    "\n",
    "- `UnestructuredParser`: this strategy is used when there are multiple images contained in the same folder. We will use this strategy when using the EOTDL to download the dataset images, as it will always format the folder structure the same way. As this is what we have done, it is the strategy that we will use for the use case of this workshop, as all the images are in the same folder.\n",
    "\n",
    "<p align=\"center\">\n",
    "    <img src=\"assets/unestructured_parser.png\" alt=\"Structured parser typical folder structure\" style=\"height:200px; width:200px;\"/>\n",
    "</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Knowing this, we can take a look over them. Let's try firs the `UnestructuredParser`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from eotdl.curation.stac.stac import STACGenerator\n",
    "from eotdl.curation.stac.assets import STACAssetGenerator\n",
    "from eotdl.curation.stac.parsers import UnestructuredParser, StructuredParser\n",
    "from eotdl.curation.stac.dataframe_labeling import LabeledStrategy\n",
    "\n",
    "stac_generator = STACGenerator(item_parser=UnestructuredParser, \n",
    "                               assets_generator=STACAssetGenerator, \n",
    "                               labeling_strategy=LabeledStrategy,\n",
    "                               image_format='tif'\n",
    "                               )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>image</th>\n",
       "      <th>label</th>\n",
       "      <th>ix</th>\n",
       "      <th>collection</th>\n",
       "      <th>extensions</th>\n",
       "      <th>bands</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>example_data/jaca_dataset/Jaca_1.tif</td>\n",
       "      <td>Jaca</td>\n",
       "      <td>0</td>\n",
       "      <td>example_data/jaca_dataset/source</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>example_data/jaca_dataset/Jaca_2.tif</td>\n",
       "      <td>Jaca</td>\n",
       "      <td>0</td>\n",
       "      <td>example_data/jaca_dataset/source</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>example_data/jaca_dataset/Jaca_3.tif</td>\n",
       "      <td>Jaca</td>\n",
       "      <td>0</td>\n",
       "      <td>example_data/jaca_dataset/source</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>example_data/jaca_dataset/Jaca_4.tif</td>\n",
       "      <td>Jaca</td>\n",
       "      <td>0</td>\n",
       "      <td>example_data/jaca_dataset/source</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                  image label  ix  \\\n",
       "0  example_data/jaca_dataset/Jaca_1.tif  Jaca   0   \n",
       "1  example_data/jaca_dataset/Jaca_2.tif  Jaca   0   \n",
       "2  example_data/jaca_dataset/Jaca_3.tif  Jaca   0   \n",
       "3  example_data/jaca_dataset/Jaca_4.tif  Jaca   0   \n",
       "\n",
       "                         collection extensions bands  \n",
       "0  example_data/jaca_dataset/source       None  None  \n",
       "1  example_data/jaca_dataset/source       None  None  \n",
       "2  example_data/jaca_dataset/source       None  None  \n",
       "3  example_data/jaca_dataset/source       None  None  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = stac_generator.get_stac_dataframe('example_data/jaca_dataset')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now, let's take a look on the `StructuredParser`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "stac_generator = STACGenerator(item_parser=StructuredParser, \n",
    "                               assets_generator=STACAssetGenerator, \n",
    "                               labeling_strategy=LabeledStrategy,\n",
    "                               image_format='tif'\n",
    "                               )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>image</th>\n",
       "      <th>label</th>\n",
       "      <th>ix</th>\n",
       "      <th>collection</th>\n",
       "      <th>extensions</th>\n",
       "      <th>bands</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>example_data/jaca_dataset_structured/Jaca_3/Ja...</td>\n",
       "      <td>Jaca</td>\n",
       "      <td>0</td>\n",
       "      <td>example_data/jaca_dataset_structured/source</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>example_data/jaca_dataset_structured/Jaca_4/Ja...</td>\n",
       "      <td>Jaca</td>\n",
       "      <td>0</td>\n",
       "      <td>example_data/jaca_dataset_structured/source</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>example_data/jaca_dataset_structured/Jaca_2/Ja...</td>\n",
       "      <td>Jaca</td>\n",
       "      <td>0</td>\n",
       "      <td>example_data/jaca_dataset_structured/source</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>example_data/jaca_dataset_structured/Jaca_1/Ja...</td>\n",
       "      <td>Jaca</td>\n",
       "      <td>0</td>\n",
       "      <td>example_data/jaca_dataset_structured/source</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               image label  ix  \\\n",
       "0  example_data/jaca_dataset_structured/Jaca_3/Ja...  Jaca   0   \n",
       "1  example_data/jaca_dataset_structured/Jaca_4/Ja...  Jaca   0   \n",
       "2  example_data/jaca_dataset_structured/Jaca_2/Ja...  Jaca   0   \n",
       "3  example_data/jaca_dataset_structured/Jaca_1/Ja...  Jaca   0   \n",
       "\n",
       "                                    collection extensions bands  \n",
       "0  example_data/jaca_dataset_structured/source       None  None  \n",
       "1  example_data/jaca_dataset_structured/source       None  None  \n",
       "2  example_data/jaca_dataset_structured/source       None  None  \n",
       "3  example_data/jaca_dataset_structured/source       None  None  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = stac_generator.get_stac_dataframe('example_data/jaca_dataset_structured')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As seen before, the main use of `item_parser` is to extract the ID of the future item from STAC. This way, `UnstructuredParser` will extract it from the file name, while `StructuredParser` will extract it from the name of the containing folder. This is useful depending on how we have the data structured. For example, the `SEN12-FLOODS` dataset is structured, so we could use `StructuredParser`, while the `EuroSAT-RGB` is unstructured and we will use `UnstructuredParser`. However, in most cases we are going to use the `UnestructuredParser`, as is the easiest one."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "sen12floods",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}