tutorials/notebooks/03_q1_datasets.ipynb

{
   "cells": [
      {
         "attachments": {},
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "# Q1 Training Datasets"
         ]
      },
      {
         "attachments": {},
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "Training Datasets (TDS) in EOTDL are categorized into different [quality levels](https://eotdl.com/docs/datasets/quality), which in turn will impact the range of functionality that will be available for each dataset.\n",
            "\n",
            "In this tutorial you will learn about Q1 datsets, datasets with STAC metadata. "
         ]
      },
      {
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "To ingest a Q1 datasets you will need its STAC metadata.\n",
            "\n",
            "Some datasets already have STAC metadata, and can be ingested directly into EOTDL. However, in case that your dataset does not have STAC metadata but you want to ingest it as a Q1 dataset, the EOTDL library also offers functionality to create the metadata. Let's see an example using the EuroSAT dataset. "
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 1,
         "metadata": {},
         "outputs": [
            {
               "data": {
                  "text/plain": [
                     "'2024.10.07'"
                  ]
               },
               "execution_count": 1,
               "metadata": {},
               "output_type": "execute_result"
            }
         ],
         "source": [
            "import eotdl \n",
            "\n",
            "eotdl.__version__"
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 1,
         "metadata": {},
         "outputs": [
            {
               "name": "stderr",
               "output_type": "stream",
               "text": [
                  "100%|██████████| 1.93G/1.93G [00:29<00:00, 69.8MiB/s]\n",
                  "100%|██████████| 1/1 [00:29<00:00, 29.96s/file]\n"
               ]
            },
            {
               "data": {
                  "text/plain": [
                     "'data/EuroSAT/v1'"
                  ]
               },
               "execution_count": 1,
               "metadata": {},
               "output_type": "execute_result"
            }
         ],
         "source": [
            "from eotdl.datasets import download_dataset\n",
            "\n",
            "download_dataset(\"EuroSAT\", version=1, path=\"data\", force=True)"
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 2,
         "metadata": {},
         "outputs": [
            {
               "name": "stdout",
               "output_type": "stream",
               "text": [
                  "EuroSAT.zip  README.md\n"
               ]
            }
         ],
         "source": [
            "!ls data/EuroSAT/v1"
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 36,
         "metadata": {},
         "outputs": [],
         "source": [
            "!unzip -q data/EuroSAT/v1/EuroSAT.zip -d data/EuroSAT"
         ]
      },
      {
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "The EuroSAT dataset contains satellite images for classification, i.e. each image has one label associated. In this case, the label can be extracted from the folder structure."
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 3,
         "metadata": {},
         "outputs": [
            {
               "data": {
                  "text/plain": [
                     "['AnnualCrop',\n",
                     " 'Forest',\n",
                     " 'HerbaceousVegetation',\n",
                     " 'Highway',\n",
                     " 'Industrial',\n",
                     " 'Pasture',\n",
                     " 'PermanentCrop',\n",
                     " 'Residential',\n",
                     " 'River',\n",
                     " 'SeaLake']"
                  ]
               },
               "execution_count": 3,
               "metadata": {},
               "output_type": "execute_result"
            }
         ],
         "source": [
            "import os \n",
            "\n",
            "base_path = 'data/EuroSAT'\n",
            "images_path = base_path + '/ds/images/remote_sensing/otherDatasets/sentinel_2/tif'\n",
            "labels = os.listdir(images_path)\n",
            "labels = sorted(labels)\n",
            "\n",
            "labels"
         ]
      },
      {
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "For faster processing, we will generate a copy of the dataset with only 10 images per class."
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 4,
         "metadata": {},
         "outputs": [],
         "source": [
            "import shutil \n",
            "\n",
            "os.makedirs('data/EuroSAT-small/', exist_ok=True)\n",
            "for label in labels:\n",
            "    os.makedirs('data/EuroSAT-small/' + label, exist_ok=True)\n",
            "    images = sorted(os.listdir(images_path + '/' + label))[:10]\n",
            "    for image in images:\n",
            "        shutil.copy(images_path + '/' + label + '/' + image, 'data/EuroSAT-small/' + label + '/' + image)"
         ]
      },
      {
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "You can use the `STACGenerator` to create the STAC metadata for your dataset in the form of a dataframe. The item parser will depend on the structure of your dataset. We offer some predefined parsers for common datasets, but you can also create your own parser."
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 5,
         "metadata": {},
         "outputs": [
            {
               "data": {
                  "text/html": [
                     "<div>\n",
                     "<style scoped>\n",
                     "    .dataframe tbody tr th:only-of-type {\n",
                     "        vertical-align: middle;\n",
                     "    }\n",
                     "\n",
                     "    .dataframe tbody tr th {\n",
                     "        vertical-align: top;\n",
                     "    }\n",
                     "\n",
                     "    .dataframe thead th {\n",
                     "        text-align: right;\n",
                     "    }\n",
                     "</style>\n",
                     "<table border=\"1\" class=\"dataframe\">\n",
                     "  <thead>\n",
                     "    <tr style=\"text-align: right;\">\n",
                     "      <th></th>\n",
                     "      <th>image</th>\n",
                     "      <th>label</th>\n",
                     "      <th>ix</th>\n",
                     "      <th>collection</th>\n",
                     "      <th>extensions</th>\n",
                     "      <th>bands</th>\n",
                     "    </tr>\n",
                     "  </thead>\n",
                     "  <tbody>\n",
                     "    <tr>\n",
                     "      <th>0</th>\n",
                     "      <td>data/EuroSAT-small/River/River_1005.tif</td>\n",
                     "      <td>River</td>\n",
                     "      <td>0</td>\n",
                     "      <td>data/EuroSAT-small/source</td>\n",
                     "      <td>None</td>\n",
                     "      <td>None</td>\n",
                     "    </tr>\n",
                     "    <tr>\n",
                     "      <th>1</th>\n",
                     "      <td>data/EuroSAT-small/River/River_1002.tif</td>\n",
                     "      <td>River</td>\n",
                     "      <td>0</td>\n",
                     "      <td>data/EuroSAT-small/source</td>\n",
                     "      <td>None</td>\n",
                     "      <td>None</td>\n",
                     "    </tr>\n",
                     "    <tr>\n",
                     "      <th>2</th>\n",
                     "      <td>data/EuroSAT-small/River/River_1001.tif</td>\n",
                     "      <td>River</td>\n",
                     "      <td>0</td>\n",
                     "      <td>data/EuroSAT-small/source</td>\n",
                     "      <td>None</td>\n",
                     "      <td>None</td>\n",
                     "    </tr>\n",
                     "    <tr>\n",
                     "      <th>3</th>\n",
                     "      <td>data/EuroSAT-small/River/River_1003.tif</td>\n",
                     "      <td>River</td>\n",
                     "      <td>0</td>\n",
                     "      <td>data/EuroSAT-small/source</td>\n",
                     "      <td>None</td>\n",
                     "      <td>None</td>\n",
                     "    </tr>\n",
                     "    <tr>\n",
                     "      <th>4</th>\n",
                     "      <td>data/EuroSAT-small/River/River_1004.tif</td>\n",
                     "      <td>River</td>\n",
                     "      <td>0</td>\n",
                     "      <td>data/EuroSAT-small/source</td>\n",
                     "      <td>None</td>\n",
                     "      <td>None</td>\n",
                     "    </tr>\n",
                     "  </tbody>\n",
                     "</table>\n",
                     "</div>"
                  ],
                  "text/plain": [
                     "                                     image  label  ix  \\\n",
                     "0  data/EuroSAT-small/River/River_1005.tif  River   0   \n",
                     "1  data/EuroSAT-small/River/River_1002.tif  River   0   \n",
                     "2  data/EuroSAT-small/River/River_1001.tif  River   0   \n",
                     "3  data/EuroSAT-small/River/River_1003.tif  River   0   \n",
                     "4  data/EuroSAT-small/River/River_1004.tif  River   0   \n",
                     "\n",
                     "                  collection extensions bands  \n",
                     "0  data/EuroSAT-small/source       None  None  \n",
                     "1  data/EuroSAT-small/source       None  None  \n",
                     "2  data/EuroSAT-small/source       None  None  \n",
                     "3  data/EuroSAT-small/source       None  None  \n",
                     "4  data/EuroSAT-small/source       None  None  "
                  ]
               },
               "execution_count": 5,
               "metadata": {},
               "output_type": "execute_result"
            }
         ],
         "source": [
            "from eotdl.curation.stac.parsers import UnestructuredParser\n",
            "from eotdl.curation.stac.stac import STACGenerator\n",
            "from eotdl.curation.stac.dataframe_labeling import LabeledStrategy\n",
            "\n",
            "stac_generator = STACGenerator(image_format='tif', item_parser=UnestructuredParser, labeling_strategy=LabeledStrategy)\n",
            "\n",
            "df = stac_generator.get_stac_dataframe('data/EuroSAT-small')\n",
            "df.head()\n"
         ]
      },
      {
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "Now we save the STAC metadata. The `id` given to the STAC catalog will be used as the name of the dataset in EOTDL (which has the same requirements than can be found in the [documentation](/docs/datasets/ingest))."
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 6,
         "metadata": {},
         "outputs": [
            {
               "name": "stdout",
               "output_type": "stream",
               "text": [
                  "Generating source collection...\n"
               ]
            },
            {
               "name": "stderr",
               "output_type": "stream",
               "text": [
                  "100%|██████████| 100/100 [00:00<00:00, 399.32it/s]\n"
               ]
            },
            {
               "name": "stdout",
               "output_type": "stream",
               "text": [
                  "Validating and saving catalog...\n",
                  "Success!\n"
               ]
            }
         ],
         "source": [
            "output = 'data/EuroSAT-small-STAC'\n",
            "stac_generator.generate_stac_metadata(\n",
            "\tstac_id='EuroSAT-Q1-small', \n",
            "\tdescription='EuroSAT dataset (only 10 samples per class)', \n",
            "\tstac_dataframe=df, \n",
            "\toutput_folder=output\n",
            ")"
         ]
      },
      {
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "And, optionally, the labels using the labels extension."
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 7,
         "metadata": {},
         "outputs": [
            {
               "name": "stdout",
               "output_type": "stream",
               "text": [
                  "Generating labels collection...\n"
               ]
            },
            {
               "name": "stderr",
               "output_type": "stream",
               "text": [
                  "100it [00:00, 1494.03it/s]\n"
               ]
            },
            {
               "name": "stdout",
               "output_type": "stream",
               "text": [
                  "Success on labels generation!\n"
               ]
            }
         ],
         "source": [
            "from eotdl.curation.stac.extensions.label import ImageNameLabeler\n",
            "\n",
            "catalog = output + '/catalog.json'\n",
            "labels_extra_properties = {'label_properties': [\"label\"],\n",
            "                          'label_methods': [\"manual\"],\n",
            "                          'label_tasks': [\"classification\"]}\n",
            "\n",
            "labeler = ImageNameLabeler()\n",
            "labeler.generate_stac_labels(catalog, stac_dataframe=df, **labels_extra_properties)"
         ]
      },
      {
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "Once the STAC metadata is generated, we can ingest the dataset into EOTDL."
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 8,
         "metadata": {},
         "outputs": [
            {
               "name": "stdout",
               "output_type": "stream",
               "text": [
                  "Loading STAC catalog...\n",
                  "New version created, version: 1\n"
               ]
            },
            {
               "name": "stderr",
               "output_type": "stream",
               "text": [
                  "100%|██████████| 200/200 [00:44<00:00,  4.54it/s]\n"
               ]
            },
            {
               "name": "stdout",
               "output_type": "stream",
               "text": [
                  "Ingesting STAC catalog...\n",
                  "Done\n"
               ]
            }
         ],
         "source": [
            "from eotdl.datasets import ingest_dataset\n",
            "\n",
            "ingest_dataset('data/EuroSAT-small-STAC')"
         ]
      },
      {
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "After the ingestion, you can explore and stage your dataset like shown in the previous tutorial."
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 8,
         "metadata": {},
         "outputs": [
            {
               "data": {
                  "text/plain": [
                     "['EuroSAT-Q1-small',\n",
                     " 'EuroSAT-RGB',\n",
                     " 'EuroSAT-RGB-Q1',\n",
                     " 'EuroSAT-RGB-Q2',\n",
                     " 'EuroSAT-RGB-STAC',\n",
                     " 'EuroSAT-STAC',\n",
                     " 'EuroSAT']"
                  ]
               },
               "execution_count": 8,
               "metadata": {},
               "output_type": "execute_result"
            }
         ],
         "source": [
            "from eotdl.datasets import retrieve_datasets\n",
            "\n",
            "datasets = retrieve_datasets('EuroSAT')\n",
            "datasets"
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 10,
         "metadata": {},
         "outputs": [
            {
               "name": "stdout",
               "output_type": "stream",
               "text": [
                  "To download assets, set assets=True or -a in the CLI.\n"
               ]
            },
            {
               "data": {
                  "text/plain": [
                     "'data/EuroSAT-Q1-small/v1'"
                  ]
               },
               "execution_count": 10,
               "metadata": {},
               "output_type": "execute_result"
            }
         ],
         "source": [
            "from eotdl.datasets import download_dataset\n",
            "\n",
            "dst_path = download_dataset('EuroSAT-Q1-small', path='data')\n",
            "dst_path"
         ]
      },
      {
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "By default it will only download the STAC metadata. In case you also want to download the actual data, you can use the `assets` parameter. "
         ]
      },
      {
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "> The `force` parameter will overwrite the dataset if it already exists."
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 11,
         "metadata": {},
         "outputs": [
            {
               "name": "stderr",
               "output_type": "stream",
               "text": [
                  "100%|██████████| 200/200 [00:49<00:00,  4.07it/s]\n"
               ]
            },
            {
               "data": {
                  "text/plain": [
                     "'data/EuroSAT-Q1-small/v1'"
                  ]
               },
               "execution_count": 11,
               "metadata": {},
               "output_type": "execute_result"
            }
         ],
         "source": [
            "from eotdl.datasets import download_dataset\n",
            "\n",
            "dst_path =  download_dataset('EuroSAT-Q1-small', path='data', assets=True, force=True)\n",
            "dst_path"
         ]
      },
      {
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "You will find the data in  the `assets` subfolder, where a subfolder for each items with its `id` will contain all the assets for that item."
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 12,
         "metadata": {},
         "outputs": [
            {
               "data": {
                  "text/plain": [
                     "['data/EuroSAT-Q1-small/v1/assets/Industrial_1000.tif',\n",
                     " 'data/EuroSAT-Q1-small/v1/assets/Industrial_10.tif',\n",
                     " 'data/EuroSAT-Q1-small/v1/assets/Highway_1.tif']"
                  ]
               },
               "execution_count": 12,
               "metadata": {},
               "output_type": "execute_result"
            }
         ],
         "source": [
            "from glob import glob\n",
            "\n",
            "glob(dst_path + '/assets/*.tif')[:3]"
         ]
      },
      {
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "Alternatively, you can download an asset using its url."
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 16,
         "metadata": {},
         "outputs": [
            {
               "data": {
                  "text/plain": [
                     "{'AnnualCrop_1': {'href': 'https://api.eotdl.com/datasets/662a3c33038fa20bdd97ead2/download/AnnualCrop_1.tif',\n",
                     "  'type': 'image/tiff; application=geotiff',\n",
                     "  'title': 'AnnualCrop_1',\n",
                     "  'roles': ['data']}}"
                  ]
               },
               "execution_count": 16,
               "metadata": {},
               "output_type": "execute_result"
            }
         ],
         "source": [
            "import json\n",
            "\n",
            "with open(dst_path + '/EuroSAT-Q1-small/source/AnnualCrop_1/AnnualCrop_1.json', 'r') as f:\n",
            "\tdata = json.load(f)\n",
            "\n",
            "data['assets']"
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 17,
         "metadata": {},
         "outputs": [
            {
               "data": {
                  "text/plain": [
                     "'data/assets/AnnualCrop_1.tif'"
                  ]
               },
               "execution_count": 17,
               "metadata": {},
               "output_type": "execute_result"
            }
         ],
         "source": [
            "from eotdl.datasets import download_file_url\n",
            "\n",
            "url = data['assets']['AnnualCrop_1']['href']\n",
            "download_file_url(url, 'data')"
         ]
      }
   ],
   "metadata": {
      "kernelspec": {
         "display_name": "Python 3 (ipykernel)",
         "language": "python",
         "name": "python3"
      },
      "language_info": {
         "codemirror_mode": {
            "name": "ipython",
            "version": 3
         },
         "file_extension": ".py",
         "mimetype": "text/x-python",
         "name": "python",
         "nbconvert_exporter": "python",
         "pygments_lexer": "ipython3",
         "version": "3.8.19"
      }
   },
   "nbformat": 4,
   "nbformat_minor": 2
}