tutorials/workshops/philab24/01_exploring.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exploring and Downloading Datasets and Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start by exploring the repository of datasets and models. \n",
    "\n",
    "You can do that at the different accessibility layers of EOTDL: the user interface, the API, the command line interface (CLI) and the Python library."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The User Interface"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The easiest way to get started with EOTDL is by exploring the user interface: [https://eotdl.com/](https://www.eotdl.com/). Through the UI you will be able to:\n",
    "\n",
    "- Explore the datasets and models available in the repository (filtering by name, tags and liked)\n",
    "- Edit your own datasets and models information.\n",
    "- Read the tutorials on the blog.\n",
    "- Read the documentation.\n",
    "- Find useful links to other resources (GitHub, Discord, ...)\n",
    "\n",
    "![web](images/web.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Quality levels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Datasets and models in EOTDL are categorized into quality levels. The quality levels are:\n",
    "\n",
    "- **Q0**: datasets in the form of an archive with arbitary files without curation. This level is ideal for easy and fast upload/download of small datasets.\n",
    "- **Q1**: datasets with STAC metadata but no QA. These datasets can leverage a limited set of EOTDL features.\n",
    "- **Q2**: datasets with STAC metadata with the EOTDL custom extensions and automated QA. These datasets can leverage the full potential of the EOTDL.\n",
    "- **Q3**: Q2 datasets that are manually curated. These datasets are the most reliable and can be used as benchmark datasets.\n",
    "\n",
    "You will learn more about the quality levels in the [data curation](05_stac.ipynb) tutorial."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Command Line Interface"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Even though the UI is the easiest way to get started, it is not the most convenient for actually working with the datasets and models. For that we recommend installing the CLI.\n",
    "\n",
    "If you are running this notebook locally, consider creating a virtual environment before installing the CLI to avoid conflicts with other packages.\n",
    "\n",
    "With conda:\n",
    "\n",
    "```bash\n",
    "conda create -n eotdl python=3.8\n",
    "conda activate eotdl\n",
    "```\n",
    "\n",
    "With python: \n",
    "\n",
    "```bash\n",
    "python -m venv eotdl\n",
    "source eotdl/bin/activate\n",
    "```\n",
    "\n",
    "You may also have to install Jupyter on the new environment and restart the notebook.\n",
    "\n",
    "Then, you can install the CLI with pip:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# uncomment to install\n",
    "\n",
    "# !pip install eotdl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once installed, you can execute the CLI with different commands. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m                                                                                \u001b[0m\n",
      "\u001b[1m \u001b[0m\u001b[1;33mUsage: \u001b[0m\u001b[1meotdl [OPTIONS] COMMAND [ARGS]...\u001b[0m\u001b[1m                                      \u001b[0m\u001b[1m \u001b[0m\n",
      "\u001b[1m                                                                                \u001b[0m\n",
      " Welcome to EOTDL. Learn more at https://www.eotdl.com/                         \n",
      "                                                                                \n",
      "\u001b[2m╭─\u001b[0m\u001b[2m Options \u001b[0m\u001b[2m───────────────────────────────────────────────────────────────────\u001b[0m\u001b[2m─╮\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-install\u001b[0m\u001b[1;36m-completion\u001b[0m          Install completion for the current shell.      \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-show\u001b[0m\u001b[1;36m-completion\u001b[0m             Show completion for the current shell, to copy \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m                               it or customize the installation.              \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-help\u001b[0m                        Show this message and exit.                    \u001b[2m│\u001b[0m\n",
      "\u001b[2m╰──────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n",
      "\u001b[2m╭─\u001b[0m\u001b[2m Commands \u001b[0m\u001b[2m──────────────────────────────────────────────────────────────────\u001b[0m\u001b[2m─╮\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36mapi        \u001b[0m\u001b[1;36m \u001b[0m Get EOTDL API URL and info.                                     \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36mauth       \u001b[0m\u001b[1;36m \u001b[0m Login to EOTDL.                                                 \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36mdatasets   \u001b[0m\u001b[1;36m \u001b[0m Explore, ingest and download training datasets.                 \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36mmodels     \u001b[0m\u001b[1;36m \u001b[0m Explore, ingest and download ML models.                         \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36mversion    \u001b[0m\u001b[1;36m \u001b[0m Get EOTDL version.                                              \u001b[2m│\u001b[0m\n",
      "\u001b[2m╰──────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!eotdl --help"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EOTDL Version: 2024.04.25\n"
     ]
    }
   ],
   "source": [
    "!eotdl version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m                                                                                \u001b[0m\n",
      "\u001b[1m \u001b[0m\u001b[1;33mUsage: \u001b[0m\u001b[1meotdl datasets [OPTIONS] COMMAND [ARGS]...\u001b[0m\u001b[1m                             \u001b[0m\u001b[1m \u001b[0m\n",
      "\u001b[1m                                                                                \u001b[0m\n",
      " Explore, ingest and download training datasets.                                \n",
      "                                                                                \n",
      "\u001b[2m╭─\u001b[0m\u001b[2m Options \u001b[0m\u001b[2m───────────────────────────────────────────────────────────────────\u001b[0m\u001b[2m─╮\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-help\u001b[0m          Show this message and exit.                                  \u001b[2m│\u001b[0m\n",
      "\u001b[2m╰──────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n",
      "\u001b[2m╭─\u001b[0m\u001b[2m Commands \u001b[0m\u001b[2m──────────────────────────────────────────────────────────────────\u001b[0m\u001b[2m─╮\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36mget      \u001b[0m\u001b[1;36m \u001b[0m Download a dataset from the EOTDL.                                \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36mingest   \u001b[0m\u001b[1;36m \u001b[0m Ingest a dataset to the EOTDL.                                    \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36mlist     \u001b[0m\u001b[1;36m \u001b[0m Retrieve a list with all the datasets in the EOTDL.               \u001b[2m│\u001b[0m\n",
      "\u001b[2m╰──────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!eotdl datasets --help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can explore datasets with the following command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['EuroSAT-Q1', 'EuroSAT-Q1-small', 'EuroSAT-Q2-small', 'SEN12MS-CR', 'DeepGlobeRoadExtraction', 'MassachusettsRoadsDataset', 'OpenEarthMap', 'ESA-Worldcover', 'AlignSAR-Groningen-Sentinel1-Q0', 'AI4EO-MapYourCity', 'Enhanced-Sentinel-2-Agriculture', 'WorldStrat', 'SEN2Venus', 'AlignSAR-Groningen-Sentinel1-Q1', 'SEN12MS', 'AlignSAR-Chennai-OilSpill-Sentinel1-Q0', 'PhilEO-downstream', 'Alignsar', 'boadella-dataset', 'EuroSAT-RGB', 'EuroSAT-RGB-Q1', 'EuroSAT-RGB-Q2', 'Boadella-BiDS23', 'COWC', 'Stanford-Drone-dataset', 'EuroSAT-RGB-STAC', 'BigEarthNet', 'xview2', 'LandcoverAI', 'open-cities-tt2-source', 'open-cities-tt1-source', 'open-cities-test', 'PASTIS-R', 'EuroCrops', 'SloveniaLandCover', 'ISPRS-Potsdam2D', 'SEN12-FLOOD', 'Urban3dChallenge', 'tropical-cyclone-dataset', 'Vessel-detection', 'Airplanes-detection', 'S2-SHIPS', 'SpaceNet-7', 'Sentinel-2-Cloud-Mask', 'PASTIS', 'FlodNet', 'SeCo100k', 'SeCo', 'AirbusAircraftDetection', 'AirbusWindTurbinesPatches', 'RoadNet', 'EuroSAT', 'UCMerced']\n"
     ]
    }
   ],
   "source": [
    "!eotdl datasets list "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m                                                                                \u001b[0m\n",
      "\u001b[1m \u001b[0m\u001b[1;33mUsage: \u001b[0m\u001b[1meotdl datasets list [OPTIONS]\u001b[0m\u001b[1m                                          \u001b[0m\u001b[1m \u001b[0m\n",
      "\u001b[1m                                                                                \u001b[0m\n",
      " Retrieve a list with all the datasets in the EOTDL.                            \n",
      " \u001b[2mIf using \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-name\u001b[0m\u001b[2m, it will filter the results by name. If no name is provided, \u001b[0m  \n",
      " \u001b[2mit will return all the datasets.\u001b[0m                                               \n",
      " \u001b[2mIf using \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-limit\u001b[0m\u001b[2m, it will limit the number of results. If no limit is \u001b[0m         \n",
      " \u001b[2mprovided, it will return all the datasets.\u001b[0m                                     \n",
      "                                                                                \n",
      " \u001b[2mExamples\u001b[0m                                                                       \n",
      " \u001b[1;2;36m--------\u001b[0m                                                                       \n",
      " \u001b[2m$ eotdl datasets list\u001b[0m                                                          \n",
      " \u001b[2m$ eotdl datasets list \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-name\u001b[0m\u001b[2m YourModel \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-limit\u001b[0m\u001b[2m 5\u001b[0m                               \n",
      "                                                                                \n",
      "\u001b[2m╭─\u001b[0m\u001b[2m Options \u001b[0m\u001b[2m───────────────────────────────────────────────────────────────────\u001b[0m\u001b[2m─╮\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-name\u001b[0m   \u001b[1;32m-n\u001b[0m      \u001b[1;33mTEXT   \u001b[0m  Filter the returned datasets by name               \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m                           \u001b[2m[default: None]                     \u001b[0m               \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-limit\u001b[0m  \u001b[1;32m-l\u001b[0m      \u001b[1;33mINTEGER\u001b[0m  Limit the number of returned results               \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m                           \u001b[2m[default: None]                     \u001b[0m               \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-help\u001b[0m           \u001b[1;33m       \u001b[0m  Show this message and exit.                        \u001b[2m│\u001b[0m\n",
      "\u001b[2m╰──────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!eotdl datasets list --help"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['EuroSAT-Q1', 'EuroSAT-Q1-small', 'EuroSAT-Q2-small', 'EuroSAT-RGB', 'EuroSAT-RGB-Q1', 'EuroSAT-RGB-Q2', 'EuroSAT-RGB-STAC', 'EuroSAT']\n"
     ]
    }
   ],
   "source": [
    "!eotdl datasets list -n eurosat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you may have guessed, you can download a dataset with the following command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  0%|                                                   | 0/1 [00:00<?, ?file/s]\n",
      "  0%|                                              | 0.00/90.3M [00:00<?, ?iB/s]\u001b[A\n",
      " 11%|████                                 | 10.0M/90.3M [00:02<00:22, 3.67MiB/s]\u001b[A\n",
      " 22%|████████▏                            | 20.0M/90.3M [00:03<00:10, 7.03MiB/s]\u001b[A\n",
      " 33%|████████████▎                        | 30.0M/90.3M [00:03<00:05, 11.0MiB/s]\u001b[A\n",
      " 44%|████████████████▍                    | 40.0M/90.3M [00:03<00:03, 15.7MiB/s]\u001b[A\n",
      " 55%|████████████████████▍                | 50.0M/90.3M [00:04<00:01, 21.4MiB/s]\u001b[A\n",
      " 66%|████████████████████████▌            | 60.0M/90.3M [00:04<00:01, 28.1MiB/s]\u001b[A\n",
      " 78%|████████████████████████████▋        | 70.0M/90.3M [00:04<00:00, 35.2MiB/s]\u001b[A\n",
      " 89%|████████████████████████████████▊    | 80.0M/90.3M [00:04<00:00, 42.2MiB/s]\u001b[A\n",
      "100%|█████████████████████████████████████| 90.3M/90.3M [00:04<00:00, 20.2MiB/s]\u001b[A\n",
      "100%|███████████████████████████████████████████| 1/1 [00:04<00:00,  4.98s/file]\n",
      "Data available at /home/juan/.cache/eotdl/datasets/EuroSAT-RGB/v1\n"
     ]
    }
   ],
   "source": [
    "!eotdl datasets get EuroSAT-RGB -v 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first time you run the command, you will be asked to login (which will require you to create an account if you haven't already). You can also login with the command"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "On your computer or mobile device navigate to:  https://earthpulse.eu.auth0.com/activate?user_code=CGZB-SPQB\n",
      "Authenticated!\n",
      "- Id Token: eyJhbGciOi...\n",
      "Saved credentials to:  /home/juan/.cache/eotdl/creds.json\n",
      "You are logged in as it@earthpulse.es\n"
     ]
    }
   ],
   "source": [
    "!eotdl auth login"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m                                                                                \u001b[0m\n",
      "\u001b[1m \u001b[0m\u001b[1;33mUsage: \u001b[0m\u001b[1meotdl auth [OPTIONS] COMMAND [ARGS]...\u001b[0m\u001b[1m                                 \u001b[0m\u001b[1m \u001b[0m\n",
      "\u001b[1m                                                                                \u001b[0m\n",
      " Login to EOTDL.                                                                \n",
      "                                                                                \n",
      "\u001b[2m╭─\u001b[0m\u001b[2m Options \u001b[0m\u001b[2m───────────────────────────────────────────────────────────────────\u001b[0m\u001b[2m─╮\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-help\u001b[0m          Show this message and exit.                                  \u001b[2m│\u001b[0m\n",
      "\u001b[2m╰──────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n",
      "\u001b[2m╭─\u001b[0m\u001b[2m Commands \u001b[0m\u001b[2m──────────────────────────────────────────────────────────────────\u001b[0m\u001b[2m─╮\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36mlogin            \u001b[0m\u001b[1;36m \u001b[0m Login to the EOTDL.                                       \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36mlogout           \u001b[0m\u001b[1;36m \u001b[0m Logout from the EOTDL.                                    \u001b[2m│\u001b[0m\n",
      "\u001b[2m╰──────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!eotdl auth --help"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m                                                                                \u001b[0m\n",
      "\u001b[1m \u001b[0m\u001b[1;33mUsage: \u001b[0m\u001b[1meotdl datasets get [OPTIONS] [DATASET]\u001b[0m\u001b[1m                                 \u001b[0m\u001b[1m \u001b[0m\n",
      "\u001b[1m                                                                                \u001b[0m\n",
      " Download a dataset from the EOTDL.                                             \n",
      " \u001b[2mIf using \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-path\u001b[0m\u001b[2m, it will download the dataset to the specified path. If no \u001b[0m    \n",
      " \u001b[2mpath is provided, it will download to ~/.eotdl/datasets.\u001b[0m                       \n",
      " \u001b[2mIf using \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-file\u001b[0m\u001b[2m, it will download the specified file. If no file is provided, \u001b[0m \n",
      " \u001b[2mit will download the entire dataset.\u001b[0m                                           \n",
      " \u001b[2mIf using \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-version\u001b[0m\u001b[2m, it will download the specified version. If no version is \u001b[0m  \n",
      " \u001b[2mprovided, it will download the latest version.\u001b[0m                                 \n",
      " \u001b[2mIf using \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-assets\u001b[0m\u001b[2m when the dataset is STAC, it will also download the STAC \u001b[0m    \n",
      " \u001b[2massets of the dataset. If not provided, it will only download the STAC \u001b[0m        \n",
      " \u001b[2mmetadata.\u001b[0m                                                                      \n",
      " \u001b[2mIf using \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-force\u001b[0m\u001b[2m, it will download the dataset even if the file already \u001b[0m       \n",
      " \u001b[2mexists.\u001b[0m                                                                        \n",
      " \u001b[2mIf using \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-verbose\u001b[0m\u001b[2m, it will print the progress of the download.\u001b[0m                \n",
      "                                                                                \n",
      " \u001b[2mExamples\u001b[0m                                                                       \n",
      " \u001b[1;2;36m--------\u001b[0m                                                                       \n",
      " \u001b[2m$ eotdl dataset get YourDataset\u001b[0m                                                \n",
      " \u001b[2m$ eotdl dataset get YourDataset \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-path\u001b[0m\u001b[2m /path/to/download \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-file\u001b[0m\u001b[2m dataset.zip \u001b[0m   \n",
      " \u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-version\u001b[0m\u001b[2m 1 \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-assets\u001b[0m\u001b[2m True \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-force\u001b[0m\u001b[2m True \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-verbose\u001b[0m\u001b[2m True\u001b[0m                          \n",
      "                                                                                \n",
      "\u001b[2m╭─\u001b[0m\u001b[2m Arguments \u001b[0m\u001b[2m─────────────────────────────────────────────────────────────────\u001b[0m\u001b[2m─╮\u001b[0m\n",
      "\u001b[2m│\u001b[0m   dataset      \u001b[1;2;33m[\u001b[0m\u001b[1;33mDATASET\u001b[0m\u001b[1;2;33m]\u001b[0m  Name of the dataset to download \u001b[2m[default: None]\u001b[0m    \u001b[2m│\u001b[0m\n",
      "\u001b[2m╰──────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n",
      "\u001b[2m╭─\u001b[0m\u001b[2m Options \u001b[0m\u001b[2m───────────────────────────────────────────────────────────────────\u001b[0m\u001b[2m─╮\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-path\u001b[0m     \u001b[1;32m-p\u001b[0m      \u001b[1;33mTEXT   \u001b[0m  Download the dataset to a specific output path   \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m                             \u001b[2m[default: None]                               \u001b[0m   \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-file\u001b[0m     \u001b[1;32m-f\u001b[0m      \u001b[1;33mTEXT   \u001b[0m  Download a specific file from the dataset        \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m                             \u001b[2m[default: None]                          \u001b[0m        \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-version\u001b[0m  \u001b[1;32m-v\u001b[0m      \u001b[1;33mINTEGER\u001b[0m  Dataset version \u001b[2m[default: None]\u001b[0m                  \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-assets\u001b[0m   \u001b[1;32m-a\u001b[0m      \u001b[1;33m       \u001b[0m  Download STAC assets from the dataset            \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-force\u001b[0m    \u001b[1;32m-f\u001b[0m      \u001b[1;33m       \u001b[0m  Force download even if file exists               \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-verbose\u001b[0m          \u001b[1;33m       \u001b[0m  Verbose output. This will print the progress of  \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m                             the download                                     \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-help\u001b[0m             \u001b[1;33m       \u001b[0m  Show this message and exit.                      \u001b[2m│\u001b[0m\n",
      "\u001b[2m╰──────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!eotdl datasets get --help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, datasets will be downloaded to your `$HOME/.cache/eotdl/datasets` folder or the path in the `EOTDL_DOWNLOAD_PATH` environment variable. You can change this with the `--path` argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  0%|                                                   | 0/1 [00:00<?, ?file/s]\n",
      "  0%|                                              | 0.00/90.3M [00:00<?, ?iB/s]\u001b[A\n",
      " 11%|████                                 | 10.0M/90.3M [00:02<00:23, 3.61MiB/s]\u001b[A\n",
      " 22%|████████▏                            | 20.0M/90.3M [00:03<00:10, 6.93MiB/s]\u001b[A\n",
      " 33%|████████████▎                        | 30.0M/90.3M [00:03<00:05, 10.8MiB/s]\u001b[A\n",
      " 44%|████████████████▍                    | 40.0M/90.3M [00:03<00:03, 15.6MiB/s]\u001b[A\n",
      " 55%|████████████████████▍                | 50.0M/90.3M [00:04<00:01, 21.2MiB/s]\u001b[A\n",
      " 66%|████████████████████████▌            | 60.0M/90.3M [00:04<00:01, 27.8MiB/s]\u001b[A\n",
      " 78%|████████████████████████████▋        | 70.0M/90.3M [00:04<00:00, 34.6MiB/s]\u001b[A\n",
      " 89%|████████████████████████████████▊    | 80.0M/90.3M [00:04<00:00, 41.3MiB/s]\u001b[A\n",
      "100%|█████████████████████████████████████| 90.3M/90.3M [00:04<00:00, 19.9MiB/s]\u001b[A\n",
      "100%|███████████████████████████████████████████| 1/1 [00:05<00:00,  5.22s/file]\n",
      "Data available at data/EuroSAT-RGB/v1\n"
     ]
    }
   ],
   "source": [
    "!eotdl datasets get EuroSAT-RGB -v 1 -p data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can choose a particular version to download with the `--version` argument. If you don't specify a version, the latest version will be downloaded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset `EuroSAT-RGB v1` already exists at data/EuroSAT-RGB/v1. To force download, use force=True or -f in the CLI.\n"
     ]
    }
   ],
   "source": [
    "!eotdl datasets get EuroSAT-RGB -p data -v 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The version number will be used to create a folder with the same name inside the path you specified. Inside this folder you will find the dataset files.\n",
    "\n",
    "If you try to re-download a datasets, the CLI will complain. You can force a re-download with the `--force` argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  0%|                                                   | 0/1 [00:00<?, ?file/s]\n",
      "  0%|                                              | 0.00/90.3M [00:00<?, ?iB/s]\u001b[A\n",
      " 11%|████                                 | 10.0M/90.3M [00:02<00:23, 3.61MiB/s]\u001b[A\n",
      " 22%|████████▏                            | 20.0M/90.3M [00:03<00:10, 6.95MiB/s]\u001b[A\n",
      " 33%|████████████▎                        | 30.0M/90.3M [00:03<00:05, 10.9MiB/s]\u001b[A\n",
      " 44%|████████████████▍                    | 40.0M/90.3M [00:03<00:03, 15.6MiB/s]\u001b[A\n",
      " 55%|████████████████████▍                | 50.0M/90.3M [00:04<00:01, 21.2MiB/s]\u001b[A\n",
      " 66%|████████████████████████▌            | 60.0M/90.3M [00:04<00:01, 27.8MiB/s]\u001b[A\n",
      " 78%|████████████████████████████▋        | 70.0M/90.3M [00:04<00:00, 34.8MiB/s]\u001b[A\n",
      " 89%|████████████████████████████████▊    | 80.0M/90.3M [00:04<00:00, 41.8MiB/s]\u001b[A\n",
      "100%|█████████████████████████████████████| 90.3M/90.3M [00:04<00:00, 19.8MiB/s]\u001b[A\n",
      "100%|███████████████████████████████████████████| 1/1 [00:05<00:00,  5.07s/file]\n",
      "Data available at data/EuroSAT-RGB/v1\n"
     ]
    }
   ],
   "source": [
    "!eotdl datasets get EuroSAT-RGB -p data -v 1 -f"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For Q1+ datasets, the `get` command will only download the STAC metadata of the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "To download assets, set assets=True or -a in the CLI.\n",
      "Data available at data/EuroSAT-Q1-small/v1\n"
     ]
    }
   ],
   "source": [
    "!eotdl datasets get EuroSAT-Q1-small -p data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Inside the metadata you will find the links to all the assets, so you can download them individually (maybe after some filtering or processing using only the metadata). However, you can download all assets with the command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████████| 200/200 [01:03<00:00,  3.16it/s]\n",
      "Data available at data/EuroSAT-Q1-small/v1\n"
     ]
    }
   ],
   "source": [
    "!eotdl datasets get EuroSAT-Q1-small -p data -a -f"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Working with models is very much the same at this point."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m                                                                                \u001b[0m\n",
      "\u001b[1m \u001b[0m\u001b[1;33mUsage: \u001b[0m\u001b[1meotdl models [OPTIONS] COMMAND [ARGS]...\u001b[0m\u001b[1m                               \u001b[0m\u001b[1m \u001b[0m\n",
      "\u001b[1m                                                                                \u001b[0m\n",
      " Explore, ingest and download ML models.                                        \n",
      "                                                                                \n",
      "\u001b[2m╭─\u001b[0m\u001b[2m Options \u001b[0m\u001b[2m───────────────────────────────────────────────────────────────────\u001b[0m\u001b[2m─╮\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-help\u001b[0m          Show this message and exit.                                  \u001b[2m│\u001b[0m\n",
      "\u001b[2m╰──────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n",
      "\u001b[2m╭─\u001b[0m\u001b[2m Commands \u001b[0m\u001b[2m──────────────────────────────────────────────────────────────────\u001b[0m\u001b[2m─╮\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36mget      \u001b[0m\u001b[1;36m \u001b[0m Download a model from the EOTDL.                                  \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36mingest   \u001b[0m\u001b[1;36m \u001b[0m Ingest a model to the EOTDL.                                      \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36mlist     \u001b[0m\u001b[1;36m \u001b[0m Retrieve a list with all the models in the EOTDL.                 \u001b[2m│\u001b[0m\n",
      "\u001b[2m╰──────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!eotdl models --help"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['RoadSegmentation', 'EuroSAT-RGB-BiDS23-Q1', 'MAVERICC', 'WALDO25', 'EuroSAT-RGB-BiDS23']\n"
     ]
    }
   ],
   "source": [
    "!eotdl models list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m                                                                                \u001b[0m\n",
      "\u001b[1m \u001b[0m\u001b[1;33mUsage: \u001b[0m\u001b[1meotdl models list [OPTIONS]\u001b[0m\u001b[1m                                            \u001b[0m\u001b[1m \u001b[0m\n",
      "\u001b[1m                                                                                \u001b[0m\n",
      " Retrieve a list with all the models in the EOTDL.                              \n",
      " \u001b[2mIf using \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-name\u001b[0m\u001b[2m, it will filter the results by name. If no name is provided, \u001b[0m  \n",
      " \u001b[2mit will return all the models.\u001b[0m                                                 \n",
      " \u001b[2mIf using \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-limit\u001b[0m\u001b[2m, it will limit the number of results. If no limit is \u001b[0m         \n",
      " \u001b[2mprovided, it will return all the models.\u001b[0m                                       \n",
      "                                                                                \n",
      " \u001b[2mExamples\u001b[0m                                                                       \n",
      " \u001b[1;2;36m--------\u001b[0m                                                                       \n",
      " \u001b[2m$ eotdl models list\u001b[0m                                                            \n",
      " \u001b[2m$ eotdl models list \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-name\u001b[0m\u001b[2m YourModel \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-limit\u001b[0m\u001b[2m 5\u001b[0m                                 \n",
      "                                                                                \n",
      "\u001b[2m╭─\u001b[0m\u001b[2m Options \u001b[0m\u001b[2m───────────────────────────────────────────────────────────────────\u001b[0m\u001b[2m─╮\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-name\u001b[0m   \u001b[1;32m-n\u001b[0m      \u001b[1;33mTEXT   \u001b[0m  Filter the returned models by name \u001b[2m[default: None]\u001b[0m \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-limit\u001b[0m  \u001b[1;32m-l\u001b[0m      \u001b[1;33mINTEGER\u001b[0m  Limit the number of returned results               \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m                           \u001b[2m[default: None]                     \u001b[0m               \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-help\u001b[0m           \u001b[1;33m       \u001b[0m  Show this message and exit.                        \u001b[2m│\u001b[0m\n",
      "\u001b[2m╰──────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!eotdl models list --help"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████████| 1/1 [00:05<00:00,  5.54s/file]\n",
      "Data available at /home/juan/.cache/eotdl/models/RoadSegmentation/v1\n"
     ]
    }
   ],
   "source": [
    "!eotdl models get RoadSegmentation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m                                                                                \u001b[0m\n",
      "\u001b[1m \u001b[0m\u001b[1;33mUsage: \u001b[0m\u001b[1meotdl models get [OPTIONS] [MODEL]\u001b[0m\u001b[1m                                     \u001b[0m\u001b[1m \u001b[0m\n",
      "\u001b[1m                                                                                \u001b[0m\n",
      " Download a model from the EOTDL.                                               \n",
      " \u001b[2mIf using \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-path\u001b[0m\u001b[2m, it will download the model to the specified path. If no path \u001b[0m \n",
      " \u001b[2mis provided, it will download to ~/.eotdl/models.\u001b[0m                              \n",
      " \u001b[2mIf using \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-file\u001b[0m\u001b[2m, it will download the specified file. If no file is provided, \u001b[0m \n",
      " \u001b[2mit will download the entire model.\u001b[0m                                             \n",
      " \u001b[2mIf using \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-version\u001b[0m\u001b[2m, it will download the specified version. If no version is \u001b[0m  \n",
      " \u001b[2mprovided, it will download the latest version.\u001b[0m                                 \n",
      " \u001b[2mIf using \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-assets\u001b[0m\u001b[2m when the model is STAC, it will also download the STAC \u001b[0m      \n",
      " \u001b[2massets of the model. If not provided, it will only download the STAC metadata.\u001b[0m \n",
      " \u001b[2mIf using \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-force\u001b[0m\u001b[2m, it will download the model even if the file already exists.\u001b[0m  \n",
      " \u001b[2mIf using \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-verbose\u001b[0m\u001b[2m, it will print the progress of the download.\u001b[0m                \n",
      "                                                                                \n",
      " \u001b[2mExamples\u001b[0m                                                                       \n",
      " \u001b[1;2;36m--------\u001b[0m                                                                       \n",
      " \u001b[2m$ eotdl models get YourModel\u001b[0m                                                   \n",
      " \u001b[2m$ eotdl models get YourModel \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-path\u001b[0m\u001b[2m /path/to/download \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-file\u001b[0m\u001b[2m model.zip \u001b[0m        \n",
      " \u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-version\u001b[0m\u001b[2m 1 \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-assets\u001b[0m\u001b[2m True \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-force\u001b[0m\u001b[2m True \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-verbose\u001b[0m\u001b[2m True\u001b[0m                          \n",
      "                                                                                \n",
      "\u001b[2m╭─\u001b[0m\u001b[2m Arguments \u001b[0m\u001b[2m─────────────────────────────────────────────────────────────────\u001b[0m\u001b[2m─╮\u001b[0m\n",
      "\u001b[2m│\u001b[0m   model      \u001b[1;2;33m[\u001b[0m\u001b[1;33mMODEL\u001b[0m\u001b[1;2;33m]\u001b[0m  Name of the model to download \u001b[2m[default: None]\u001b[0m          \u001b[2m│\u001b[0m\n",
      "\u001b[2m╰──────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n",
      "\u001b[2m╭─\u001b[0m\u001b[2m Options \u001b[0m\u001b[2m───────────────────────────────────────────────────────────────────\u001b[0m\u001b[2m─╮\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-path\u001b[0m     \u001b[1;32m-p\u001b[0m      \u001b[1;33mTEXT   \u001b[0m  Download the model to a specific output path     \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m                             \u001b[2m[default: None]                             \u001b[0m     \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-file\u001b[0m     \u001b[1;32m-f\u001b[0m      \u001b[1;33mTEXT   \u001b[0m  Download a specific file from the model          \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m                             \u001b[2m[default: None]                        \u001b[0m          \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-version\u001b[0m  \u001b[1;32m-v\u001b[0m      \u001b[1;33mINTEGER\u001b[0m  Model version \u001b[2m[default: None]\u001b[0m                    \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-assets\u001b[0m   \u001b[1;32m-a\u001b[0m      \u001b[1;33m       \u001b[0m  Download STAC assets from the model              \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-force\u001b[0m    \u001b[1;32m-f\u001b[0m      \u001b[1;33m       \u001b[0m  Force download even if file exists               \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-verbose\u001b[0m          \u001b[1;33m       \u001b[0m  Verbose output. This will print the progress of  \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m                             the download                                     \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-help\u001b[0m             \u001b[1;33m       \u001b[0m  Show this message and exit.                      \u001b[2m│\u001b[0m\n",
      "\u001b[2m╰──────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!eotdl models get --help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will explore how to ingest datasets and models in the next tutorials."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Library"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Everything that we have done so far with the CLI is also enabled through the Python library. When installing the CLI, the library is automatically installed as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2024.04.25'"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import eotdl\n",
    "\n",
    "eotdl.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "53"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from eotdl.datasets import retrieve_datasets\n",
    "\n",
    "datasets = retrieve_datasets()\n",
    "len(datasets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['EuroSAT-Q1',\n",
       " 'EuroSAT-Q1-small',\n",
       " 'EuroSAT-Q2-small',\n",
       " 'EuroSAT-RGB',\n",
       " 'EuroSAT-RGB-Q1',\n",
       " 'EuroSAT-RGB-Q2',\n",
       " 'EuroSAT-RGB-STAC',\n",
       " 'EuroSAT']"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "retrieve_datasets(\"eurosat\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the library, you have full control over the datasets and models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['EuroSAT-Q1',\n",
       " 'EuroSAT-Q1-small',\n",
       " 'EuroSAT-Q2-small',\n",
       " 'EuroSAT-RGB',\n",
       " 'EuroSAT-RGB-Q1',\n",
       " 'EuroSAT-RGB-Q2',\n",
       " 'EuroSAT-RGB-STAC',\n",
       " 'EuroSAT']"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[d for d in datasets if \"eurosat\" in d.lower()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can download datasets as well, but now you will have to manage potential errors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "ename": "Exception",
     "evalue": "Dataset `EuroSAT-RGB v3` already exists at /home/juan/.cache/eotdl/datasets/EuroSAT-RGB/v3. To force download, use force=True or -f in the CLI.",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mException\u001b[0m                                 Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[29], line 3\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01meotdl\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mdatasets\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m download_dataset\n\u001b[0;32m----> 3\u001b[0m \u001b[43mdownload_dataset\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mEuroSAT-RGB\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/miniconda3/lib/python3.11/site-packages/eotdl/auth/auth.py:61\u001b[0m, in \u001b[0;36mwith_auth.<locals>.wrapper\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m     59\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mwrapper\u001b[39m(\u001b[38;5;241m*\u001b[39margs, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs):\n\u001b[1;32m     60\u001b[0m     user \u001b[38;5;241m=\u001b[39m auth()\n\u001b[0;32m---> 61\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mfunc\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43muser\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43muser\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/miniconda3/lib/python3.11/site-packages/eotdl/datasets/download.py:43\u001b[0m, in \u001b[0;36mdownload_dataset\u001b[0;34m(dataset_name, version, path, logger, assets, force, verbose, user, file)\u001b[0m\n\u001b[1;32m     41\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m os\u001b[38;5;241m.\u001b[39mpath\u001b[38;5;241m.\u001b[39mexists(download_path) \u001b[38;5;129;01mand\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m force:\n\u001b[1;32m     42\u001b[0m     os\u001b[38;5;241m.\u001b[39mmakedirs(download_path, exist_ok\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mTrue\u001b[39;00m)\n\u001b[0;32m---> 43\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mException\u001b[39;00m(\n\u001b[1;32m     44\u001b[0m         \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mDataset `\u001b[39m\u001b[38;5;132;01m{\u001b[39;00mdataset[\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mname\u001b[39m\u001b[38;5;124m'\u001b[39m]\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m v\u001b[39m\u001b[38;5;132;01m{\u001b[39;00m\u001b[38;5;28mstr\u001b[39m(version)\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m` already exists at \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mdownload_path\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m. To force download, use force=True or -f in the CLI.\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m     45\u001b[0m     )\n\u001b[1;32m     46\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m dataset[\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mquality\u001b[39m\u001b[38;5;124m\"\u001b[39m] \u001b[38;5;241m==\u001b[39m \u001b[38;5;241m0\u001b[39m:\n\u001b[1;32m     47\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m file:\n",
      "\u001b[0;31mException\u001b[0m: Dataset `EuroSAT-RGB v3` already exists at /home/juan/.cache/eotdl/datasets/EuroSAT-RGB/v3. To force download, use force=True or -f in the CLI."
     ]
    }
   ],
   "source": [
    "from eotdl.datasets import download_dataset\n",
    "\n",
    "download_dataset(\"EuroSAT-RGB\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 90.3M/90.3M [00:04<00:00, 20.1MiB/s]\n",
      "100%|██████████| 1/1 [00:05<00:00,  5.01s/file]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'/home/juan/.cache/eotdl/datasets/EuroSAT-RGB/v1'"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "download_dataset(\"EuroSAT-RGB\", version=1, force=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 90.3M/90.3M [00:04<00:00, 20.2MiB/s]\n",
      "100%|██████████| 1/1 [00:04<00:00,  4.96s/file]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'data/EuroSAT-RGB/v1'"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "download_dataset(\"EuroSAT-RGB\",  version=1, force=True, path=\"data\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In fact, the CLI is built on top of the library."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And the same for the models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['RoadSegmentation',\n",
       " 'EuroSAT-RGB-BiDS23-Q1',\n",
       " 'MAVERICC',\n",
       " 'WALDO25',\n",
       " 'EuroSAT-RGB-BiDS23']"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from eotdl.models import retrieve_models\n",
    "\n",
    "retrieve_models()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1/1 [00:05<00:00,  5.85s/file]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'/home/juan/.cache/eotdl/models/RoadSegmentation/v1'"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from eotdl.models import download_model \n",
    "\n",
    "path = download_model(\"RoadSegmentation\", force=True)\n",
    "path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['README.md', 'unet-resnet50.onnx']"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os \n",
    "\n",
    "os.listdir(path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Discussion and Contribution opportunities"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Feel free to ask questions now (live or through Discord) and make suggestions for future improvements.\n",
    "\n",
    "\n",
    "- What features would like to see for exploration and downloading?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "eotdl",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}