tutorials/workshops/bids23/01_exploring.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exploring and Downloading Datasets and Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start by exploring the repository of datasets and models. \n",
    "\n",
    "You can do that at the different accessibility layers of EOTDL: the user interface, the API, the command line interface (CLI) and the Python library."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The User Interface"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The easiest way to get started with EOTDL is by exploring the user interface: [https://eotdl.com/](https://www.eotdl.com/). Through the UI you will be able to:\n",
    "\n",
    "- Explore the datasets and models available in the repository (filtering by name, tags and liked)\n",
    "- Edit your own datasets and models information.\n",
    "- Read the tutorials on the blog.\n",
    "- Read the documentation.\n",
    "- Find useful links to other resources (GitHub, Discord, ...)\n",
    "\n",
    "![web](images/web.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Quality levels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Datasets and models in EOTDL are categorized into quality levels. The quality levels are:\n",
    "\n",
    "- **Q0**: datasets in the form of an archive with arbitary files without curation. This level is ideal for easy and fast upload/download of small datasets.\n",
    "- **Q1**: datasets with STAC metadata but no QA. These datasets can leverage a limited set of EOTDL features.\n",
    "- **Q2**: datasets with STAC metadata with the EOTDL custom extensions and automated QA. These datasets can leverage the full potential of the EOTDL.\n",
    "- **Q3**: Q2 datasets that are manually curated. These datasets are the most reliable and can be used as benchmark datasets.\n",
    "\n",
    "You will learn more about the quality levels in the [data curation](05_stac.ipynb) tutorial."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Command Line Interface"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Even though the UI is the easiest way to get started, it is not the most convenient for actually working with the datasets and models. For that we recommend installing the CLI.\n",
    "\n",
    "If you are running this notebook locally, consider creating a virtual environment before installing the CLI to avoid conflicts with other packages.\n",
    "\n",
    "With conda:\n",
    "\n",
    "```bash\n",
    "conda create -n eotdl python=3.8\n",
    "conda activate eotdl\n",
    "```\n",
    "\n",
    "With python: \n",
    "\n",
    "```bash\n",
    "python -m venv eotdl\n",
    "source eotdl/bin/activate\n",
    "```\n",
    "\n",
    "You may also have to install Jupyter on the new environment and restart the notebook.\n",
    "\n",
    "Then, you can install the CLI with pip:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# uncomment to install\n",
    "\n",
    "# !pip install eotdl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once installed, you can execute the CLI with different commands. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m                                                                                \u001b[0m\n",
      "\u001b[1m \u001b[0m\u001b[1;33mUsage: \u001b[0m\u001b[1meotdl [OPTIONS] COMMAND [ARGS]...\u001b[0m\u001b[1m                                      \u001b[0m\u001b[1m \u001b[0m\n",
      "\u001b[1m                                                                                \u001b[0m\n",
      " Welcome to EOTDL. Learn more at https://www.eotdl.com/                         \n",
      "                                                                                \n",
      "\u001b[2m╭─\u001b[0m\u001b[2m Options \u001b[0m\u001b[2m───────────────────────────────────────────────────────────────────\u001b[0m\u001b[2m─╮\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-install\u001b[0m\u001b[1;36m-completion\u001b[0m          Install completion for the current shell.      \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-show\u001b[0m\u001b[1;36m-completion\u001b[0m             Show completion for the current shell, to copy \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m                               it or customize the installation.              \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-help\u001b[0m                        Show this message and exit.                    \u001b[2m│\u001b[0m\n",
      "\u001b[2m╰──────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n",
      "\u001b[2m╭─\u001b[0m\u001b[2m Commands \u001b[0m\u001b[2m──────────────────────────────────────────────────────────────────\u001b[0m\u001b[2m─╮\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36mauth       \u001b[0m\u001b[1;36m \u001b[0m Login to EOTDL.                                                 \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36mdatasets   \u001b[0m\u001b[1;36m \u001b[0m Explore, ingest and download training datasets.                 \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36mmodels     \u001b[0m\u001b[1;36m \u001b[0m Explore, ingest and download ML models.                         \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36mversion    \u001b[0m\u001b[1;36m \u001b[0m Get EOTDL version.                                              \u001b[2m│\u001b[0m\n",
      "\u001b[2m╰──────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!eotdl --help"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EOTDL Version: 2023.11.03-4\n"
     ]
    }
   ],
   "source": [
    "!eotdl version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m                                                                                \u001b[0m\n",
      "\u001b[1m \u001b[0m\u001b[1;33mUsage: \u001b[0m\u001b[1meotdl datasets [OPTIONS] COMMAND [ARGS]...\u001b[0m\u001b[1m                             \u001b[0m\u001b[1m \u001b[0m\n",
      "\u001b[1m                                                                                \u001b[0m\n",
      " Explore, ingest and download training datasets.                                \n",
      "                                                                                \n",
      "\u001b[2m╭─\u001b[0m\u001b[2m Options \u001b[0m\u001b[2m───────────────────────────────────────────────────────────────────\u001b[0m\u001b[2m─╮\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-help\u001b[0m          Show this message and exit.                                  \u001b[2m│\u001b[0m\n",
      "\u001b[2m╰──────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n",
      "\u001b[2m╭─\u001b[0m\u001b[2m Commands \u001b[0m\u001b[2m──────────────────────────────────────────────────────────────────\u001b[0m\u001b[2m─╮\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36mget      \u001b[0m\u001b[1;36m \u001b[0m Download a dataset from the EOTDL.                                \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36mingest   \u001b[0m\u001b[1;36m \u001b[0m Ingest a dataset to the EOTDL.                                    \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36mlist     \u001b[0m\u001b[1;36m \u001b[0m Retrieve a list with all the datasets in the EOTDL.               \u001b[2m│\u001b[0m\n",
      "\u001b[2m╰──────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!eotdl datasets --help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can explore datasets with the following command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['test-bids', 'Boadella-test-bids', 'boadella-dataset', 'EuroSAT-RGB', 'EuroSAT-RGB-Q1', 'EuroSAT-small', 'EuroSAT-RGB-Q2', 'Boadella-BiDS23', 'test-q0', 'COWC', 'Stanford-Drone-dataset', 'EuroSAT-RGB-STAC', 'EuroSAT-STAC', 'BigEarthNet', 'xview2', 'LandcoverAI', 'open-cities-tt2-source', 'open-cities-tt1-source', 'open-cities-test', 'PASTIS-R', 'EuroCrops', 'SloveniaLandCover', 'ISPRS-Potsdam2D', 'SEN12-FLOOD', 'Urban3dChallenge', 'tropical-cyclone-dataset', 'Vessel-detection', 'Airplanes-detection', 'S2-SHIPS', 'SpaceNet-7', 'Sentinel-2-Cloud-Mask', 'PASTIS', 'FlodNet', 'SeCo100k', 'SeCo', 'AirbusAircraftDetection', 'AirbusWindTurbinesPatches', 'RoadNet', 'EuroSAT', 'UCMerced']\n"
     ]
    }
   ],
   "source": [
    "!eotdl datasets list "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m                                                                                \u001b[0m\n",
      "\u001b[1m \u001b[0m\u001b[1;33mUsage: \u001b[0m\u001b[1meotdl datasets list [OPTIONS]\u001b[0m\u001b[1m                                          \u001b[0m\u001b[1m \u001b[0m\n",
      "\u001b[1m                                                                                \u001b[0m\n",
      " Retrieve a list with all the datasets in the EOTDL.                            \n",
      " \u001b[2mIf using \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-name\u001b[0m\u001b[2m, it will filter the results by name. If no name is provided, \u001b[0m  \n",
      " \u001b[2mit will return all the datasets.\u001b[0m                                               \n",
      " \u001b[2mIf using \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-limit\u001b[0m\u001b[2m, it will limit the number of results. If no limit is \u001b[0m         \n",
      " \u001b[2mprovided, it will return all the datasets.\u001b[0m                                     \n",
      "                                                                                \n",
      " \u001b[2mExamples\u001b[0m                                                                       \n",
      " \u001b[1;2;36m--------\u001b[0m                                                                       \n",
      " \u001b[2m$ eotdl datasets list\u001b[0m                                                          \n",
      " \u001b[2m$ eotdl datasets list \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-name\u001b[0m\u001b[2m YourModel \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-limit\u001b[0m\u001b[2m 5\u001b[0m                               \n",
      "                                                                                \n",
      "\u001b[2m╭─\u001b[0m\u001b[2m Options \u001b[0m\u001b[2m───────────────────────────────────────────────────────────────────\u001b[0m\u001b[2m─╮\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-name\u001b[0m   \u001b[1;32m-n\u001b[0m      \u001b[1;33mTEXT   \u001b[0m  Filter the returned datasets by name               \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m                           \u001b[2m[default: None]                     \u001b[0m               \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-limit\u001b[0m  \u001b[1;32m-l\u001b[0m      \u001b[1;33mINTEGER\u001b[0m  Limit the number of returned results               \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m                           \u001b[2m[default: None]                     \u001b[0m               \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-help\u001b[0m           \u001b[1;33m       \u001b[0m  Show this message and exit.                        \u001b[2m│\u001b[0m\n",
      "\u001b[2m╰──────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!eotdl datasets list --help"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['EuroSAT-RGB', 'EuroSAT-RGB-Q1', 'EuroSAT-small', 'EuroSAT-RGB-Q2', 'EuroSAT-RGB-STAC', 'EuroSAT-STAC', 'EuroSAT']\n"
     ]
    }
   ],
   "source": [
    "!eotdl datasets list -n eurosat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you may have guessed, you can download a dataset with the following command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████████| 8/8 [00:02<00:00,  3.06file/s]\n",
      "Data available at /home/juan/.cache/eotdl/datasets/EuroSAT-small/v2\n"
     ]
    }
   ],
   "source": [
    "!eotdl datasets get EuroSAT-small"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first time you run the command, you will be asked to login (which will require you to create an account if you haven't already). You can also login with the command"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "On your computer or mobile device navigate to:  https://earthpulse.eu.auth0.com/activate?user_code=CGZB-SPQB\n",
      "Authenticated!\n",
      "- Id Token: eyJhbGciOi...\n",
      "Saved credentials to:  /home/juan/.cache/eotdl/creds.json\n",
      "You are logged in as it@earthpulse.es\n"
     ]
    }
   ],
   "source": [
    "!eotdl auth login"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m                                                                                \u001b[0m\n",
      "\u001b[1m \u001b[0m\u001b[1;33mUsage: \u001b[0m\u001b[1meotdl auth [OPTIONS] COMMAND [ARGS]...\u001b[0m\u001b[1m                                 \u001b[0m\u001b[1m \u001b[0m\n",
      "\u001b[1m                                                                                \u001b[0m\n",
      " Login to EOTDL.                                                                \n",
      "                                                                                \n",
      "\u001b[2m╭─\u001b[0m\u001b[2m Options \u001b[0m\u001b[2m───────────────────────────────────────────────────────────────────\u001b[0m\u001b[2m─╮\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-help\u001b[0m          Show this message and exit.                                  \u001b[2m│\u001b[0m\n",
      "\u001b[2m╰──────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n",
      "\u001b[2m╭─\u001b[0m\u001b[2m Commands \u001b[0m\u001b[2m──────────────────────────────────────────────────────────────────\u001b[0m\u001b[2m─╮\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36mlogin            \u001b[0m\u001b[1;36m \u001b[0m Login to the EOTDL.                                       \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36mlogout           \u001b[0m\u001b[1;36m \u001b[0m Logout from the EOTDL.                                    \u001b[2m│\u001b[0m\n",
      "\u001b[2m╰──────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!eotdl auth --help"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m                                                                                \u001b[0m\n",
      "\u001b[1m \u001b[0m\u001b[1;33mUsage: \u001b[0m\u001b[1meotdl datasets get [OPTIONS] [DATASET]\u001b[0m\u001b[1m                                 \u001b[0m\u001b[1m \u001b[0m\n",
      "\u001b[1m                                                                                \u001b[0m\n",
      " Download a dataset from the EOTDL.                                             \n",
      " \u001b[2mIf using \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-path\u001b[0m\u001b[2m, it will download the dataset to the specified path. If no \u001b[0m    \n",
      " \u001b[2mpath is provided, it will download to ~/.eotdl/datasets.\u001b[0m                       \n",
      " \u001b[2mIf using \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-file\u001b[0m\u001b[2m, it will download the specified file. If no file is provided, \u001b[0m \n",
      " \u001b[2mit will download the entire dataset.\u001b[0m                                           \n",
      " \u001b[2mIf using \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-version\u001b[0m\u001b[2m, it will download the specified version. If no version is \u001b[0m  \n",
      " \u001b[2mprovided, it will download the latest version.\u001b[0m                                 \n",
      " \u001b[2mIf using \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-assets\u001b[0m\u001b[2m when the dataset is STAC, it will also download the STAC \u001b[0m    \n",
      " \u001b[2massets of the dataset. If not provided, it will only download the STAC \u001b[0m        \n",
      " \u001b[2mmetadata.\u001b[0m                                                                      \n",
      " \u001b[2mIf using \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-force\u001b[0m\u001b[2m, it will download the dataset even if the file already \u001b[0m       \n",
      " \u001b[2mexists.\u001b[0m                                                                        \n",
      " \u001b[2mIf using \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-verbose\u001b[0m\u001b[2m, it will print the progress of the download.\u001b[0m                \n",
      "                                                                                \n",
      " \u001b[2mExamples\u001b[0m                                                                       \n",
      " \u001b[1;2;36m--------\u001b[0m                                                                       \n",
      " \u001b[2m$ eotdl dataset get YourDataset\u001b[0m                                                \n",
      " \u001b[2m$ eotdl dataset get YourDataset \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-path\u001b[0m\u001b[2m /path/to/download \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-file\u001b[0m\u001b[2m dataset.zip \u001b[0m   \n",
      " \u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-version\u001b[0m\u001b[2m 1 \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-assets\u001b[0m\u001b[2m True \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-force\u001b[0m\u001b[2m True \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-verbose\u001b[0m\u001b[2m True\u001b[0m                          \n",
      "                                                                                \n",
      "\u001b[2m╭─\u001b[0m\u001b[2m Arguments \u001b[0m\u001b[2m─────────────────────────────────────────────────────────────────\u001b[0m\u001b[2m─╮\u001b[0m\n",
      "\u001b[2m│\u001b[0m   dataset      \u001b[1;2;33m[\u001b[0m\u001b[1;33mDATASET\u001b[0m\u001b[1;2;33m]\u001b[0m  Name of the dataset to download \u001b[2m[default: None]\u001b[0m    \u001b[2m│\u001b[0m\n",
      "\u001b[2m╰──────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n",
      "\u001b[2m╭─\u001b[0m\u001b[2m Options \u001b[0m\u001b[2m───────────────────────────────────────────────────────────────────\u001b[0m\u001b[2m─╮\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-path\u001b[0m     \u001b[1;32m-p\u001b[0m      \u001b[1;33mTEXT   \u001b[0m  Download the dataset to a specific output path   \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m                             \u001b[2m[default: None]                               \u001b[0m   \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-file\u001b[0m     \u001b[1;32m-f\u001b[0m      \u001b[1;33mTEXT   \u001b[0m  Download a specific file from the dataset        \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m                             \u001b[2m[default: None]                          \u001b[0m        \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-version\u001b[0m  \u001b[1;32m-v\u001b[0m      \u001b[1;33mINTEGER\u001b[0m  Dataset version \u001b[2m[default: None]\u001b[0m                  \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-assets\u001b[0m   \u001b[1;32m-a\u001b[0m      \u001b[1;33m       \u001b[0m  Download STAC assets from the dataset            \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-force\u001b[0m    \u001b[1;32m-f\u001b[0m      \u001b[1;33m       \u001b[0m  Force download even if file exists               \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-verbose\u001b[0m          \u001b[1;33m       \u001b[0m  Verbose output. This will print the progress of  \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m                             the download                                     \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-help\u001b[0m             \u001b[1;33m       \u001b[0m  Show this message and exit.                      \u001b[2m│\u001b[0m\n",
      "\u001b[2m╰──────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!eotdl datasets get --help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, datasets will be downloaded to your `$HOME/.cache/eotdl/datasets` folder or the path in the `EOTDL_DOWNLOAD_PATH` environment variable. You can change this with the `--path` argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████████| 8/8 [00:02<00:00,  2.88file/s]\n",
      "Data available at data/EuroSAT-small/v2\n"
     ]
    }
   ],
   "source": [
    "!eotdl datasets get EuroSAT-small -p data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can choose a particular version to download with the `--version` argument. If you don't specify a version, the latest version will be downloaded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset `EuroSAT-small v1` already exists at data/EuroSAT-small/v1. To force download, use force=True or -f in the CLI.\n"
     ]
    }
   ],
   "source": [
    "!eotdl datasets get EuroSAT-small -p data -v 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The version number will be used to create a folder with the same name inside the path you specified. Inside this folder you will find the dataset files.\n",
    "\n",
    "If you try to re-download a datasets, the CLI will complain. You can force a re-download with the `--force` argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset `EuroSAT-small v1` already exists at data/EuroSAT-small/v1. To force download, use force=True or -f in the CLI.\n"
     ]
    }
   ],
   "source": [
    "!eotdl datasets get EuroSAT-small -p data -v 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████████| 7/7 [00:02<00:00,  2.83file/s]\n",
      "Data available at data/EuroSAT-small/v1\n"
     ]
    }
   ],
   "source": [
    "!eotdl datasets get EuroSAT-small -p data -v 1 -f"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For Q1+ datasets, the `get` command will only download the STAC metadata of the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset `EuroSAT-RGB-Q1 v1` already exists at data/EuroSAT-RGB-Q1/v1. To force download, use force=True or -f in the CLI.\n"
     ]
    }
   ],
   "source": [
    "!eotdl datasets get EuroSAT-RGB-Q1 -p data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Inside the metadata you will find the links to all the assets, so you can download them individually (maybe after some filtering or processing using only the metadata). However, you can download all assets with the command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████████| 200/200 [00:31<00:00,  6.41it/s]\n",
      "Data available at data/EuroSAT-RGB-Q1/v2\n"
     ]
    }
   ],
   "source": [
    "!eotdl datasets get EuroSAT-RGB-Q1 -p data -a -f -v 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Working with models is very much the same at this point."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m                                                                                \u001b[0m\n",
      "\u001b[1m \u001b[0m\u001b[1;33mUsage: \u001b[0m\u001b[1meotdl models [OPTIONS] COMMAND [ARGS]...\u001b[0m\u001b[1m                               \u001b[0m\u001b[1m \u001b[0m\n",
      "\u001b[1m                                                                                \u001b[0m\n",
      " Explore, ingest and download ML models.                                        \n",
      "                                                                                \n",
      "\u001b[2m╭─\u001b[0m\u001b[2m Options \u001b[0m\u001b[2m───────────────────────────────────────────────────────────────────\u001b[0m\u001b[2m─╮\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-help\u001b[0m          Show this message and exit.                                  \u001b[2m│\u001b[0m\n",
      "\u001b[2m╰──────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n",
      "\u001b[2m╭─\u001b[0m\u001b[2m Commands \u001b[0m\u001b[2m──────────────────────────────────────────────────────────────────\u001b[0m\u001b[2m─╮\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36mget      \u001b[0m\u001b[1;36m \u001b[0m Download a model from the EOTDL.                                  \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36mingest   \u001b[0m\u001b[1;36m \u001b[0m Ingest a model to the EOTDL.                                      \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36mlist     \u001b[0m\u001b[1;36m \u001b[0m Retrieve a list with all the models in the EOTDL.                 \u001b[2m│\u001b[0m\n",
      "\u001b[2m╰──────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!eotdl models --help"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['EuroSAT-RGB-BiDS23']\n"
     ]
    }
   ],
   "source": [
    "!eotdl models list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m                                                                                \u001b[0m\n",
      "\u001b[1m \u001b[0m\u001b[1;33mUsage: \u001b[0m\u001b[1meotdl models list [OPTIONS]\u001b[0m\u001b[1m                                            \u001b[0m\u001b[1m \u001b[0m\n",
      "\u001b[1m                                                                                \u001b[0m\n",
      " Retrieve a list with all the models in the EOTDL.                              \n",
      " \u001b[2mIf using \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-name\u001b[0m\u001b[2m, it will filter the results by name. If no name is provided, \u001b[0m  \n",
      " \u001b[2mit will return all the models.\u001b[0m                                                 \n",
      " \u001b[2mIf using \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-limit\u001b[0m\u001b[2m, it will limit the number of results. If no limit is \u001b[0m         \n",
      " \u001b[2mprovided, it will return all the models.\u001b[0m                                       \n",
      "                                                                                \n",
      " \u001b[2mExamples\u001b[0m                                                                       \n",
      " \u001b[1;2;36m--------\u001b[0m                                                                       \n",
      " \u001b[2m$ eotdl models list\u001b[0m                                                            \n",
      " \u001b[2m$ eotdl models list \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-name\u001b[0m\u001b[2m YourModel \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-limit\u001b[0m\u001b[2m 5\u001b[0m                                 \n",
      "                                                                                \n",
      "\u001b[2m╭─\u001b[0m\u001b[2m Options \u001b[0m\u001b[2m───────────────────────────────────────────────────────────────────\u001b[0m\u001b[2m─╮\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-name\u001b[0m   \u001b[1;32m-n\u001b[0m      \u001b[1;33mTEXT   \u001b[0m  Filter the returned models by name \u001b[2m[default: None]\u001b[0m \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-limit\u001b[0m  \u001b[1;32m-l\u001b[0m      \u001b[1;33mINTEGER\u001b[0m  Limit the number of returned results               \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m                           \u001b[2m[default: None]                     \u001b[0m               \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-help\u001b[0m           \u001b[1;33m       \u001b[0m  Show this message and exit.                        \u001b[2m│\u001b[0m\n",
      "\u001b[2m╰──────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!eotdl models list --help"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████████| 2/2 [00:03<00:00,  1.94s/file]\n",
      "Data available at /home/juan/.cache/eotdl/models/EuroSAT-RGB-BiDS23/v1\n"
     ]
    }
   ],
   "source": [
    "!eotdl models get EuroSAT-RGB-BiDS23"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m                                                                                \u001b[0m\n",
      "\u001b[1m \u001b[0m\u001b[1;33mUsage: \u001b[0m\u001b[1meotdl models get [OPTIONS] [MODEL]\u001b[0m\u001b[1m                                     \u001b[0m\u001b[1m \u001b[0m\n",
      "\u001b[1m                                                                                \u001b[0m\n",
      " Download a model from the EOTDL.                                               \n",
      " \u001b[2mIf using \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-path\u001b[0m\u001b[2m, it will download the model to the specified path. If no path \u001b[0m \n",
      " \u001b[2mis provided, it will download to ~/.eotdl/models.\u001b[0m                              \n",
      " \u001b[2mIf using \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-file\u001b[0m\u001b[2m, it will download the specified file. If no file is provided, \u001b[0m \n",
      " \u001b[2mit will download the entire model.\u001b[0m                                             \n",
      " \u001b[2mIf using \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-version\u001b[0m\u001b[2m, it will download the specified version. If no version is \u001b[0m  \n",
      " \u001b[2mprovided, it will download the latest version.\u001b[0m                                 \n",
      " \u001b[2mIf using \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-assets\u001b[0m\u001b[2m when the model is STAC, it will also download the STAC \u001b[0m      \n",
      " \u001b[2massets of the model. If not provided, it will only download the STAC metadata.\u001b[0m \n",
      " \u001b[2mIf using \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-force\u001b[0m\u001b[2m, it will download the model even if the file already exists.\u001b[0m  \n",
      " \u001b[2mIf using \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-verbose\u001b[0m\u001b[2m, it will print the progress of the download.\u001b[0m                \n",
      "                                                                                \n",
      " \u001b[2mExamples\u001b[0m                                                                       \n",
      " \u001b[1;2;36m--------\u001b[0m                                                                       \n",
      " \u001b[2m$ eotdl models get YourModel\u001b[0m                                                   \n",
      " \u001b[2m$ eotdl models get YourModel \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-path\u001b[0m\u001b[2m /path/to/download \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-file\u001b[0m\u001b[2m model.zip \u001b[0m        \n",
      " \u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-version\u001b[0m\u001b[2m 1 \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-assets\u001b[0m\u001b[2m True \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-force\u001b[0m\u001b[2m True \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-verbose\u001b[0m\u001b[2m True\u001b[0m                          \n",
      "                                                                                \n",
      "\u001b[2m╭─\u001b[0m\u001b[2m Arguments \u001b[0m\u001b[2m─────────────────────────────────────────────────────────────────\u001b[0m\u001b[2m─╮\u001b[0m\n",
      "\u001b[2m│\u001b[0m   model      \u001b[1;2;33m[\u001b[0m\u001b[1;33mMODEL\u001b[0m\u001b[1;2;33m]\u001b[0m  Name of the model to download \u001b[2m[default: None]\u001b[0m          \u001b[2m│\u001b[0m\n",
      "\u001b[2m╰──────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n",
      "\u001b[2m╭─\u001b[0m\u001b[2m Options \u001b[0m\u001b[2m───────────────────────────────────────────────────────────────────\u001b[0m\u001b[2m─╮\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-path\u001b[0m     \u001b[1;32m-p\u001b[0m      \u001b[1;33mTEXT   \u001b[0m  Download the model to a specific output path     \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m                             \u001b[2m[default: None]                             \u001b[0m     \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-file\u001b[0m     \u001b[1;32m-f\u001b[0m      \u001b[1;33mTEXT   \u001b[0m  Download a specific file from the model          \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m                             \u001b[2m[default: None]                        \u001b[0m          \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-version\u001b[0m  \u001b[1;32m-v\u001b[0m      \u001b[1;33mINTEGER\u001b[0m  Model version \u001b[2m[default: None]\u001b[0m                    \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-assets\u001b[0m   \u001b[1;32m-a\u001b[0m      \u001b[1;33m       \u001b[0m  Download STAC assets from the model              \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-force\u001b[0m    \u001b[1;32m-f\u001b[0m      \u001b[1;33m       \u001b[0m  Force download even if file exists               \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-verbose\u001b[0m          \u001b[1;33m       \u001b[0m  Verbose output. This will print the progress of  \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m                             the download                                     \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-help\u001b[0m             \u001b[1;33m       \u001b[0m  Show this message and exit.                      \u001b[2m│\u001b[0m\n",
      "\u001b[2m╰──────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!eotdl models get --help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will explore how to ingest datasets and models in the next tutorials."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Library"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Everything that we have done so far with the CLI is also enabled through the Python library. When installing the CLI, the library is automatically installed as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2023.11.03-4'"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import eotdl\n",
    "\n",
    "eotdl.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "38"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from eotdl.datasets import retrieve_datasets\n",
    "\n",
    "datasets = retrieve_datasets()\n",
    "len(datasets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['EuroSAT-RGB',\n",
       " 'EuroSAT-RGB-Q1',\n",
       " 'EuroSAT-small',\n",
       " 'EuroSAT-RGB-Q2',\n",
       " 'EuroSAT-RGB-STAC',\n",
       " 'EuroSAT-STAC',\n",
       " 'EuroSAT']"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "retrieve_datasets(\"eurosat\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the library, you have full control over the datasets and models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['EuroSAT-RGB',\n",
       " 'EuroSAT-RGB-Q1',\n",
       " 'EuroSAT-small',\n",
       " 'EuroSAT-RGB-Q2',\n",
       " 'EuroSAT-RGB-STAC',\n",
       " 'EuroSAT-STAC',\n",
       " 'EuroSAT']"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[d for d in datasets if \"eurosat\" in d.lower()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can download datasets as well, but now you will have to manage potential errors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "ename": "Exception",
     "evalue": "Dataset `EuroSAT-small v2` already exists at /home/juan/.cache/eotdl/datasets/EuroSAT-small/v2. To force download, use force=True or -f in the CLI.",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mException\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m/home/juan/Desktop/eotdl/tutorials/workshops/bids23/01_exploring.ipynb Cell 50\u001b[0m line \u001b[0;36m3\n\u001b[1;32m      <a href='vscode-notebook-cell://ssh-remote%2Bharley/home/juan/Desktop/eotdl/tutorials/workshops/bids23/01_exploring.ipynb#Y100sdnNjb2RlLXJlbW90ZQ%3D%3D?line=0'>1</a>\u001b[0m \u001b[39mfrom\u001b[39;00m \u001b[39meotdl\u001b[39;00m\u001b[39m.\u001b[39;00m\u001b[39mdatasets\u001b[39;00m \u001b[39mimport\u001b[39;00m download_dataset\n\u001b[0;32m----> <a href='vscode-notebook-cell://ssh-remote%2Bharley/home/juan/Desktop/eotdl/tutorials/workshops/bids23/01_exploring.ipynb#Y100sdnNjb2RlLXJlbW90ZQ%3D%3D?line=2'>3</a>\u001b[0m download_dataset(\u001b[39m\"\u001b[39;49m\u001b[39mEuroSAT-small\u001b[39;49m\u001b[39m\"\u001b[39;49m)\n",
      "File \u001b[0;32m~/miniconda3/envs/eotdl/lib/python3.8/site-packages/eotdl/auth/auth.py:47\u001b[0m, in \u001b[0;36mwith_auth.<locals>.wrapper\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m     45\u001b[0m \u001b[39mdef\u001b[39;00m \u001b[39mwrapper\u001b[39m(\u001b[39m*\u001b[39margs, \u001b[39m*\u001b[39m\u001b[39m*\u001b[39mkwargs):\n\u001b[1;32m     46\u001b[0m     user \u001b[39m=\u001b[39m auth()\n\u001b[0;32m---> 47\u001b[0m     \u001b[39mreturn\u001b[39;00m func(\u001b[39m*\u001b[39;49margs, \u001b[39m*\u001b[39;49m\u001b[39m*\u001b[39;49mkwargs, user\u001b[39m=\u001b[39;49muser)\n",
      "File \u001b[0;32m~/miniconda3/envs/eotdl/lib/python3.8/site-packages/eotdl/datasets/download.py:42\u001b[0m, in \u001b[0;36mdownload_dataset\u001b[0;34m(dataset_name, version, path, logger, assets, force, verbose, user, file)\u001b[0m\n\u001b[1;32m     40\u001b[0m \u001b[39mif\u001b[39;00m os\u001b[39m.\u001b[39mpath\u001b[39m.\u001b[39mexists(download_path) \u001b[39mand\u001b[39;00m \u001b[39mnot\u001b[39;00m force:\n\u001b[1;32m     41\u001b[0m     os\u001b[39m.\u001b[39mmakedirs(download_path, exist_ok\u001b[39m=\u001b[39m\u001b[39mTrue\u001b[39;00m)\n\u001b[0;32m---> 42\u001b[0m     \u001b[39mraise\u001b[39;00m \u001b[39mException\u001b[39;00m(\n\u001b[1;32m     43\u001b[0m         \u001b[39mf\u001b[39m\u001b[39m\"\u001b[39m\u001b[39mDataset `\u001b[39m\u001b[39m{\u001b[39;00mdataset[\u001b[39m'\u001b[39m\u001b[39mname\u001b[39m\u001b[39m'\u001b[39m]\u001b[39m}\u001b[39;00m\u001b[39m v\u001b[39m\u001b[39m{\u001b[39;00m\u001b[39mstr\u001b[39m(version)\u001b[39m}\u001b[39;00m\u001b[39m` already exists at \u001b[39m\u001b[39m{\u001b[39;00mdownload_path\u001b[39m}\u001b[39;00m\u001b[39m. To force download, use force=True or -f in the CLI.\u001b[39m\u001b[39m\"\u001b[39m\n\u001b[1;32m     44\u001b[0m     )\n\u001b[1;32m     45\u001b[0m \u001b[39mif\u001b[39;00m dataset[\u001b[39m\"\u001b[39m\u001b[39mquality\u001b[39m\u001b[39m\"\u001b[39m] \u001b[39m==\u001b[39m \u001b[39m0\u001b[39m:\n\u001b[1;32m     46\u001b[0m     \u001b[39mif\u001b[39;00m file:\n",
      "\u001b[0;31mException\u001b[0m: Dataset `EuroSAT-small v2` already exists at /home/juan/.cache/eotdl/datasets/EuroSAT-small/v2. To force download, use force=True or -f in the CLI."
     ]
    }
   ],
   "source": [
    "from eotdl.datasets import download_dataset\n",
    "\n",
    "download_dataset(\"EuroSAT-small\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 7/7 [00:02<00:00,  2.51file/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'/home/juan/.cache/eotdl/datasets/EuroSAT-small/v10'"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "download_dataset(\"EuroSAT-small\", force=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 7/7 [00:02<00:00,  2.77file/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'data/EuroSAT-small/v10'"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "download_dataset(\"EuroSAT-small\", force=True, path=\"data\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In fact, the CLI is built on top of the library."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And the same for the models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['EuroSAT-RGB-BiDS23']"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from eotdl.models import retrieve_models\n",
    "\n",
    "retrieve_models()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 2/2 [00:04<00:00,  2.01s/file]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'/home/juan/.cache/eotdl/models/EuroSAT-RGB-BiDS23/v3'"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from eotdl.models import download_model \n",
    "\n",
    "path = download_model(\"EuroSAT-RGB-BiDS23\", force=True)\n",
    "path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['metadata.yml', 'model.onnx']"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os \n",
    "\n",
    "os.listdir(path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Application Programming Interface"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The last way to interact with EOTDL is using the API. You can explore the interactive documentation at [https://api.eotdl.com/docs](https://api.eotdl.com/docs)\n",
    "\n",
    "You can get the full list of datasets hosted in the EOTDL with the followgin API call:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'uid': 'auth0|616b0057af0c7500691a026e',\n",
       "  'id': '6454b4ba05740a8762edfcdb',\n",
       "  'name': 'EuroSAT-RGB',\n",
       "  'authors': [' Patrick Helber'],\n",
       "  'source': 'http://madm.dfki.de/downloads',\n",
       "  'license': '-',\n",
       "  'files': '6526972d7d4d50bd035d033d',\n",
       "  'versions': [{'version_id': 1,\n",
       "    'createdAt': '2023-10-11T14:37:38.155',\n",
       "    'size': 377122268},\n",
       "   {'version_id': 2, 'createdAt': '2023-10-11T15:38:45.833', 'size': 0},\n",
       "   {'version_id': 3, 'createdAt': '2023-10-11T15:38:45.833', 'size': 0},\n",
       "   {'version_id': 4, 'createdAt': '2023-10-11T15:38:45.833', 'size': 0},\n",
       "   {'version_id': 5, 'createdAt': '2023-10-11T15:38:45.833', 'size': 0},\n",
       "   {'version_id': 6, 'createdAt': '2023-10-11T15:38:45.833', 'size': 0},\n",
       "   {'version_id': 7, 'createdAt': '2023-10-11T15:38:45.833', 'size': 0},\n",
       "   {'version_id': 8, 'createdAt': '2023-10-12T07:14:16.642', 'size': 5406191},\n",
       "   {'version_id': 9, 'createdAt': '2023-10-12T07:14:16.642', 'size': 5610422},\n",
       "   {'version_id': 10,\n",
       "    'createdAt': '2023-10-12T07:14:16.642',\n",
       "    'size': 13106283},\n",
       "   {'version_id': 11,\n",
       "    'createdAt': '2023-10-12T07:14:16.642',\n",
       "    'size': 13031233},\n",
       "   {'version_id': 12,\n",
       "    'createdAt': '2023-10-12T07:14:16.642',\n",
       "    'size': 15113295},\n",
       "   {'version_id': 13,\n",
       "    'createdAt': '2023-10-12T07:14:16.642',\n",
       "    'size': 13494311}],\n",
       "  'description': '<p><strong>EuroSAT: A land use and land cover classification dataset based on Sentinel-2 satellite images.</strong></p><p><br></p><p>This is the RGB version of <a href=\"https://www.eotdl.com/datasets/EuroSAT\" rel=\"noopener noreferrer\" target=\"_blank\">EuroSAT</a>.</p><p><br></p><p><a href=\"https://arxiv.org/abs/1709.00029\" rel=\"noopener noreferrer\" target=\"_blank\">Paper</a></p><p><a href=\"http://madm.dfki.de/downloads\" rel=\"noopener noreferrer\" target=\"_blank\">Alternative download link</a></p><p><br></p><p><span style=\"color: rgb(0, 0, 0);\">Land use and land cover classification using Sentinel-2 satellite images. </span></p><p><br></p><p><span style=\"color: rgb(0, 0, 0);\">The Sentinel-2 satellite images are openly and freely accessible provided in the Earth observation program Copernicus. We present a novel dataset based on Sentinel-2 satellite images covering 13 spectral bands and consisting out of 10 classes with in total 27,000 labeled and geo-referenced images. </span></p><p><br></p><p><br></p>',\n",
       "  'tags': ['image classification', 'land cover', 'land use', 'sentinel-2'],\n",
       "  'createdAt': '2023-05-03T15:46:15.491',\n",
       "  'updatedAt': '2023-10-25T16:19:16.611',\n",
       "  'likes': 1,\n",
       "  'downloads': 40,\n",
       "  'quality': 0},\n",
       " {'uid': 'auth0|616b0057af0c7500691a026e',\n",
       "  'id': '645a26564d1c8b7b364ee631',\n",
       "  'name': 'UCMerced',\n",
       "  'authors': ['Yi Yang and Shawn Newsam'],\n",
       "  'source': 'http://weegee.vision.ucmerced.edu/datasets/landuse.html',\n",
       "  'license': '-',\n",
       "  'files': '6526972d7d4d50bd035d033e',\n",
       "  'versions': [{'version_id': 1,\n",
       "    'createdAt': '2023-10-11T14:37:38.155',\n",
       "    'size': 94280567}],\n",
       "  'description': '<h1>UC Merced Land Use Dataset</h1><p>*info extracted from the <a href=\"http://weegee.vision.ucmerced.edu/datasets/landuse.html\" rel=\"noopener noreferrer\" target=\"_blank\">official page</a>.</p><p><br></p><p>This is a 21 class land use image dataset meant for research purposes.</p><p>There are 100 images for each of the following classes:</p><ul><li>agricultural</li><li>airplane</li><li>baseballdiamond</li><li>beach</li><li>buildings</li><li>chaparral</li><li>denseresidential</li><li>forest</li><li>freeway</li><li>golfcourse</li><li>harbor</li><li>intersection</li><li>mediumresidential</li><li>mobilehomepark</li><li>overpass</li><li>parkinglot</li><li>river</li><li>runway</li><li>sparseresidential</li><li>storagetanks</li><li>tenniscourt</li></ul><p>Each image measures 256x256 pixels.</p><p><br></p><p>The images were manually extracted from large images from the USGS National Map Urban Area Imagery collection for various urban areas around the country. The pixel resolution of this public domain imagery is 1 foot.</p><p><br></p><p>Please cite the following paper when publishing results that use this dataset:</p><p><br></p><p><em>Yi Yang and Shawn Newsam, \"Bag-Of-Visual-Words and Spatial Extensions for Land-Use Classification,\" ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems (ACM GIS), 2010.</em></p><p><br></p><p>Shawn D. Newsam</p><p>Assistant Professor and Founding Faculty</p><p>Electrical Engineering &amp; Computer Science</p><p>University of California, Merced</p><p><br></p><p>Email: snewsam@ucmerced.edu</p><p><br></p><p>Web: http://faculty.ucmerced.edu/snewsam</p><p><br></p><p>This material is based upon work supported by the National Science Foundation under Grant No.&nbsp;<a href=\"http://nsf.gov/awardsearch/showAward.do?AwardNumber=0917069\" rel=\"noopener noreferrer\" target=\"_blank\">0917069</a>.</p>',\n",
       "  'tags': ['sentinel-2', 'land use', 'image classification'],\n",
       "  'createdAt': '2023-05-09T10:52:06.487',\n",
       "  'updatedAt': '2023-06-08T17:30:07.36',\n",
       "  'likes': 1,\n",
       "  'downloads': 9,\n",
       "  'quality': 0},\n",
       " {'uid': 'auth0|616b0057af0c7500691a026e',\n",
       "  'id': '645d1e4ec3060c653291ce87',\n",
       "  'name': 'EuroSAT',\n",
       "  'authors': [' Patrick Helber'],\n",
       "  'source': 'http://madm.dfki.de/downloads',\n",
       "  'license': '-',\n",
       "  'files': '652697317d4d50bd035d033f',\n",
       "  'versions': [{'version_id': 1,\n",
       "    'createdAt': '2023-10-11T14:37:38.155',\n",
       "    'size': 2067725275}],\n",
       "  'description': '<h1><strong style=\"color: rgb(31, 41, 55);\">EuroSAT: A land use and land cover classification dataset based on Sentinel-2 satellite images.</strong></h1><p><br></p><p><a href=\"https://arxiv.org/abs/1709.00029\" rel=\"noopener noreferrer\" target=\"_blank\">Paper</a></p><p><br></p><p><span style=\"color: rgb(0, 0, 0);\">Land use and land cover classification using Sentinel-2 satellite images. </span></p><p><br></p><p><span style=\"color: rgb(0, 0, 0);\">The Sentinel-2 satellite images are openly and freely accessible provided in the Earth observation program Copernicus. We present a novel dataset based on Sentinel-2 satellite images covering 13 spectral bands and consisting out of 10 classes with in total 27,000 labeled and geo-referenced images. </span></p>',\n",
       "  'tags': ['land cover', 'land use', 'sentinel-2', 'image classification'],\n",
       "  'createdAt': '2023-05-11T18:10:15.405',\n",
       "  'updatedAt': '2023-05-23T18:00:20.776',\n",
       "  'likes': 1,\n",
       "  'downloads': 13,\n",
       "  'quality': 0},\n",
       " {'uid': 'auth0|616b0057af0c7500691a026e',\n",
       "  'id': '645dff70c2f02fdd88c47b67',\n",
       "  'name': 'SeCo100k',\n",
       "  'authors': ['ServiceNow'],\n",
       "  'source': 'https://github.com/ServiceNow/seasonal-contrast',\n",
       "  'license': '-',\n",
       "  'files': '6526973d7d4d50bd035d0340',\n",
       "  'versions': [{'version_id': 1,\n",
       "    'createdAt': '2023-10-11T14:37:38.155',\n",
       "    'size': 7302001636}],\n",
       "  'description': '<h1>Seasonal Contrast: Unsupervised Pre-Training from Uncurated Remote Sensing Data</h1><p><br></p><p>This is the small version of <a href=\"https://www.eotdl.com/datasets/SeCo\" rel=\"noopener noreferrer\" target=\"_blank\">SeCo</a>.</p><p><br></p><p><a href=\"https://arxiv.org/abs/2103.16607\" rel=\"noopener noreferrer\" target=\"_blank\">Paper</a></p><p><a href=\"https://github.com/ServiceNow/seasonal-contrast\" rel=\"noopener noreferrer\" target=\"_blank\">Github</a></p><p><br></p><p><span style=\"color: rgb(0, 0, 0);\">Remote sensing and automatic earth monitoring are key to solve global-scale challenges such as disaster prevention, land use monitoring, or tackling climate change. Although there exist vast amounts of remote sensing data, most of it remains unlabeled and thus inaccessible for supervised learning algorithms. Transfer learning approaches can reduce the data requirements of deep learning algorithms. However, most of these methods are pre-trained on ImageNet and their generalization to remote sensing imagery is not guaranteed due to the domain gap. In this work, we propose Seasonal Contrast (SeCo), an effective pipeline to leverage unlabeled data for in-domain pre-training of remote sensing representations. The SeCo pipeline is composed of two parts. First, a principled procedure to gather large-scale, unlabeled and uncurated remote sensing datasets containing images from multiple Earth locations at different timestamps. Second, a self-supervised algorithm that takes advantage of time and position invariance to learn transferable representations for remote sensing applications. We empirically show that models trained with SeCo achieve better performance than their ImageNet pre-trained counterparts and state-of-the-art self-supervised learning methods on multiple downstream tasks. The datasets and models in SeCo will be made public to facilitate transfer learning and enable rapid progress in remote sensing applications.</span></p>',\n",
       "  'tags': ['unsupervised learning', 'sentinel-2'],\n",
       "  'createdAt': '2023-05-11T19:15:20.586',\n",
       "  'updatedAt': '2023-05-23T18:02:08.795',\n",
       "  'likes': 1,\n",
       "  'downloads': 2,\n",
       "  'quality': 0},\n",
       " {'uid': 'auth0|616b0057af0c7500691a026e',\n",
       "  'id': '645e0fb2c2f02fdd88c47b70',\n",
       "  'name': 'SeCo',\n",
       "  'authors': ['ServiceNow'],\n",
       "  'source': 'https://github.com/ServiceNow/seasonal-contrast',\n",
       "  'license': '-',\n",
       "  'files': '6526977c7d4d50bd035d0341',\n",
       "  'versions': [{'version_id': 1,\n",
       "    'createdAt': '2023-10-11T14:37:38.155',\n",
       "    'size': 36325857720}],\n",
       "  'description': '<h1>Seasonal Contrast: Unsupervised Pre-Training from Uncurated Remote Sensing Data</h1><p><br></p><p><a href=\"https://arxiv.org/abs/2103.16607\" rel=\"noopener noreferrer\" target=\"_blank\">Paper</a></p><p><a href=\"https://github.com/ServiceNow/seasonal-contrast\" rel=\"noopener noreferrer\" target=\"_blank\">Github</a></p><p><br></p><p><span style=\"color: rgb(0, 0, 0);\">Remote sensing and automatic earth monitoring are key to solve global-scale challenges such as disaster prevention, land use monitoring, or tackling climate change. Although there exist vast amounts of remote sensing data, most of it remains unlabeled and thus inaccessible for supervised learning algorithms. Transfer learning approaches can reduce the data requirements of deep learning algorithms. However, most of these methods are pre-trained on ImageNet and their generalization to remote sensing imagery is not guaranteed due to the domain gap. In this work, we propose Seasonal Contrast (SeCo), an effective pipeline to leverage unlabeled data for in-domain pre-training of remote sensing representations. The SeCo pipeline is composed of two parts. First, a principled procedure to gather large-scale, unlabeled and uncurated remote sensing datasets containing images from multiple Earth locations at different timestamps. Second, a self-supervised algorithm that takes advantage of time and position invariance to learn transferable representations for remote sensing applications. We empirically show that models trained with SeCo achieve better performance than their ImageNet pre-trained counterparts and state-of-the-art self-supervised learning methods on multiple downstream tasks. The datasets and models in SeCo will be made public to facilitate transfer learning and enable rapid progress in remote sensing applications.</span></p>',\n",
       "  'tags': ['unsupervised learning', 'sentinel-2'],\n",
       "  'createdAt': '2023-05-11T19:15:20.586',\n",
       "  'updatedAt': '2023-05-23T18:02:25.652',\n",
       "  'likes': 1,\n",
       "  'downloads': 4,\n",
       "  'quality': 0},\n",
       " {'uid': 'auth0|616b0057af0c7500691a026e',\n",
       "  'id': '645e3b2dc2f02fdd88c47b77',\n",
       "  'name': 'AirbusAircraftDetection',\n",
       "  'authors': ['-'],\n",
       "  'source': '',\n",
       "  'license': '-',\n",
       "  'files': '6526977d7d4d50bd035d0342',\n",
       "  'versions': [{'version_id': 1,\n",
       "    'createdAt': '2023-10-11T14:37:38.155',\n",
       "    'size': 91853048}],\n",
       "  'description': '<p>Dataset extracted from <a href=\"https://www.kaggle.com/datasets/airbusgeo/airbus-aircrafts-sample-dataset\" rel=\"noopener noreferrer\" target=\"_blank\">Kaggle</a>.</p><p><br></p><p><span style=\"color: rgb(60, 64, 67);\">Aircrafts are usually seen on airports. Earth observation satellites like Airbus\\' Pleiades twin satellites acquire pictures of airports all over the world on a regular basis. Deep Learning can be used to detect automatically the number, size and type of aircrafts present on the site. In turn, this can provide information about the activity of any airport.</span></p><p><br></p><p><span style=\"color: rgb(60, 64, 67);\">License: </span>CC BY-NC-SA 4.0</p>',\n",
       "  'tags': ['object detection', 'airbus'],\n",
       "  'createdAt': '2023-05-11T19:15:20.586',\n",
       "  'updatedAt': '2023-05-16T12:20:03.653',\n",
       "  'likes': 1,\n",
       "  'downloads': 5,\n",
       "  'quality': 0},\n",
       " {'uid': 'auth0|616b0057af0c7500691a026e',\n",
       "  'id': '645e3e4fc2f02fdd88c47b7a',\n",
       "  'name': 'AirbusWindTurbinesPatches',\n",
       "  'authors': ['-'],\n",
       "  'source': '',\n",
       "  'license': 'CC BY-NC-SA 4.0',\n",
       "  'files': '6526977f7d4d50bd035d0343',\n",
       "  'versions': [{'version_id': 1,\n",
       "    'createdAt': '2023-10-11T14:37:38.155',\n",
       "    'size': 1083685010}],\n",
       "  'description': '<h3>Information extracted from <a href=\"https://www.kaggle.com/datasets/airbusgeo/airbus-wind-turbines-patches\" rel=\"noopener noreferrer\" target=\"_blank\">Kaggle</a>.</h3><h3><br></h3><p><strong>Context</strong></p><p>Wind turbines make electricity from wind. Wind turns the propeller-like blades of a turbine around a rotor, which spins a generator, which creates electricity. Generating electricity from wind rather than from gas helps fight global warming. More and more states and private companies are investing in renewable energies and building huge wind turbines plants. Knowing where this wind turbines are located is essential and could help in energy production forecast. Deep Learning could help identify wind turbines on satellite image.</p><p><br></p><h3><strong>Content</strong></h3><p>This is very simple dataset which objective is to build a classifier of satellite images extract&nbsp;<strong>with</strong>&nbsp;or&nbsp;<strong>without</strong>&nbsp;wind turbines. Extracts of 128 x 128 pixels with wind turbines are located in a folder called&nbsp;<code style=\"background-color: rgb(241, 243, 244);\">targets</code>&nbsp;; extracts without wind turbines are located in a folder called&nbsp;<code style=\"background-color: rgb(241, 243, 244);\">background</code>. Images are extracted from satellite acquisitions from the Airbus&nbsp;<a href=\"https://www.intelligence-airbusds.com/en/8693-spot-67\" rel=\"noopener noreferrer\" target=\"_blank\" style=\"color: rgb(32, 33, 36);\">SPOT6 and SPOT7 satellites</a>&nbsp;which resolution is 1.5 meters per pixel. So extracts of 128 x 128 pixels represents roughly 192 meters on the ground which is compatible with the typical size of a wind turbine.</p><p><br></p><h3><strong>Acknowledgements</strong></h3><p>This dataset would not exists without the contribution of the Innovation team at Airbus DS GEO S.A. Thank you for all your hard work and the fun during the tagging and hacking sessions.</p><p><br></p><h3><strong>Inspiration</strong></h3><p>Building a classifier is a very common task in Deep Learning. When dealing with imagery, it is also very common to use&nbsp;<strong>convolutions</strong>&nbsp;to create CNN. By replacing the last fully connected layers of the model by 1-d convolutions, it is possible to create what is called a&nbsp;<strong>fully convolutional neural network</strong>&nbsp;a.k.a. FCN. This model can then be used on imagery of any size to build a detector. The technical paper is available here&nbsp;<a href=\"https://arxiv.org/abs/1411.4038\" rel=\"noopener noreferrer\" target=\"_blank\" style=\"color: rgb(32, 33, 36);\">https://arxiv.org/abs/1411.4038</a></p><p><br></p><p>License: CC BY-NC-SA 4.0</p>',\n",
       "  'tags': ['image classification', 'airbus'],\n",
       "  'createdAt': '2023-05-11T19:15:20.586',\n",
       "  'updatedAt': '2023-05-31T13:50:51.915',\n",
       "  'likes': 1,\n",
       "  'downloads': 2,\n",
       "  'quality': 0},\n",
       " {'uid': 'auth0|6461eed50d3e450fa7f48648',\n",
       "  'id': '64633fdfc2f02fdd88c47b85',\n",
       "  'name': 'RoadNet',\n",
       "  'authors': ['Liu, Yahui; Yao, Jian; Lu, Xiaohu; Xia, Menghan; Wang, Xingbo; Liu, Yuan'],\n",
       "  'source': 'https://github.com/yhlleo/RoadNet',\n",
       "  'license': '-',\n",
       "  'files': '652697817d4d50bd035d0344',\n",
       "  'versions': [{'version_id': 1,\n",
       "    'createdAt': '2023-10-11T14:37:38.155',\n",
       "    'size': 948649414}],\n",
       "  'description': '<p><a href=\"https://ieeexplore.ieee.org/document/8506600/figures#figures\" rel=\"noopener noreferrer\" target=\"_blank\">Paper</a></p><p><br></p><p>A multi-task benchmark dataset used for extraction of road networks from VHR remotely sensed images in complex urban scenes.</p>',\n",
       "  'tags': ['land cover', 'segmentation', 'object detection'],\n",
       "  'createdAt': '2023-05-11T19:15:20.586',\n",
       "  'updatedAt': '2023-05-24T14:25:29.963',\n",
       "  'likes': 0,\n",
       "  'downloads': 0,\n",
       "  'quality': 0},\n",
       " {'uid': 'auth0|6461eed50d3e450fa7f48648',\n",
       "  'id': '6463589e59028dfdbee33336',\n",
       "  'name': 'SloveniaLandCover',\n",
       "  'authors': ['Sinergise'],\n",
       "  'source': 'http://eo-learn.sentinel-hub.com/',\n",
       "  'license': '-',\n",
       "  'files': '652697937d4d50bd035d0345',\n",
       "  'versions': [{'version_id': 1,\n",
       "    'createdAt': '2023-10-11T14:37:38.155',\n",
       "    'size': 9957077222}],\n",
       "  'description': '<p>Sample dataset of EOPatches, for the region of Slovenia, for the year 2019. This data can be used in remote sensing applications, such as land cover classification.</p><p><br></p><p><span style=\"color: rgb(0, 0, 0);\">This example dataset will help you get started with Remote Sensing data and analysis in the open-source framework of eo-learn.</span></p><p><br></p>',\n",
       "  'tags': ['sentinel-2', 'land cover', 'segmentation'],\n",
       "  'createdAt': '2023-05-16T12:16:48.985',\n",
       "  'updatedAt': '2023-05-24T14:21:20.939',\n",
       "  'likes': 0,\n",
       "  'downloads': 2,\n",
       "  'quality': 0},\n",
       " {'uid': 'auth0|6461eed50d3e450fa7f48648',\n",
       "  'id': '6463716b59028dfdbee33338',\n",
       "  'name': 'ISPRS-Potsdam2D',\n",
       "  'authors': ['Wuhan University; Lancaster University; University of Twente'],\n",
       "  'source': 'https://opendatalab.com/ISPRS_Potsdam/download',\n",
       "  'license': '-',\n",
       "  'files': '652697aa7d4d50bd035d0346',\n",
       "  'versions': [{'version_id': 1,\n",
       "    'createdAt': '2023-10-11T14:37:38.155',\n",
       "    'size': 13324219686}],\n",
       "  'description': '<p><a href=\"https://arxiv.org/ftp/arxiv/papers/2109/2109.08937.pdf\" rel=\"noopener noreferrer\" target=\"_blank\">Paper</a></p><p><br></p><p>The dataset contains 38 patches (of the same size), each consisting of a true orthophoto (TOP) extracted from a larger TOP mosaic and a DSM.</p><p><br></p><p>The ground sampling distance of both, the TOP and the DSM, is 5 cm. The DSM was generated via dense image matching with Trimble INPHO 5.6 software and Trimble INPHO OrthoVista was used to generate the TOP mosaic. In order to avoid areas without data (“holes”) in the TOP and the DSM, the patches were selected from the central part of the TOP mosaic and none at the boundaries. Remaining (very small) holes in the TOP and the DSM were interpolated.</p><p>The TOP come as TIFF files in different channel composistions, where each channel has a spectral resolution of 8bit.</p>',\n",
       "  'tags': [],\n",
       "  'createdAt': '2023-05-16T12:16:48.985',\n",
       "  'updatedAt': '2023-05-24T14:19:03.823',\n",
       "  'likes': 0,\n",
       "  'downloads': 1,\n",
       "  'quality': 0},\n",
       " {'uid': 'auth0|642adbfdb3da3ab51492d60a',\n",
       "  'id': '646477ac59028dfdbee33341',\n",
       "  'name': 'SEN12-FLOOD',\n",
       "  'authors': ['Clément Rambour, Nicolas Audebert, Elise Koeniguer, Bertrand Le Saux, Michel Crucianu, Mihai Datcu, September 14, 2020, \"SEN12-FLOOD : a SAR and Multispectral Dataset for Flood Detection \", IEEE Dataport, doi: https://dx.doi.org/10.21227/w6xz-s898.'],\n",
       "  'source': 'https://mlhub.earth/data/sen12floods',\n",
       "  'license': 'Creative Commons Attribution 4.0 International',\n",
       "  'files': '652697bd7d4d50bd035d0347',\n",
       "  'versions': [{'version_id': 1,\n",
       "    'createdAt': '2023-10-11T14:37:38.155',\n",
       "    'size': 10750956154}],\n",
       "  'description': '<p>SEN12-FLOOD is a set of multimodal (SAR + multispectral) satellite image time-series for flood classification. See the paper <a href=\"https://radiantearth.blob.core.windows.net/mlhub/sen12floods/documentation.pdf\" rel=\"noopener noreferrer\" target=\"_blank\">here</a>.</p><p><br></p><p>The observed areas correspond to 337 locations (cities and their surroundings ) in West and SouthEast Africa, Middle-East, and Australia where a flood event occurred during the considered period. The period of acquisition goes from December 2018 to May 2019. </p><p><br></p><p>For each location, the following data are provided: </p><ul><li> Time series of Sentinel-2 multispectral images. These images are composed of 12 bands, at 10m ground-sampling distance and are provided with Level 2A atmospheric correction. </li><li> Time series of Sentinel-1 Synthetic Aperture Radar (SAR) images. The images are provided with radiometric calibration and range doppler terrain correction based on the SRTM digital elevation model. For one acquisition, two raster images are available corresponding to the polarimetry channels VV and VH. </li><li> Time series of binary labels for each image / date: flood or no flood. The original dataset was split into 262 sequences for the train and 68 sequences for the test.</li></ul>',\n",
       "  'tags': ['sentinel-1',\n",
       "   'sentinel-2',\n",
       "   'sar',\n",
       "   'image classification',\n",
       "   'flood detection'],\n",
       "  'createdAt': '2023-05-16T12:16:48.985',\n",
       "  'updatedAt': '2023-05-29T17:35:35.149',\n",
       "  'likes': 0,\n",
       "  'downloads': 14,\n",
       "  'quality': 0},\n",
       " {'uid': 'auth0|6461eed50d3e450fa7f48648',\n",
       "  'id': '6464899459028dfdbee33344',\n",
       "  'name': 'Urban3dChallenge',\n",
       "  'authors': ['Hirsh Goldberg; Myron Brown; Sean Wang'],\n",
       "  'source': 'https://spacenet.ai/the-ussocom-urban-3d-competition/',\n",
       "  'license': '-',\n",
       "  'files': '652697d67d4d50bd035d0348',\n",
       "  'versions': [{'version_id': 1,\n",
       "    'createdAt': '2023-10-11T14:37:38.155',\n",
       "    'size': 13913901752}],\n",
       "  'description': '<p><a href=\"https://ieeexplore.ieee.org/document/8457973\" rel=\"noopener noreferrer\" target=\"_blank\">Paper</a></p><p><br></p><p>This challenge published a large-scale dataset containing 2D orthrorectified RGB and 3D Digital Surface Models and Digital Terrain Models generated from commercial satellite imagery covering over 360 km of terrain and containing roughly 157,000 annotated building footprints. All imagery products are provided at 50 cm ground sample distance (GSD). This unique 2D/3D large scale dataset provides researchers an opportunity to utilize machine learning techniques to further improve state of the art performance.</p>',\n",
       "  'tags': [],\n",
       "  'createdAt': '2023-05-16T12:16:48.985',\n",
       "  'updatedAt': '2023-05-24T14:11:36.119',\n",
       "  'likes': 0,\n",
       "  'downloads': 1,\n",
       "  'quality': 0},\n",
       " {'uid': 'auth0|630cc8fe75e5e824b9e863c6',\n",
       "  'id': '64649f3f59028dfdbee3334a',\n",
       "  'name': 'tropical-cyclone-dataset',\n",
       "  'authors': ['-'],\n",
       "  'source': '',\n",
       "  'license': '-',\n",
       "  'files': '652697da7d4d50bd035d0349',\n",
       "  'versions': [{'version_id': 1,\n",
       "    'createdAt': '2023-10-11T14:37:38.155',\n",
       "    'size': 2369002579}],\n",
       "  'description': 'https://mlhub.earth/data/nasa_tropical_storm_competition',\n",
       "  'tags': [],\n",
       "  'createdAt': '2023-05-16T12:16:48.985',\n",
       "  'updatedAt': '2023-05-16T12:16:48.985',\n",
       "  'likes': 0,\n",
       "  'downloads': 1,\n",
       "  'quality': 0},\n",
       " {'uid': 'auth0|645df677f9c0b75b29963900',\n",
       "  'id': '646640b359028dfdbee3335c',\n",
       "  'name': 'Vessel-detection',\n",
       "  'authors': ['RHEA Group'],\n",
       "  'source': 'https://eodashboard.org/',\n",
       "  'license': 'Dataset copyrighted under the ORCS project license. ',\n",
       "  'files': '652697dc7d4d50bd035d034a',\n",
       "  'versions': [{'version_id': 1,\n",
       "    'createdAt': '2023-10-11T14:37:38.155',\n",
       "    'size': 973498275}],\n",
       "  'description': '<p>Detecting ships by automatically processing Sentinel-2 data (10 m spatial resolution) over well-defined Area Of Interest (AOI) to be used within ESA Euro Data Cube (EDC) infrastructure to support the Earth Observing Dashboard and RACE initiatives. The detection task is performed by means of a deep learning model using a customized Faster R-CNN architecture.</p><p><br></p><p>Learn more: <a href=\"https://eodashboard.org/story?id=shipping\" rel=\"noopener noreferrer\" target=\"_blank\">https://eodashboard.org/story?id=shipping</a></p>',\n",
       "  'tags': ['sentinel-2', 'image classification', 'object detection'],\n",
       "  'createdAt': '2023-05-16T12:16:48.985',\n",
       "  'updatedAt': '2023-06-27T15:21:24.039',\n",
       "  'likes': 0,\n",
       "  'downloads': 8,\n",
       "  'quality': 0},\n",
       " {'uid': 'auth0|645df677f9c0b75b29963900',\n",
       "  'id': '646642d659028dfdbee3335e',\n",
       "  'name': 'Airplanes-detection',\n",
       "  'authors': ['RHEA Group'],\n",
       "  'source': 'https://eodashboard.org/',\n",
       "  'license': 'Dataset copyrighted under the ORCS project license. ',\n",
       "  'files': '652697dc7d4d50bd035d034b',\n",
       "  'versions': [{'version_id': 1,\n",
       "    'createdAt': '2023-10-11T14:37:38.155',\n",
       "    'size': 41500516}],\n",
       "  'description': '<p>Detecting parked airplanes by automatically processing Sentinel-2 data (10 m spatial resolution) over well-defined Area Of Interest (AOI) to be used within ESA Euro Data Cube (EDC) infrastructure to support the Earth Observing Dashboard and RACE initiatives. The detection task is performed by means of a deep learning model using a customized Faster R-CNN architecture.</p><p><br></p><p>Learn more: <a href=\"https://www.eodashboard.org/story?id=airports\" rel=\"noopener noreferrer\" target=\"_blank\">https://www.eodashboard.org/story?id=airports</a></p>',\n",
       "  'tags': ['object detection', 'sentinel-2', 'image classification'],\n",
       "  'createdAt': '2023-05-16T12:16:48.985',\n",
       "  'updatedAt': '2023-06-27T15:29:09.835',\n",
       "  'likes': 0,\n",
       "  'downloads': 2,\n",
       "  'quality': 0},\n",
       " {'uid': 'auth0|6440fc22834ccb24b613fa5a',\n",
       "  'id': '6467305859028dfdbee33362',\n",
       "  'name': 'S2-SHIPS',\n",
       "  'authors': ['Alina Ciocarlan (IMT Atlantique, 29280 Plouzané, France)'],\n",
       "  'source': 'https://github.com/alina2204/contrastive_SSL_ship_detection',\n",
       "  'license': 'MIT license',\n",
       "  'files': '652697e77d4d50bd035d034c',\n",
       "  'versions': [{'version_id': 1,\n",
       "    'createdAt': '2023-10-11T14:37:38.155',\n",
       "    'size': 6223750329}],\n",
       "  'description': '<p>contains the COCO annotations files, the 12 spectral bands for each S2-SHIPS tile in a tif or numpy array version, the S2-SHIPS segmentation masks, the water masks and some pretrained backbones.</p>',\n",
       "  'tags': ['object detection', 'sentinel-2'],\n",
       "  'createdAt': '2023-05-16T12:16:48.985',\n",
       "  'updatedAt': '2023-05-25T14:42:31.993',\n",
       "  'likes': 0,\n",
       "  'downloads': 9,\n",
       "  'quality': 0},\n",
       " {'uid': 'auth0|64672e7ce74dbb9b67fd16e4',\n",
       "  'id': '64674b4459028dfdbee33364',\n",
       "  'name': 'SpaceNet-7',\n",
       "  'authors': ['SpaceNet Partners'],\n",
       "  'source': 'https://spacenet.ai/sn7-challenge/',\n",
       "  'license': 'http://creativecommons.org/licenses/by-sa/4.0/',\n",
       "  'files': '652697f67d4d50bd035d034d',\n",
       "  'versions': [{'version_id': 1,\n",
       "    'createdAt': '2023-10-11T14:37:38.155',\n",
       "    'size': 9165250722}],\n",
       "  'description': '<p><strong>SpaceNet 7</strong> Multi-Temporal Urban Development Challenge dataset (https://medium.com/the-downlinq/the-spacenet-7-multi-temporal-urban-development-challenge-dataset-release-9e6e5f65c8d5, https://spacenet.ai/sn7-challenge/)</p>',\n",
       "  'tags': [],\n",
       "  'createdAt': '2023-05-16T12:16:48.985',\n",
       "  'updatedAt': '2023-05-23T17:36:41.007',\n",
       "  'likes': 1,\n",
       "  'downloads': 36,\n",
       "  'quality': 0},\n",
       " {'uid': 'auth0|642adbfdb3da3ab51492d60a',\n",
       "  'id': '646b3dcd59028dfdbee33372',\n",
       "  'name': 'Sentinel-2-Cloud-Mask',\n",
       "  'authors': ['Francis, Alistair, Mrziglod, John, Sidiropoulos, Panagiotis, & Muller, Jan-Peter. (2020). Sentinel-2 Cloud Mask Catalogue [Data set]. Zenodo. https://doi.org/10.5281/zenodo.4172871'],\n",
       "  'source': 'https://zenodo.org/record/4172871#.ZHTGYexBz0p',\n",
       "  'license': 'Creative Commons Attribution 4.0 International',\n",
       "  'files': '652698117d4d50bd035d034e',\n",
       "  'versions': [{'version_id': 1,\n",
       "    'createdAt': '2023-10-11T14:37:38.155',\n",
       "    'size': 15357118520}],\n",
       "  'description': '<p>This dataset comprises cloud masks for 513 1022-by-1022 pixel subscenes, at 20m resolution, sampled random from the 2018 Level-1C Sentinel-2 archive.</p><p><br></p><p>You can see further information <a href=\"https://zenodo.org/record/4172871#.ZGs4QOxBz0o\" rel=\"noopener noreferrer\" target=\"_blank\">here</a>.</p><p><br></p><p><br></p>',\n",
       "  'tags': ['segmentation', 'sentinel-2'],\n",
       "  'createdAt': '2023-05-16T12:16:48.985',\n",
       "  'updatedAt': '2023-05-29T17:36:29.256',\n",
       "  'likes': 0,\n",
       "  'downloads': 0,\n",
       "  'quality': 0},\n",
       " {'uid': 'auth0|642adbfdb3da3ab51492d60a',\n",
       "  'id': '646b448d59028dfdbee33374',\n",
       "  'name': 'PASTIS',\n",
       "  'authors': ['Sainte Fare Garnot Vivien, & Landrieu Loic. (2021). PASTIS - Panoptic Segmentation of Satellite image TIme Series (1.0) [Data set]. Zenodo. https://doi.org/10.5281/zenodo.5012942'],\n",
       "  'source': 'https://zenodo.org/record/5012942#.ZHTFgOxBz0o',\n",
       "  'license': 'Creative Commons Attribution 4.0 International',\n",
       "  'files': '652698457d4d50bd035d034f',\n",
       "  'versions': [{'version_id': 1,\n",
       "    'createdAt': '2023-10-11T14:37:38.155',\n",
       "    'size': 28760245504}],\n",
       "  'description': '<p>PASTIS (Panoptic Segmentation of Satellite Image Time Series with Convolutional Temporal Attention Networks) is a benchmark dataset for panoptic and semantic segmentation of agricultural parcels from satellite time series. It contains 2,433 patches within the French metropolitan territory with panoptic annotations (instance index + semantic label for each pixel). Each patch is a Sentinel-2 multispectral image time series of variable length. </p><p><br></p><p>This dataset is the original PASTIS dataset for semantic and panoptic segmentation on Sentinel-2 time series.</p><p><br></p><p>You can see further information <a href=\"https://arxiv.org/abs/2107.07933\" rel=\"noopener noreferrer\" target=\"_blank\">here</a>.</p>',\n",
       "  'tags': ['segmentation',\n",
       "   'sentinel-2',\n",
       "   'agriculture',\n",
       "   'land cover',\n",
       "   'image classification'],\n",
       "  'createdAt': '2023-05-16T12:16:48.985',\n",
       "  'updatedAt': '2023-05-29T17:32:37.596',\n",
       "  'likes': 0,\n",
       "  'downloads': 7,\n",
       "  'quality': 0},\n",
       " {'uid': 'auth0|64675bc4a7419507f5c69052',\n",
       "  'id': '646b814559028dfdbee33378',\n",
       "  'name': 'FlodNet',\n",
       "  'authors': ['-'],\n",
       "  'source': '',\n",
       "  'license': '-',\n",
       "  'files': '652698717d4d50bd035d0350',\n",
       "  'versions': [{'version_id': 1,\n",
       "    'createdAt': '2023-10-11T14:37:38.155',\n",
       "    'size': 25001433318}],\n",
       "  'description': '<p>FloodNet: A High Resolution Aerial Imagery Dataset for Post Flood Scene Understanding</p><p><br></p><p>Visual scene understanding is the core task in making any crucial decision in any computer vision system. Although popular computer vision datasets like Cityscapes, MS-COCO, PASCAL provide good benchmarks for several tasks (e.g. image classification, segmentation, object detection), these datasets are hardly suitable for post disaster damage assessments. On the other hand, existing natural disaster datasets include mainly satellite imagery which have low spatial resolution and a high revisit period. Therefore, they do not have a scope to provide quick and efficient damage assessment tasks. Unmanned Aerial Vehicle(UAV) can effortlessly access difficult places during any disaster and collect high resolution imagery that is required for aforementioned tasks of computer vision. To address these issues we present a high resolution UAV imagery, FloodNet, captured after the hurricane Harvey. This dataset demonstrates the post flooded damages of the affected areas. The images are labeled pixel-wise for semantic segmentation task and questions are produced for the task of visual question answering. FloodNet poses several challenges including detection of flooded roads and buildings and distinguishing between natural water and flooded water. With the advancement of deep learning algorithms, we can analyze the impact of any disaster which can make a precise understanding of the affected areas. In this paper, we compare and contrast the performances of baseline methods for image classification, semantic segmentation, and visual question answering on our dataset.</p><p><br></p><p>paper: https://arxiv.org/abs/2012.02951</p><p>dataset: https://github.com/BinaLab/FloodNet-Challenge-EARTHVISION2021</p>',\n",
       "  'tags': [],\n",
       "  'createdAt': '2023-05-16T12:16:48.985',\n",
       "  'updatedAt': '2023-05-22T21:17:54.778',\n",
       "  'likes': 0,\n",
       "  'downloads': 1,\n",
       "  'quality': 0},\n",
       " {'uid': 'auth0|616b0057af0c7500691a026e',\n",
       "  'id': '646cf14046f51deed53f6d5b',\n",
       "  'name': 'EuroCrops',\n",
       "  'authors': ['Schneider, Maja; Chan, Ayshah; Körner, Marco'],\n",
       "  'source': 'https://zenodo.org/record/7851838',\n",
       "  'license': 'CC BY 4.0',\n",
       "  'files': '6526987f7d4d50bd035d0351',\n",
       "  'versions': [{'version_id': 1,\n",
       "    'createdAt': '2023-10-11T14:37:38.155',\n",
       "    'size': 8489703546}],\n",
       "  'description': '<p>Version 7 (Apr 21, 2023)</p><p><br></p><p><strong style=\"color: rgb(51, 51, 51);\">EuroCrops</strong><span style=\"color: rgb(51, 51, 51);\">&nbsp;is a dataset collection combining all publicly available self-declared crop reporting datasets from countries of the European Union.</span></p><p><br></p><p><span style=\"color: rgb(51, 51, 51);\">Learn more: </span><a href=\"https://github.com/maja601/EuroCrops#vectordata\" rel=\"noopener noreferrer\" target=\"_blank\">https://github.com/maja601/EuroCrops#vectordata</a></p>',\n",
       "  'tags': ['segmentation', 'agriculture', 'vector'],\n",
       "  'createdAt': '2023-05-23T15:57:29.413',\n",
       "  'updatedAt': '2023-05-24T11:01:13.8',\n",
       "  'likes': 1,\n",
       "  'downloads': 0,\n",
       "  'quality': 0},\n",
       " {'uid': 'auth0|6462a7c70d3e450fa7f4acde',\n",
       "  'id': '646f55c8f11dc70cb67317df',\n",
       "  'name': 'open-cities-test',\n",
       "  'authors': ['Global Facility for Disaster Reduction and Recovery (GFDRR)'],\n",
       "  'source': 'https://mlhub.earth/data/open_cities_ai_challenge',\n",
       "  'license': 'CC-BY-4.0',\n",
       "  'files': '6526988f7d4d50bd035d0352',\n",
       "  'versions': [{'version_id': 1,\n",
       "    'createdAt': '2023-10-11T14:37:38.155',\n",
       "    'size': 9003672752}],\n",
       "  'description': '<p><span style=\"color: rgb(26, 32, 39);\">Open Cities AI Challenge Test Dataset</span></p><p><br></p><p><span style=\"color: rgb(26, 32, 39);\">This dataset was developed as part of a challenge to segment building footprints from aerial imagery. The goal of the challenge was to accelerate the development of more accurate, relevant, and usable open-source AI models to support mapping for disaster risk management in African cities. The data consists of drone imagery from 10 different cities and regions across Africa.</span></p>',\n",
       "  'tags': [],\n",
       "  'createdAt': '2023-05-23T19:54:36.294',\n",
       "  'updatedAt': '2023-06-01T13:19:42.382',\n",
       "  'likes': 0,\n",
       "  'downloads': 1,\n",
       "  'quality': 0},\n",
       " {'uid': 'auth0|642adbfdb3da3ab51492d60a',\n",
       "  'id': '6474acf3f11dc70cb67317e9',\n",
       "  'name': 'PASTIS-R',\n",
       "  'authors': ['Vivien SAINTE FARE GARNOT, & Loic LANDRIEU. (2021). PASTIS-R - Panoptic Segmentation of Radar and Optical Satellite image TIme Series [Data set]. Zenodo. https://doi.org/10.5281/zenodo.5735646'],\n",
       "  'source': 'https://zenodo.org/record/5735646#.ZHTE7uxBz0o',\n",
       "  'license': 'Creative Commons Attribution 4.0 International',\n",
       "  'files': '652698f27d4d50bd035d0353',\n",
       "  'versions': [{'version_id': 1,\n",
       "    'createdAt': '2023-10-11T14:37:38.155',\n",
       "    'size': 53695489854}],\n",
       "  'description': '<p>PASTIS (Panoptic Segmentation of Satellite Image Time Series with Convolutional Temporal Attention Networks) is a benchmark dataset for panoptic and semantic segmentation of agricultural parcels from satellite time series. It contains 2,433 patches within the French metropolitan territory with panoptic annotations (instance index + semantic label for each pixel). Each patch is a Sentinel-2 multispectral image time series of variable lentgh.</p><p><br></p><p>This dataset is the extended PASTIS dataset with aligned radar Sentinel-1 observations for all 2433 patches in addition to the Sentinel-2 images. For each patch, there have been added approximately 70 observations of Sentinel-1 in ascending orbit, and 70 observations in descending orbit. PASTIS-R can be used to evaluate optical-radar fusion methods for parcel-based classification, semantic segmentation, and panoptic segmentation.</p><p><br></p><p>You can see further information <a href=\"https://arxiv.org/abs/2112.07558v1\" rel=\"noopener noreferrer\" target=\"_blank\">here</a>.</p>',\n",
       "  'tags': ['sar',\n",
       "   'image classification',\n",
       "   'segmentation',\n",
       "   'sentinel-1',\n",
       "   'agriculture',\n",
       "   'land cover',\n",
       "   'sentinel-2'],\n",
       "  'createdAt': '2023-05-23T19:54:36.294',\n",
       "  'updatedAt': '2023-05-29T17:33:09.591',\n",
       "  'likes': 0,\n",
       "  'downloads': 0,\n",
       "  'quality': 0},\n",
       " {'uid': 'auth0|6462a7c70d3e450fa7f4acde',\n",
       "  'id': '64776d6898edc5f751083891',\n",
       "  'name': 'open-cities-tt1-source',\n",
       "  'authors': ['Global Facility for Disaster Reduction and Recovery (GFDRR)'],\n",
       "  'source': 'https://mlhub.earth/data/open_cities_ai_challenge',\n",
       "  'license': 'CC-BY-4.0',\n",
       "  'files': '6526992b7d4d50bd035d0354',\n",
       "  'versions': [{'version_id': 1,\n",
       "    'createdAt': '2023-10-11T14:37:38.155',\n",
       "    'size': 32363731670}],\n",
       "  'description': '<p><span style=\"color: rgb(26, 32, 39);\">Open Cities AI Challenge Train Tier 1 Source Imagery</span></p><p><br></p><p><span style=\"color: rgb(26, 32, 39);\">This dataset was developed as part of a challenge to segment building footprints from aerial imagery. The goal of the challenge was to accelerate the development of more accurate, relevant, and usable open-source AI models to support mapping for disaster risk management in African cities. The data consists of drone imagery from 10 different cities and regions across Africa.</span></p>',\n",
       "  'tags': [],\n",
       "  'createdAt': '2023-05-31T17:06:35.422',\n",
       "  'updatedAt': '2023-06-01T13:19:28.047',\n",
       "  'likes': 0,\n",
       "  'downloads': 0,\n",
       "  'quality': 0},\n",
       " {'uid': 'auth0|6462a7c70d3e450fa7f4acde',\n",
       "  'id': '647860b298edc5f751083893',\n",
       "  'name': 'open-cities-tt2-source',\n",
       "  'authors': ['Global Facility for Disaster Reduction and Recovery (GFDRR)'],\n",
       "  'source': 'https://mlhub.earth/data/open_cities_ai_challenge',\n",
       "  'license': 'CC-BY-4.0',\n",
       "  'files': '652699757d4d50bd035d0355',\n",
       "  'versions': [{'version_id': 1,\n",
       "    'createdAt': '2023-10-11T14:37:38.155',\n",
       "    'size': 41143613634}],\n",
       "  'description': '<p><span style=\"color: rgb(26, 32, 39);\">Open Cities AI Challenge Train Tier 2 Source Imagery</span></p><p><br></p><p><span style=\"color: rgb(26, 32, 39);\"><span class=\"ql-cursor\">\\ufeff</span>This dataset was developed as part of a challenge to segment building footprints from aerial imagery. The goal of the challenge was to accelerate the development of more accurate, relevant, and usable open-source AI models to support mapping for disaster risk management in African cities. The data consists of drone imagery from 10 different cities and regions across Africa.</span></p>',\n",
       "  'tags': [],\n",
       "  'createdAt': '2023-06-01T11:45:16.731',\n",
       "  'updatedAt': '2023-06-01T13:22:04.136',\n",
       "  'likes': 0,\n",
       "  'downloads': 0,\n",
       "  'quality': 0},\n",
       " {'uid': 'auth0|64454081a1131bb5f16ec4e0',\n",
       "  'id': '6487589b24d9f398bfb7bb18',\n",
       "  'name': 'LandcoverAI',\n",
       "  'authors': ['-'],\n",
       "  'source': '',\n",
       "  'license': '-',\n",
       "  'files': '652699787d4d50bd035d0356',\n",
       "  'versions': [{'version_id': 1,\n",
       "    'createdAt': '2023-10-11T14:37:38.155',\n",
       "    'size': 1538212277}],\n",
       "  'description': '',\n",
       "  'tags': [],\n",
       "  'createdAt': '2023-06-09T10:41:01.958',\n",
       "  'updatedAt': '2023-06-09T10:41:01.958',\n",
       "  'likes': 0,\n",
       "  'downloads': 3,\n",
       "  'quality': 0},\n",
       " {'uid': 'auth0|616b0057af0c7500691a026e',\n",
       "  'id': '64888a8e471c3b884b24c022',\n",
       "  'name': 'xview2',\n",
       "  'authors': ['-'],\n",
       "  'source': '',\n",
       "  'license': '-',\n",
       "  'files': '65269a237d4d50bd035d0358',\n",
       "  'versions': [{'version_id': 1,\n",
       "    'createdAt': '2023-10-11T14:37:38.155',\n",
       "    'size': 54882773479}],\n",
       "  'description': '',\n",
       "  'tags': [],\n",
       "  'createdAt': '2023-06-13T16:45:39.852',\n",
       "  'updatedAt': '2023-06-13T16:45:39.852',\n",
       "  'likes': 1,\n",
       "  'downloads': 0,\n",
       "  'quality': 0},\n",
       " {'uid': 'auth0|616b0057af0c7500691a026e',\n",
       "  'id': '64896b59f0d696bce297b36f',\n",
       "  'name': 'BigEarthNet',\n",
       "  'authors': ['Remote Sensing Image Analysis (RSiM) Group',\n",
       "   ' the Database Systems and Information Management (DIMA) Group at the Technische Universität Berlin (TU Berlin)'],\n",
       "  'source': 'https://bigearth.net/',\n",
       "  'license': 'Community Data License Agreement - Permissive - Version 1.0 ',\n",
       "  'files': '65269ba97d4d50bd035d035b',\n",
       "  'versions': [{'version_id': 1,\n",
       "    'createdAt': '2023-10-11T14:37:38.155',\n",
       "    'size': 129301889692}],\n",
       "  'description': '<p><em style=\"color: rgb(33, 37, 41);\">BigEarthNet</em><span style=\"color: rgb(33, 37, 41);\">&nbsp;is a benchmark archive, consisting of&nbsp;</span><strong style=\"color: rgb(50, 107, 52);\">590,326</strong><span style=\"color: rgb(33, 37, 41);\">&nbsp;pairs of&nbsp;</span><strong style=\"color: rgb(50, 107, 52);\">Sentinel-1</strong><span style=\"color: rgb(33, 37, 41);\">&nbsp;and&nbsp;</span><strong style=\"color: rgb(50, 107, 52);\">Sentinel-2</strong><span style=\"color: rgb(33, 37, 41);\">&nbsp;image patches. The first version (v1.0-beta) of&nbsp;</span><em style=\"color: rgb(33, 37, 41);\">BigEarthNet</em><span style=\"color: rgb(33, 37, 41);\">&nbsp;includes only Sentinel 2 images. Recently, it has been enriched by Sentinel-1 images to create a multi-modal&nbsp;</span><em style=\"color: rgb(33, 37, 41);\">BigEarthNet</em><span style=\"color: rgb(33, 37, 41);\">&nbsp;benchmark archive (called also as BigEarthNet-MM).</span></p>',\n",
       "  'tags': ['image classification', 'sentinel-2', 'sentinel-1'],\n",
       "  'createdAt': '2023-06-14T09:18:17.368',\n",
       "  'updatedAt': '2023-07-20T12:10:09.209',\n",
       "  'likes': 1,\n",
       "  'downloads': 4,\n",
       "  'quality': 0},\n",
       " {'uid': 'auth0|642adbfdb3da3ab51492d60a',\n",
       "  'id': '64bfbb6f2a65dcd4ae2ca613',\n",
       "  'name': 'EuroSAT-RGB-STAC',\n",
       "  'authors': ['Patrick Helber'],\n",
       "  'source': 'http://madm.dfki.de/downloads',\n",
       "  'license': 'MIT License',\n",
       "  'files': '65269baa7d4d50bd035d035c',\n",
       "  'versions': [{'version_id': 1,\n",
       "    'createdAt': '2023-10-11T14:37:38.155',\n",
       "    'size': 458654511}],\n",
       "  'description': '<p><strong>EuroSAT: A land use and land cover classification dataset based on Sentinel-2 satellite images.</strong></p><p><br></p><p>This is the RGB version of<strong>&nbsp;</strong><a href=\"https://www.eotdl.com/datasets/EuroSAT\" rel=\"noopener noreferrer\" target=\"_blank\">EuroSAT</a> with STAC metadata.</p><p><br></p><p><a href=\"https://arxiv.org/abs/1709.00029\" rel=\"noopener noreferrer\" target=\"_blank\">Paper</a></p><p><a href=\"http://madm.dfki.de/downloads\" rel=\"noopener noreferrer\" target=\"_blank\">Alternative download link</a></p><p><br></p><p><span style=\"color: rgb(0, 0, 0);\">Land use and land cover classification using Sentinel-2 satellite images.</span></p><p><br></p><p><span style=\"color: rgb(0, 0, 0);\">The Sentinel-2 satellite images are openly and freely accessible provided in the Earth observation program Copernicus. We present a novel dataset based on Sentinel-2 satellite images covering 13 spectral bands and consisting out of 10 classes with in total 27,000 labeled and geo-referenced images.</span></p><p><br></p>',\n",
       "  'tags': ['image classification', 'land cover', 'land use', 'sentinel-2'],\n",
       "  'createdAt': '2023-07-19T13:19:12.136',\n",
       "  'updatedAt': '2023-07-26T16:13:56.092',\n",
       "  'likes': 0,\n",
       "  'downloads': 8,\n",
       "  'quality': 0},\n",
       " {'uid': 'auth0|642adbfdb3da3ab51492d60a',\n",
       "  'id': '64c788902a65dcd4ae2ca629',\n",
       "  'name': 'EuroSAT-STAC',\n",
       "  'authors': ['Patrick Helber'],\n",
       "  'source': 'http://madm.dfki.de/downloads',\n",
       "  'license': 'MIT License',\n",
       "  'files': '65269bb07d4d50bd035d035e',\n",
       "  'versions': [{'version_id': 1,\n",
       "    'createdAt': '2023-10-11T14:37:38.155',\n",
       "    'size': 4249698396}],\n",
       "  'description': '<p><strong>EuroSAT: A land use and land cover classification dataset based on Sentinel-2 satellite images.</strong></p><p><br></p><p><a href=\"https://arxiv.org/abs/1709.00029\" rel=\"noopener noreferrer\" target=\"_blank\">Paper</a></p><p><br></p><p><span style=\"color: rgb(0, 0, 0);\">Land use and land cover classification using Sentinel-2 satellite images with STAC.</span></p><p><br></p><p><span style=\"color: rgb(0, 0, 0);\">The Sentinel-2 satellite images are openly and freely accessible provided in the Earth observation program Copernicus. We present a novel dataset based on Sentinel-2 satellite images covering 13 spectral bands and consisting out of 10 classes with in total 27,000 labeled and geo-referenced images.</span></p>',\n",
       "  'tags': ['land cover', 'sentinel-2', 'image classification', 'land use'],\n",
       "  'createdAt': '2023-07-19T13:19:12.136',\n",
       "  'updatedAt': '2023-09-13T17:23:52.97',\n",
       "  'likes': 0,\n",
       "  'downloads': 8,\n",
       "  'quality': 0},\n",
       " {'uid': 'auth0|642adbfdb3da3ab51492d60a',\n",
       "  'id': '65006c9a751700cadfbd28fb',\n",
       "  'name': 'COWC',\n",
       "  'authors': ['Nathan Mundhenk',\n",
       "   ' Goran Konjevod',\n",
       "   ' Wesam A. Sakla',\n",
       "   ' Kofi Boakye'],\n",
       "  'source': 'https://gdo152.llnl.gov/cowc/',\n",
       "  'license': 'GNU AFFERO GENERAL PUBLIC LICENSE',\n",
       "  'files': '65269c817d4d50bd035d0361',\n",
       "  'versions': [{'version_id': 1,\n",
       "    'createdAt': '2023-10-11T14:37:38.155',\n",
       "    'size': 39935370223}],\n",
       "  'description': '<p><span style=\"color: rgb(0, 0, 0);\">The&nbsp;</span><strong style=\"color: rgb(0, 0, 0);\">Cars Overhead With Context (COWC)&nbsp;</strong><span style=\"color: rgb(0, 0, 0);\">data set is a large set of annotated cars from overhead. It is useful for training a device such as a deep neural network to learn to detect and/or count cars. More information can be obtained by&nbsp;</span><strong style=\"color: rgb(0, 0, 0);\">reading the paper</strong><span style=\"color: rgb(0, 0, 0);\">&nbsp;</span><a href=\"http://gdo-datasci.ucllnl.org/cowc/mundhenk_et_al_eccv_2016.pdf\" rel=\"noopener noreferrer\" target=\"_blank\">here</a><span style=\"color: rgb(0, 0, 0);\">. </span></p><p><br></p><p><span style=\"color: rgb(0, 0, 0);\">The dataset has the following attributes:</span></p><ul><li class=\"ql-align-justify\"><span style=\"background-color: transparent;\">Data from overhead at 15 cm per pixel resolution at ground (all data is EO).&nbsp;</span></li><li class=\"ql-align-justify\"><span style=\"background-color: transparent;\">Data from six distinct locations:&nbsp;</span><a href=\"http://www2.isprs.org/commissions/comm3/wg4/tests.html\" rel=\"noopener noreferrer\" target=\"_blank\" style=\"background-color: transparent;\">Toronto</a><span style=\"background-color: transparent;\">&nbsp;Canada,&nbsp;</span><a href=\"http://www.linz.govt.nz/\" rel=\"noopener noreferrer\" target=\"_blank\" style=\"background-color: transparent;\">Selwyn</a>&nbsp;<span style=\"background-color: transparent;\">New Zealand,&nbsp;</span><a href=\"http://www2.isprs.org/commissions/comm3/wg4/tests.html\" rel=\"noopener noreferrer\" target=\"_blank\" style=\"background-color: transparent;\">Potsdam</a><span style=\"background-color: transparent;\">&nbsp;and&nbsp;</span><a href=\"http://www2.isprs.org/commissions/comm3/wg4/tests.html\" rel=\"noopener noreferrer\" target=\"_blank\" style=\"background-color: transparent;\">Vaihingen</a>&nbsp;<span style=\"background-color: transparent;\">&nbsp;Germany,&nbsp;</span><a href=\"https://www.sdms.afrl.af.mil/index.php?collection=csuav\" rel=\"noopener noreferrer\" target=\"_blank\" style=\"background-color: transparent;\">Columbus</a>&nbsp;Ohio<span style=\"background-color: transparent;\">&nbsp;and&nbsp;</span><a href=\"http://gis.utah.gov/data/aerial-photography/\" rel=\"noopener noreferrer\" target=\"_blank\" style=\"background-color: transparent;\">Utah</a>&nbsp;<span style=\"background-color: transparent;\">United States.&nbsp;</span></li><li class=\"ql-align-justify\"><span style=\"background-color: transparent;\">32,716 unique annotated cars. 58,247 unique negative examples.</span></li><li class=\"ql-align-justify\"><span style=\"background-color: transparent;\">Intentional selection of hard negative examples.</span></li><li class=\"ql-align-justify\"><span style=\"background-color: transparent;\">Established baseline for detection and counting tasks.</span></li><li class=\"ql-align-justify\"><span style=\"background-color: transparent;\">Extra testing scenes for use after validation.</span></li></ul><p><br></p><p><br></p>',\n",
       "  'tags': ['object detection', 'sentinel-2'],\n",
       "  'createdAt': '2023-09-08T11:29:26.627',\n",
       "  'updatedAt': '2023-09-13T16:38:24.989',\n",
       "  'likes': 0,\n",
       "  'downloads': 0,\n",
       "  'quality': 0},\n",
       " {'uid': 'auth0|642adbfdb3da3ab51492d60a',\n",
       "  'id': '6501e112751700cadfbd2904',\n",
       "  'name': 'Stanford-Drone-dataset',\n",
       "  'authors': ['A. Robicquet', ' A. Sadeghian', ' A. Alahi', ' S. Savarese'],\n",
       "  'source': 'https://cvgl.stanford.edu/projects/uav_data/',\n",
       "  'license': 'Creative Commons Attribution-NonCommercial-ShareAlike 3.0 License',\n",
       "  'files': '65269da77d4d50bd035d0364',\n",
       "  'versions': [{'version_id': 1,\n",
       "    'createdAt': '2023-10-11T14:37:38.155',\n",
       "    'size': 71002113639}],\n",
       "  'description': '<p><span style=\"color: rgb(0, 0, 0);\">When humans navigate a crowed space such as a university campus or the sidewalks of a busy street, they follow common sense rules based on social etiquette. In order to enable the design of new algorithms that can fully take advantage of these rules to better solve tasks such as target tracking or trajectory forecasting, it is needed to have access to better data. </span></p><p><br></p><p><span style=\"color: rgb(0, 0, 0);\">To that end, this dataset is a large scale dataset that collects images and videos of various types of agents (not just pedestrians, but also bicyclists, skateboarders, cars, buses, and golf carts) that navigate in a real world outdoor environment such as a university campus. In the above images, pedestrians are labeled in pink, bicyclists in red, skateboarders in orange, and cars in green</span></p>',\n",
       "  'tags': ['object detection'],\n",
       "  'createdAt': '2023-09-08T11:29:26.627',\n",
       "  'updatedAt': '2023-09-13T20:37:09.812',\n",
       "  'likes': 0,\n",
       "  'downloads': 0,\n",
       "  'quality': 0},\n",
       " {'uid': 'auth0|616b0057af0c7500691a026e',\n",
       "  'id': '6503f8a3d05a1b62cc273ea4',\n",
       "  'name': 'eurosat-rgb',\n",
       "  'description': '',\n",
       "  'tags': [],\n",
       "  'createdAt': '2023-09-15T06:10:21.544',\n",
       "  'updatedAt': '2023-09-15T08:42:30.275',\n",
       "  'likes': 0,\n",
       "  'downloads': 4,\n",
       "  'quality': 1,\n",
       "  'size': 0,\n",
       "  'catalog': {'type': 'Catalog',\n",
       "   'id': 'eurosat-rgb',\n",
       "   'stac_version': '1.0.0',\n",
       "   'description': 'EuroSAT-RGB dataset',\n",
       "   'links': [{'rel': 'self',\n",
       "     'href': '/home/juan/Desktop/eotdl/tutorials/data/EuroSAT-STAC/catalog.json',\n",
       "     'type': 'application/json'},\n",
       "    {'rel': 'root', 'href': './catalog.json', 'type': 'application/json'},\n",
       "    {'rel': 'child',\n",
       "     'href': './source/collection.json',\n",
       "     'type': 'application/json'},\n",
       "    {'rel': 'child',\n",
       "     'href': './labels/collection.json',\n",
       "     'type': 'application/json'}],\n",
       "   'extent': None,\n",
       "   'license': None,\n",
       "   'stac_extensions': None,\n",
       "   'summaries': None,\n",
       "   'properties': None,\n",
       "   'assets': None,\n",
       "   'bbox': None,\n",
       "   'collection': None},\n",
       "  'versions': [{'version_id': 1,\n",
       "    'createdAt': '2023-10-11T14:37:38.155',\n",
       "    'size': 1773680}]},\n",
       " {'uid': 'auth0|616b0057af0c7500691a026e',\n",
       "  'id': '6503f994d05a1b62cc273fdd',\n",
       "  'name': 'eurosat-rgb-q2',\n",
       "  'description': '',\n",
       "  'tags': [],\n",
       "  'createdAt': '2023-09-15T06:10:21.544',\n",
       "  'updatedAt': '2023-09-15T08:29:04.656',\n",
       "  'likes': 0,\n",
       "  'downloads': 3,\n",
       "  'quality': 2,\n",
       "  'size': 0,\n",
       "  'catalog': {'type': 'Catalog',\n",
       "   'id': 'eurosat-rgb-q2',\n",
       "   'stac_version': '1.0.0',\n",
       "   'description': 'EuroSAT-RGB dataset',\n",
       "   'links': [{'rel': 'self',\n",
       "     'href': '/home/juan/Desktop/eotdl/tutorials/data/EuroSAT-Q2/catalog.json',\n",
       "     'type': 'application/json'},\n",
       "    {'rel': 'root', 'href': './catalog.json', 'type': 'application/json'},\n",
       "    {'rel': 'child',\n",
       "     'href': './source/collection.json',\n",
       "     'type': 'application/json'},\n",
       "    {'rel': 'child',\n",
       "     'href': './labels/collection.json',\n",
       "     'type': 'application/json'}],\n",
       "   'stac_extensions': ['https://raw.githubusercontent.com/earthpulse/ml-dataset/main/json-schema/schema.json'],\n",
       "   'ml-dataset:name': 'EuroSAT Q2 Dataset',\n",
       "   'ml-dataset:tasks': ['image classification'],\n",
       "   'ml-dataset:inputs-type': ['satellite imagery'],\n",
       "   'ml-dataset:annotations-type': 'raster',\n",
       "   'ml-dataset:version': '0.1.0',\n",
       "   'ml-dataset:splits': ['Training', 'Validation', 'Test'],\n",
       "   'ml-dataset:quality-metrics': [{'name': 'spatial-duplicates',\n",
       "     'values': [{'item': 'Industrial_1273', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_1117', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_1121', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_1641', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_259', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_435', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_674', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_905', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_238', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_631', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_2292', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_1952', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_1980', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_524', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_689', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_1967', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_305', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_1184', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_2193', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_1015', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_1532', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_146', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_962', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_1543', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_1328', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_1721', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_2024', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_1233', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_94', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_910', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_364', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_1096', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_512', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_1291', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_59', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_124', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_1899', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_78', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_448', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_1920', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_1434', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_421', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_125', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_2216', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_265', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_354', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_1327', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_1104', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_1923', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_1312', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_2486', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_239', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_212', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_1978', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_1378', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_1535', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_2037', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_1881', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_1797', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_1739', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_1225', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_665', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_1710', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_1708', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_253', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_1754', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_2359', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_40', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_1491', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_2281', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_2471', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_767', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_496', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_1049', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_807', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_1376', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_2479', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_850', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_1124', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_1955', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_594', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_1469', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_1073', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_299', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_2367', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_153', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_1959', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_1522', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_1694', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_419', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_882', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_1215', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_2062', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_233', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_2063', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_2394', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_412', 'duplicate': 'Industrial_1743'},\n",
       "      {'item': 'Industrial_702', 'duplicate': 'Industrial_1743'}],\n",
       "     'total': 98},\n",
       "    {'name': 'classes-balance',\n",
       "     'values': [{'class': 'Industrial', 'total': 99, 'percentage': 10},\n",
       "      {'class': 'Forest', 'total': 99, 'percentage': 10},\n",
       "      {'class': 'HerbaceousVegetation', 'total': 99, 'percentage': 10},\n",
       "      {'class': 'PermanentCrop', 'total': 99, 'percentage': 10},\n",
       "      {'class': 'Highway', 'total': 99, 'percentage': 10},\n",
       "      {'class': 'Residential', 'total': 99, 'percentage': 10},\n",
       "      {'class': 'SeaLake', 'total': 99, 'percentage': 10},\n",
       "      {'class': 'River', 'total': 99, 'percentage': 10},\n",
       "      {'class': 'AnnualCrop', 'total': 99, 'percentage': 10},\n",
       "      {'class': 'Pasture', 'total': 99, 'percentage': 10}]}],\n",
       "   'extent': None,\n",
       "   'license': None,\n",
       "   'ml-dataset:split-items': None,\n",
       "   'summaries': None,\n",
       "   'properties': None,\n",
       "   'assets': None,\n",
       "   'bbox': None,\n",
       "   'collection': None},\n",
       "  'versions': [{'version_id': 1,\n",
       "    'createdAt': '2023-10-11T14:37:38.155',\n",
       "    'size': 453353}]},\n",
       " {'uid': 'auth0|616b0057af0c7500691a026e',\n",
       "  'id': '6526accffd974011abc2413a',\n",
       "  'name': 'EuroSAT-small',\n",
       "  'authors': ['juan'],\n",
       "  'source': 'http://km.com',\n",
       "  'license': 'open',\n",
       "  'files': '6526accffd974011abc2413b',\n",
       "  'versions': [{'version_id': 1,\n",
       "    'createdAt': '2023-10-11T16:08:47.864',\n",
       "    'size': 643464},\n",
       "   {'version_id': 2, 'createdAt': '2023-10-11T16:08:47.864', 'size': 643464},\n",
       "   {'version_id': 3, 'createdAt': '2023-10-12T07:14:16.642', 'size': 643464},\n",
       "   {'version_id': 4, 'createdAt': '2023-10-12T07:14:16.642', 'size': 643464},\n",
       "   {'version_id': 5, 'createdAt': '2023-10-12T07:14:16.642', 'size': 643464},\n",
       "   {'version_id': 6, 'createdAt': '2023-10-12T07:14:16.642', 'size': 643464},\n",
       "   {'version_id': 7, 'createdAt': '2023-10-12T07:14:16.642', 'size': 0},\n",
       "   {'version_id': 8, 'createdAt': '2023-10-12T07:14:16.642', 'size': 0},\n",
       "   {'version_id': 9, 'createdAt': '2023-11-02T13:11:36.142', 'size': 643562}],\n",
       "  'description': '',\n",
       "  'tags': [],\n",
       "  'createdAt': '2023-10-11T16:08:47.865',\n",
       "  'updatedAt': '2023-11-02T14:37:31.095',\n",
       "  'likes': 0,\n",
       "  'downloads': 0,\n",
       "  'quality': 0},\n",
       " {'uid': 'auth0|645df677f9c0b75b29963900',\n",
       "  'id': '653a31c09171acf769e2731b',\n",
       "  'name': 'test-q0',\n",
       "  'authors': [''],\n",
       "  'source': 'https://www.eotdl.com',\n",
       "  'license': '',\n",
       "  'files': '653a31c09171acf769e2731c',\n",
       "  'versions': [{'version_id': 1,\n",
       "    'createdAt': '2023-10-12T07:14:16.642',\n",
       "    'size': 518196}],\n",
       "  'description': '',\n",
       "  'tags': [],\n",
       "  'createdAt': '2023-10-25T16:08:29.666',\n",
       "  'updatedAt': '2023-10-26T11:31:21.189',\n",
       "  'likes': 0,\n",
       "  'downloads': 0,\n",
       "  'quality': 0}]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import requests\n",
    "\n",
    "datasets = requests.get(\"https://api.eotdl.com/datasets\").json()\n",
    "datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, here you get all the information about the dataset, not only the name (author, license, versions, etc). This is why the API is ideal for building third party applications on top of EOTDL."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('EuroSAT-small', '6526accffd974011abc2413a', '6526accffd974011abc2413b', 9)]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "datasets = requests.get(\"https://api.eotdl.com/datasets?match=eurosat-small&limit=1\").json()\n",
    "[(d['name'], d['id'], d['files'], len(d['versions'])) for d in datasets]\t"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In fact, the library (and CLI) are built on top of the API, so you can achieve the same functionality (or even better!) on your own applications."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "files = requests.get(\"https://api.eotdl.com/datasets/6526accffd974011abc2413a/files?version=2\").json()\n",
    "len(files)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'filename': 'Forest/Forest_3.tif',\n",
       " 'version': 1,\n",
       " 'checksum': '3e7bb982f9db5f7dabc556016c3d081dfb1fb73d'}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "files[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some API calls requires you to be authenticated. You can do that with as follows:\n",
    "\n",
    "- Use the `auth/login` endpoint to get a login URL and a code\n",
    "- Navigate to the login URL to login\n",
    "- Use the `auth/token` endpoint to get a token with the provided code\n",
    "- Use the token to authenticate your requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "ename": "HTTPError",
     "evalue": "401 Client Error: Unauthorized for url: https://api.eotdl.com/datasets/6526accffd974011abc2413a/download/Forest/Forest_3.tif?version=1",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mHTTPError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m/home/juan/Desktop/eotdl/tutorials/workshops/bids23/01_exploring_datasets_and_models..ipynb Cell 53\u001b[0m line \u001b[0;36m1\n\u001b[1;32m      <a href='vscode-notebook-cell://ssh-remote%2Bharley/home/juan/Desktop/eotdl/tutorials/workshops/bids23/01_exploring_datasets_and_models..ipynb#Y144sdnNjb2RlLXJlbW90ZQ%3D%3D?line=8'>9</a>\u001b[0m os\u001b[39m.\u001b[39mmakedirs(os\u001b[39m.\u001b[39mpath\u001b[39m.\u001b[39mdirname(filepath), exist_ok\u001b[39m=\u001b[39m\u001b[39mTrue\u001b[39;00m)\n\u001b[1;32m     <a href='vscode-notebook-cell://ssh-remote%2Bharley/home/juan/Desktop/eotdl/tutorials/workshops/bids23/01_exploring_datasets_and_models..ipynb#Y144sdnNjb2RlLXJlbW90ZQ%3D%3D?line=9'>10</a>\u001b[0m response \u001b[39m=\u001b[39m requests\u001b[39m.\u001b[39mget(\n\u001b[1;32m     <a href='vscode-notebook-cell://ssh-remote%2Bharley/home/juan/Desktop/eotdl/tutorials/workshops/bids23/01_exploring_datasets_and_models..ipynb#Y144sdnNjb2RlLXJlbW90ZQ%3D%3D?line=10'>11</a>\u001b[0m     \u001b[39mf\u001b[39m\u001b[39m'\u001b[39m\u001b[39mhttps://api.eotdl.com/datasets/6526accffd974011abc2413a/download/\u001b[39m\u001b[39m{\u001b[39;00mfilename\u001b[39m}\u001b[39;00m\u001b[39m?version=1\u001b[39m\u001b[39m'\u001b[39m, \n\u001b[1;32m     <a href='vscode-notebook-cell://ssh-remote%2Bharley/home/juan/Desktop/eotdl/tutorials/workshops/bids23/01_exploring_datasets_and_models..ipynb#Y144sdnNjb2RlLXJlbW90ZQ%3D%3D?line=11'>12</a>\u001b[0m     headers\u001b[39m=\u001b[39m{\u001b[39m'\u001b[39m\u001b[39mAuthorization\u001b[39m\u001b[39m'\u001b[39m: \u001b[39mf\u001b[39m\u001b[39m'\u001b[39m\u001b[39mBearer \u001b[39m\u001b[39m{\u001b[39;00mtoken\u001b[39m}\u001b[39;00m\u001b[39m'\u001b[39m},\n\u001b[1;32m     <a href='vscode-notebook-cell://ssh-remote%2Bharley/home/juan/Desktop/eotdl/tutorials/workshops/bids23/01_exploring_datasets_and_models..ipynb#Y144sdnNjb2RlLXJlbW90ZQ%3D%3D?line=12'>13</a>\u001b[0m     stream\u001b[39m=\u001b[39m\u001b[39mTrue\u001b[39;00m\n\u001b[1;32m     <a href='vscode-notebook-cell://ssh-remote%2Bharley/home/juan/Desktop/eotdl/tutorials/workshops/bids23/01_exploring_datasets_and_models..ipynb#Y144sdnNjb2RlLXJlbW90ZQ%3D%3D?line=13'>14</a>\u001b[0m )\n\u001b[0;32m---> <a href='vscode-notebook-cell://ssh-remote%2Bharley/home/juan/Desktop/eotdl/tutorials/workshops/bids23/01_exploring_datasets_and_models..ipynb#Y144sdnNjb2RlLXJlbW90ZQ%3D%3D?line=14'>15</a>\u001b[0m response\u001b[39m.\u001b[39;49mraise_for_status()\n\u001b[1;32m     <a href='vscode-notebook-cell://ssh-remote%2Bharley/home/juan/Desktop/eotdl/tutorials/workshops/bids23/01_exploring_datasets_and_models..ipynb#Y144sdnNjb2RlLXJlbW90ZQ%3D%3D?line=16'>17</a>\u001b[0m \u001b[39mwith\u001b[39;00m \u001b[39mopen\u001b[39m(filepath, \u001b[39m'\u001b[39m\u001b[39mwb\u001b[39m\u001b[39m'\u001b[39m) \u001b[39mas\u001b[39;00m file:\n\u001b[1;32m     <a href='vscode-notebook-cell://ssh-remote%2Bharley/home/juan/Desktop/eotdl/tutorials/workshops/bids23/01_exploring_datasets_and_models..ipynb#Y144sdnNjb2RlLXJlbW90ZQ%3D%3D?line=17'>18</a>\u001b[0m     \u001b[39mfor\u001b[39;00m chunk \u001b[39min\u001b[39;00m response\u001b[39m.\u001b[39miter_content(chunk_size\u001b[39m=\u001b[39m\u001b[39m8192\u001b[39m):\n",
      "File \u001b[0;32m~/miniconda3/envs/eotdl/lib/python3.8/site-packages/requests/models.py:1021\u001b[0m, in \u001b[0;36mResponse.raise_for_status\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m   1016\u001b[0m     http_error_msg \u001b[39m=\u001b[39m (\n\u001b[1;32m   1017\u001b[0m         \u001b[39mf\u001b[39m\u001b[39m\"\u001b[39m\u001b[39m{\u001b[39;00m\u001b[39mself\u001b[39m\u001b[39m.\u001b[39mstatus_code\u001b[39m}\u001b[39;00m\u001b[39m Server Error: \u001b[39m\u001b[39m{\u001b[39;00mreason\u001b[39m}\u001b[39;00m\u001b[39m for url: \u001b[39m\u001b[39m{\u001b[39;00m\u001b[39mself\u001b[39m\u001b[39m.\u001b[39murl\u001b[39m}\u001b[39;00m\u001b[39m\"\u001b[39m\n\u001b[1;32m   1018\u001b[0m     )\n\u001b[1;32m   1020\u001b[0m \u001b[39mif\u001b[39;00m http_error_msg:\n\u001b[0;32m-> 1021\u001b[0m     \u001b[39mraise\u001b[39;00m HTTPError(http_error_msg, response\u001b[39m=\u001b[39m\u001b[39mself\u001b[39m)\n",
      "\u001b[0;31mHTTPError\u001b[0m: 401 Client Error: Unauthorized for url: https://api.eotdl.com/datasets/6526accffd974011abc2413a/download/Forest/Forest_3.tif?version=1"
     ]
    }
   ],
   "source": [
    "import os\n",
    "\n",
    "token = '...'\n",
    "\n",
    "file = files[0]\n",
    "filename = file[\"filename\"]\n",
    "filepath = f'data/{filename}'\n",
    "\n",
    "os.makedirs(os.path.dirname(filepath), exist_ok=True)\n",
    "response = requests.get(\n",
    "    f'https://api.eotdl.com/datasets/6526accffd974011abc2413a/download/{filename}?version=1', \n",
    "    headers={'Authorization': f'Bearer {token}'},\n",
    "    stream=True\n",
    ")\n",
    "response.raise_for_status()\n",
    "\n",
    "with open(filepath, 'wb') as file:\n",
    "    for chunk in response.iter_content(chunk_size=8192):\n",
    "        file.write(chunk)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Roadmap"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are some features we are planning to add in order to enhance the exploration and download experience:\n",
    "\n",
    "- Visual exploration tools in the UI\n",
    "- Geographical and temporal search for Q1+ datasets (bounding box)\n",
    "- Metadata queries for Q1+ datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Discussion and Contribution opportunities"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Feel free to ask questions now (live or through Discord) and make suggestions for future improvements.\n",
    "\n",
    "\n",
    "- What features would like to see for exploration and downloading?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "eotdl",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}