tutorials/notebooks/02_ingesting.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "# import os\n",
    "# os.environ[\"EOTDL_API_URL\"] = \"http://localhost:8000/\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ingesting Datasets and Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook we are going to showcase how to ingest an existing dataset or model into EOTDL.\n",
    "\n",
    "Once it is ingested, you can use it in the same way as any other dataset or model in EOTDL (exploring, staging, etc.)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The recommended way to ingest a dataset is using the CLI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m                                                                                \u001b[0m\n",
      "\u001b[1m \u001b[0m\u001b[1;33mUsage: \u001b[0m\u001b[1meotdl datasets ingest [OPTIONS]\u001b[0m\u001b[1m                                        \u001b[0m\u001b[1m \u001b[0m\n",
      "\u001b[1m                                                                                \u001b[0m\n",
      " Ingest a dataset to the EOTDL.asdf                                             \n",
      " \u001b[2mThis command ingests the dataset to the EOTDL. The dataset must be a folder \u001b[0m   \n",
      " \u001b[2mwith the dataset files, and at least a README.md file (and a catalog.json file\u001b[0m \n",
      " \u001b[2mfor Q1+). If these files are missing, the ingestion will not work. All the \u001b[0m    \n",
      " \u001b[2mfiles in the folder will be uploaded to the EOTDL.\u001b[0m                             \n",
      "                                                                                \n",
      " \u001b[2mThe following constraints apply to the dataset name:\u001b[0m                           \n",
      " \u001b[2m- It must be unique\u001b[0m                                                            \n",
      " \u001b[2m- It must be between 3 and 45 characters long\u001b[0m                                  \n",
      " \u001b[2m- It can only contain alphanumeric characters and dashes.\u001b[0m                      \n",
      "                                                                                \n",
      " \u001b[2mThe README.md file must contain the following fields in the metadata header:\u001b[0m   \n",
      " \u001b[2m- name: the name of the dataset\u001b[0m                                                \n",
      " \u001b[2m- authors: the author or authors of the dataset\u001b[0m                                \n",
      " \u001b[2m- license: the license of the dataset\u001b[0m                                          \n",
      " \u001b[2m- source: the source of the dataset\u001b[0m                                            \n",
      " \u001b[2m- thumbnail: an image to use as the thumbnail of the dataset in the website\u001b[0m    \n",
      " \u001b[2mThe rest of the content in the README.md file will be used as the description \u001b[0m \n",
      " \u001b[2mof the dataset in the website. If using \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-verbose\u001b[0m\u001b[2m, it will print the progress \u001b[0m \n",
      " \u001b[2mof the ingestion.\u001b[0m                                                              \n",
      "                                                                                \n",
      " \u001b[2mExamples\u001b[0m                                                                       \n",
      " \u001b[1;2;36m--------\u001b[0m                                                                       \n",
      " \u001b[2m$ eotdl dataset ingest \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-path\u001b[0m\u001b[2m /path/to/folder-with-dataset \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-verbose\u001b[0m\u001b[2m True\u001b[0m      \n",
      "                                                                                \n",
      "\u001b[2m╭─\u001b[0m\u001b[2m Options \u001b[0m\u001b[2m───────────────────────────────────────────────────────────────────\u001b[0m\u001b[2m─╮\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[31m*\u001b[0m  \u001b[1;36m-\u001b[0m\u001b[1;36m-path\u001b[0m     \u001b[1;32m-p\u001b[0m      \u001b[1;33mPATH\u001b[0m  Path to the dataset to ingest \u001b[2m[default: None]\u001b[0m    \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m                             \u001b[2;31m[required]                   \u001b[0m                    \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m    \u001b[1;36m-\u001b[0m\u001b[1;36m-verbose\u001b[0m  \u001b[1;32m-v\u001b[0m      \u001b[1;33m    \u001b[0m  Verbose output. This will print the progress of  \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m                             the ingestion                                    \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m    \u001b[1;36m-\u001b[0m\u001b[1;36m-force\u001b[0m    \u001b[1;32m-f\u001b[0m      \u001b[1;33m    \u001b[0m  Force metadata update even if it already exists. \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m                             Will overwrite the current metadata in EOTDL     \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m    \u001b[1;36m-\u001b[0m\u001b[1;36m-sync\u001b[0m     \u001b[1;32m-s\u001b[0m      \u001b[1;33m    \u001b[0m  Sync local metadata with the EOTDL. Will         \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m                             overwrite the local metadata                     \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m    \u001b[1;36m-\u001b[0m\u001b[1;36m-help\u001b[0m             \u001b[1;33m    \u001b[0m  Show this message and exit.                      \u001b[2m│\u001b[0m\n",
      "\u001b[2m╰──────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!eotdl datasets ingest --help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are several ways in which you can ingest a dataset:\n",
    "\n",
    "1. From a local folder in your system with the data you want to upload (with or without a STAC catalog).\n",
    "2. From a list of links to assets in another repository (cloud bucket, huggingface, etc.)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ingesting a local dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m\u001b[36mEuroSAT-RGB-small\u001b[m\u001b[m      \u001b[1m\u001b[36mEuroSAT-small\u001b[m\u001b[m\n",
      "\u001b[1m\u001b[36mEuroSAT-RGB-small-STAC\u001b[m\u001b[m \u001b[1m\u001b[36mRoadSegmentation\u001b[m\u001b[m\n"
     ]
    }
   ],
   "source": [
    "!ls example_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this tutorial we are going to work with a subsample of the [EuroSAT](https://www.eotdl.com/datasets/EuroSAT-RGB) dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm -rf example_data/EuroSAT-small/catalog.parquet\n",
    "!rm -rf example_data/EuroSAT-small/README.md"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['example_data/EuroSAT-small/Forest/Forest_1.tif',\n",
       " 'example_data/EuroSAT-small/Forest/Forest_2.tif',\n",
       " 'example_data/EuroSAT-small/Forest/Forest_3.tif',\n",
       " 'example_data/EuroSAT-small/AnnualCrop/AnnualCrop_2.tif',\n",
       " 'example_data/EuroSAT-small/AnnualCrop/AnnualCrop_3.tif',\n",
       " 'example_data/EuroSAT-small/AnnualCrop/AnnualCrop_1.tif']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from glob import glob \n",
    "\n",
    "path = \"example_data/EuroSAT-small\"\n",
    "files = glob(f'{path}/**/*.*', recursive=True)\n",
    "files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For all cases, a `README.md` file is required in order to ingest datasets and models, containing some basic required information (dataset authors, licens, link to source and dataset name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create README.md\n",
    "\n",
    "text = \"\"\"---\n",
    "name: EuroSAT-small\n",
    "authors: \n",
    "  - Juan B. Pedro\n",
    "license: free\n",
    "source: https://github.com/earthpulse/eotdl/blob/main/tutorials/notebooks/02_ingesting.ipynb\n",
    "---\n",
    "\n",
    "# EuroSAT-small\n",
    "\n",
    "This is a small subet of the EuroSAT dataset.\n",
    "\"\"\"\n",
    "\n",
    "with open(f\"{path}/README.md\", \"w\") as outfile:\n",
    "    outfile.write(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---\n",
      "name: EuroSAT-small\n",
      "authors: \n",
      "  - Juan B. Pedro\n",
      "license: free\n",
      "source: https://github.com/earthpulse/eotdl/blob/main/tutorials/notebooks/02_ingesting.ipynb\n",
      "---\n",
      "\n",
      "# EuroSAT-small\n",
      "\n",
      "This is a small subet of the EuroSAT dataset.\n"
     ]
    }
   ],
   "source": [
    "!cat {path}/README.md"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `name` property in the `README.md` file is used for the name of the dataset or model in the repository, hence it must be unique, between 3 and 45 characters long and can only contain alphanumeric characters and dashes (learn more at [https://www.eotdl.com/docs/datasets/ingest](https://www.eotdl.com/docs/datasets/ingest)).\n",
    "\n",
    "Trying to ingest a dataset without a `README.md` file will fail.\n",
    "\n",
    "If everything is correct, the ingestion process should suceed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ingesting directory: example_data/EuroSAT-small\n",
      "Ingesting files: 100%|████████████████████████████| 7/7 [00:04<00:00,  1.41it/s]\n"
     ]
    }
   ],
   "source": [
    "!eotdl datasets ingest -p example_data/EuroSAT-small"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now your dataset is avilable at EOTDL"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['EuroSAT-small']\n"
     ]
    }
   ],
   "source": [
    "!eotdl datasets list -n eurosat-small"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Since the `EuroSAT-small` name is already taken, this process should fail for you. To solve it, just upload the dataset with a different name. However, this will polute the EOTDL with test datasets so we encourage you to try the ingestion process with a real dataset that you want to ingest (or overwrite your test dataset in the future with useful data)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ingesting a local STAC catalog"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before the ingestion, the CLI will create a STAC-compliant `parquet` file with the metadata of the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>type</th>\n",
       "      <th>stac_version</th>\n",
       "      <th>stac_extensions</th>\n",
       "      <th>datetime</th>\n",
       "      <th>id</th>\n",
       "      <th>bbox</th>\n",
       "      <th>geometry</th>\n",
       "      <th>assets</th>\n",
       "      <th>links</th>\n",
       "      <th>repository</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Feature</td>\n",
       "      <td>1.0.0</td>\n",
       "      <td>[]</td>\n",
       "      <td>2025-04-02 15:54:27.593437</td>\n",
       "      <td>README.md</td>\n",
       "      <td>{'xmax': 0.0, 'xmin': 0.0, 'ymax': 0.0, 'ymin'...</td>\n",
       "      <td>POLYGON EMPTY</td>\n",
       "      <td>{'asset': {'checksum': 'a6bb30a57d0f5ff0aaa65b...</td>\n",
       "      <td>[]</td>\n",
       "      <td>eotdl</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Feature</td>\n",
       "      <td>1.0.0</td>\n",
       "      <td>[]</td>\n",
       "      <td>2025-04-02 15:54:27.593591</td>\n",
       "      <td>Forest/Forest_1.tif</td>\n",
       "      <td>{'xmax': 0.0, 'xmin': 0.0, 'ymax': 0.0, 'ymin'...</td>\n",
       "      <td>POLYGON EMPTY</td>\n",
       "      <td>{'asset': {'checksum': 'f3b8b9fef6b2df6f24792e...</td>\n",
       "      <td>[]</td>\n",
       "      <td>eotdl</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Feature</td>\n",
       "      <td>1.0.0</td>\n",
       "      <td>[]</td>\n",
       "      <td>2025-04-02 15:54:27.594124</td>\n",
       "      <td>Forest/Forest_2.tif</td>\n",
       "      <td>{'xmax': 0.0, 'xmin': 0.0, 'ymax': 0.0, 'ymin'...</td>\n",
       "      <td>POLYGON EMPTY</td>\n",
       "      <td>{'asset': {'checksum': '2e38dab64435bfbab25bab...</td>\n",
       "      <td>[]</td>\n",
       "      <td>eotdl</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Feature</td>\n",
       "      <td>1.0.0</td>\n",
       "      <td>[]</td>\n",
       "      <td>2025-04-02 15:54:27.594343</td>\n",
       "      <td>Forest/Forest_3.tif</td>\n",
       "      <td>{'xmax': 0.0, 'xmin': 0.0, 'ymax': 0.0, 'ymin'...</td>\n",
       "      <td>POLYGON EMPTY</td>\n",
       "      <td>{'asset': {'checksum': '3e7bb982f9db5f7dabc556...</td>\n",
       "      <td>[]</td>\n",
       "      <td>eotdl</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Feature</td>\n",
       "      <td>1.0.0</td>\n",
       "      <td>[]</td>\n",
       "      <td>2025-04-02 15:54:27.594590</td>\n",
       "      <td>AnnualCrop/AnnualCrop_2.tif</td>\n",
       "      <td>{'xmax': 0.0, 'xmin': 0.0, 'ymax': 0.0, 'ymin'...</td>\n",
       "      <td>POLYGON EMPTY</td>\n",
       "      <td>{'asset': {'checksum': 'c406cb8920858b98898b9e...</td>\n",
       "      <td>[]</td>\n",
       "      <td>eotdl</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      type stac_version stac_extensions                   datetime  \\\n",
       "0  Feature        1.0.0              [] 2025-04-02 15:54:27.593437   \n",
       "1  Feature        1.0.0              [] 2025-04-02 15:54:27.593591   \n",
       "2  Feature        1.0.0              [] 2025-04-02 15:54:27.594124   \n",
       "3  Feature        1.0.0              [] 2025-04-02 15:54:27.594343   \n",
       "4  Feature        1.0.0              [] 2025-04-02 15:54:27.594590   \n",
       "\n",
       "                            id  \\\n",
       "0                    README.md   \n",
       "1          Forest/Forest_1.tif   \n",
       "2          Forest/Forest_2.tif   \n",
       "3          Forest/Forest_3.tif   \n",
       "4  AnnualCrop/AnnualCrop_2.tif   \n",
       "\n",
       "                                                bbox       geometry  \\\n",
       "0  {'xmax': 0.0, 'xmin': 0.0, 'ymax': 0.0, 'ymin'...  POLYGON EMPTY   \n",
       "1  {'xmax': 0.0, 'xmin': 0.0, 'ymax': 0.0, 'ymin'...  POLYGON EMPTY   \n",
       "2  {'xmax': 0.0, 'xmin': 0.0, 'ymax': 0.0, 'ymin'...  POLYGON EMPTY   \n",
       "3  {'xmax': 0.0, 'xmin': 0.0, 'ymax': 0.0, 'ymin'...  POLYGON EMPTY   \n",
       "4  {'xmax': 0.0, 'xmin': 0.0, 'ymax': 0.0, 'ymin'...  POLYGON EMPTY   \n",
       "\n",
       "                                              assets links repository  \n",
       "0  {'asset': {'checksum': 'a6bb30a57d0f5ff0aaa65b...    []      eotdl  \n",
       "1  {'asset': {'checksum': 'f3b8b9fef6b2df6f24792e...    []      eotdl  \n",
       "2  {'asset': {'checksum': '2e38dab64435bfbab25bab...    []      eotdl  \n",
       "3  {'asset': {'checksum': '3e7bb982f9db5f7dabc556...    []      eotdl  \n",
       "4  {'asset': {'checksum': 'c406cb8920858b98898b9e...    []      eotdl  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import geopandas as gpd\n",
    "\n",
    "catalog = f\"{path}/catalog.parquet\"\n",
    "\n",
    "gdf = gpd.read_parquet(catalog)\n",
    "gdf.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, if your local dataset already contains a STAC catalog, the available schema will be used to create the EOTDL `parquet` catalog (including the different STAC extensions or properties that might be present)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"type\": \"Catalog\",\n",
      "  \"id\": \"EuroSAT-RGB-Q1\",\n",
      "  \"stac_version\": \"1.0.0\",\n",
      "  \"description\": \"EuroSAT-RGB dataset\",\n",
      "  \"links\": [\n",
      "    {\n",
      "      \"rel\": \"root\",\n",
      "      \"href\": \"./catalog.json\",\n",
      "      \"type\": \"application/json\"\n",
      "    },\n",
      "    {\n",
      "      \"rel\": \"child\",\n",
      "      \"href\": \"./source/collection.json\",\n",
      "      \"type\": \"application/json\"\n",
      "    },\n",
      "    {\n",
      "      \"rel\": \"child\",\n",
      "      \"href\": \"./labels/collection.json\",\n",
      "      \"type\": \"application/json\"\n",
      "    }\n",
      "  ]\n",
      "}"
     ]
    }
   ],
   "source": [
    "import os\n",
    "\n",
    "path = 'example_data/EuroSAT-RGB-small-STAC'\n",
    "\n",
    "files = os.listdir(path)\n",
    "assert 'catalog.json' in files, \"catalog.json not found\"\n",
    "\n",
    "!cat example_data/EuroSAT-RGB-small-STAC/catalog.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create README.md\n",
    "\n",
    "text = \"\"\"---\n",
    "name: EuroSAT-RGB-small-STAC\n",
    "authors: \n",
    "  - Juan B. Pedro\n",
    "license: free\n",
    "source: https://github.com/earthpulse/eotdl/blob/develop/tutorials/workshops/philab24/02_prototype_ingesting.ipynb\n",
    "---\n",
    "\n",
    "# EuroSAT-RGB-small-STAC\n",
    "\n",
    "This is a prototype of the EuroSAT dataset.\n",
    "\"\"\"\n",
    "\n",
    "with open(f\"{path}/README.md\", \"w\") as outfile:\n",
    "    outfile.write(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ingesting items from collection source: 100it [00:00, 12518.82it/s]\n",
      "Ingesting items from collection labels: 100it [00:00, 16041.24it/s]\n",
      "Ingesting files: 100%|████████████████████████| 200/200 [01:49<00:00,  1.83it/s]\n"
     ]
    }
   ],
   "source": [
    "!eotdl datasets ingest -p example_data/EuroSAT-RGB-small-STAC"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The resulting `catalog.parquet` file contains the same information as the STAC catalog for all the items."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>assets</th>\n",
       "      <th>collection</th>\n",
       "      <th>geometry</th>\n",
       "      <th>id</th>\n",
       "      <th>links</th>\n",
       "      <th>stac_extensions</th>\n",
       "      <th>stac_version</th>\n",
       "      <th>type</th>\n",
       "      <th>datetime</th>\n",
       "      <th>label:classes</th>\n",
       "      <th>label:description</th>\n",
       "      <th>label:methods</th>\n",
       "      <th>label:properties</th>\n",
       "      <th>label:tasks</th>\n",
       "      <th>label:type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>{'asset': {'checksum': '582fb1e054885a609c1e25...</td>\n",
       "      <td>source</td>\n",
       "      <td>POLYGON ((0 0, 0 0, 0 0, 0 0))</td>\n",
       "      <td>Industrial_1743</td>\n",
       "      <td>[{'href': '/Users/juan/Desktop/eotdl/tutorials...</td>\n",
       "      <td>[]</td>\n",
       "      <td>1.1.0</td>\n",
       "      <td>Feature</td>\n",
       "      <td>2000-01-01 00:00:00+00:00</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>{'asset': {'checksum': '2d267caf0ef060780fec89...</td>\n",
       "      <td>source</td>\n",
       "      <td>POLYGON ((0 0, 0 0, 0 0, 0 0))</td>\n",
       "      <td>Industrial_1273</td>\n",
       "      <td>[{'href': '/Users/juan/Desktop/eotdl/tutorials...</td>\n",
       "      <td>[]</td>\n",
       "      <td>1.1.0</td>\n",
       "      <td>Feature</td>\n",
       "      <td>2000-01-01 00:00:00+00:00</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>{'asset': {'checksum': '0204dd4a3296ea8be3b388...</td>\n",
       "      <td>source</td>\n",
       "      <td>POLYGON ((0 0, 0 0, 0 0, 0 0))</td>\n",
       "      <td>Industrial_1117</td>\n",
       "      <td>[{'href': '/Users/juan/Desktop/eotdl/tutorials...</td>\n",
       "      <td>[]</td>\n",
       "      <td>1.1.0</td>\n",
       "      <td>Feature</td>\n",
       "      <td>2000-01-01 00:00:00+00:00</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>{'asset': {'checksum': 'a6d23a61d5c20d4b953117...</td>\n",
       "      <td>source</td>\n",
       "      <td>POLYGON ((0 0, 0 0, 0 0, 0 0))</td>\n",
       "      <td>Industrial_1121</td>\n",
       "      <td>[{'href': '/Users/juan/Desktop/eotdl/tutorials...</td>\n",
       "      <td>[]</td>\n",
       "      <td>1.1.0</td>\n",
       "      <td>Feature</td>\n",
       "      <td>2000-01-01 00:00:00+00:00</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>{'asset': {'checksum': 'ed45a188a146a26eae0f2a...</td>\n",
       "      <td>source</td>\n",
       "      <td>POLYGON ((0 0, 0 0, 0 0, 0 0))</td>\n",
       "      <td>Industrial_1641</td>\n",
       "      <td>[{'href': '/Users/juan/Desktop/eotdl/tutorials...</td>\n",
       "      <td>[]</td>\n",
       "      <td>1.1.0</td>\n",
       "      <td>Feature</td>\n",
       "      <td>2000-01-01 00:00:00+00:00</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              assets collection  \\\n",
       "0  {'asset': {'checksum': '582fb1e054885a609c1e25...     source   \n",
       "1  {'asset': {'checksum': '2d267caf0ef060780fec89...     source   \n",
       "2  {'asset': {'checksum': '0204dd4a3296ea8be3b388...     source   \n",
       "3  {'asset': {'checksum': 'a6d23a61d5c20d4b953117...     source   \n",
       "4  {'asset': {'checksum': 'ed45a188a146a26eae0f2a...     source   \n",
       "\n",
       "                         geometry               id  \\\n",
       "0  POLYGON ((0 0, 0 0, 0 0, 0 0))  Industrial_1743   \n",
       "1  POLYGON ((0 0, 0 0, 0 0, 0 0))  Industrial_1273   \n",
       "2  POLYGON ((0 0, 0 0, 0 0, 0 0))  Industrial_1117   \n",
       "3  POLYGON ((0 0, 0 0, 0 0, 0 0))  Industrial_1121   \n",
       "4  POLYGON ((0 0, 0 0, 0 0, 0 0))  Industrial_1641   \n",
       "\n",
       "                                               links stac_extensions  \\\n",
       "0  [{'href': '/Users/juan/Desktop/eotdl/tutorials...              []   \n",
       "1  [{'href': '/Users/juan/Desktop/eotdl/tutorials...              []   \n",
       "2  [{'href': '/Users/juan/Desktop/eotdl/tutorials...              []   \n",
       "3  [{'href': '/Users/juan/Desktop/eotdl/tutorials...              []   \n",
       "4  [{'href': '/Users/juan/Desktop/eotdl/tutorials...              []   \n",
       "\n",
       "  stac_version     type                  datetime label:classes  \\\n",
       "0        1.1.0  Feature 2000-01-01 00:00:00+00:00          None   \n",
       "1        1.1.0  Feature 2000-01-01 00:00:00+00:00          None   \n",
       "2        1.1.0  Feature 2000-01-01 00:00:00+00:00          None   \n",
       "3        1.1.0  Feature 2000-01-01 00:00:00+00:00          None   \n",
       "4        1.1.0  Feature 2000-01-01 00:00:00+00:00          None   \n",
       "\n",
       "  label:description label:methods label:properties label:tasks label:type  \n",
       "0              None          None             None        None       None  \n",
       "1              None          None             None        None       None  \n",
       "2              None          None             None        None       None  \n",
       "3              None          None             None        None       None  \n",
       "4              None          None             None        None       None  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import geopandas as gpd\n",
    "\n",
    "gdf = gpd.read_parquet(path + \"/catalog.parquet\")\n",
    "gdf.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ingesting with the library"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also ingest datasets with the library (you will need to create a `README.md` file as well)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ingesting directory: example_data/EuroSAT-small\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Ingesting files: 100%|██████████| 7/7 [00:02<00:00,  2.57it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No new version was created, your dataset has not changed.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "from eotdl.datasets import ingest_dataset\n",
    "\n",
    "ingest_dataset(\"example_data/EuroSAT-small\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ingesting a virtual dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Option 2 consists on creating a `virtual dataset` from a list of links to assets in another repository (cloud bucket, huggingface, etc.), and is only available through the library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "links = [\n",
    "\t'https://link1.com',\n",
    "\t'https://link2.com',\n",
    "\t'https://link3.com',\n",
    "]\n",
    "\n",
    "metadata = {\n",
    "\t'name': 'Test-links',\n",
    "\t'authors': ['Juan B. Pedro'],\n",
    "\t'license': 'free',\n",
    "\t'source': 'https://github.com/earthpulse/eotdl/blob/develop/tutorials/workshops/philab24/02_prototype_ingesting.ipynb',\n",
    "\t'description': \"\"\"# Test links\n",
    "\n",
    "Testing the ingestion of a dataset from a list of links.\n",
    "\"\"\"\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Ingesting files: 100%|██████████| 4/4 [00:00<00:00,  7.83it/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "PosixPath('data/test-links/catalog.parquet')"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from eotdl.datasets import ingest_virtual_dataset\n",
    "\n",
    "path = 'data/test-links'\n",
    "\n",
    "ingest_virtual_dataset(path, links, metadata)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you already have a `README.md` file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/juan/Desktop/eotdl/.venv/lib/python3.12/site-packages/geopandas/array.py:968: RuntimeWarning: All-NaN slice encountered\n",
      "  np.nanmin(b[:, 0]),  # minx\n",
      "/Users/juan/Desktop/eotdl/.venv/lib/python3.12/site-packages/geopandas/array.py:969: RuntimeWarning: All-NaN slice encountered\n",
      "  np.nanmin(b[:, 1]),  # miny\n",
      "/Users/juan/Desktop/eotdl/.venv/lib/python3.12/site-packages/geopandas/array.py:970: RuntimeWarning: All-NaN slice encountered\n",
      "  np.nanmax(b[:, 2]),  # maxx\n",
      "/Users/juan/Desktop/eotdl/.venv/lib/python3.12/site-packages/geopandas/array.py:971: RuntimeWarning: All-NaN slice encountered\n",
      "  np.nanmax(b[:, 3]),  # maxy\n",
      "Ingesting files: 100%|██████████| 4/4 [00:00<00:00, 973.27it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No new version was created, your dataset has not changed.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "ingest_virtual_dataset(path, links)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `catalog.parquet` file will be created in the provided path."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>type</th>\n",
       "      <th>stac_version</th>\n",
       "      <th>stac_extensions</th>\n",
       "      <th>datetime</th>\n",
       "      <th>id</th>\n",
       "      <th>bbox</th>\n",
       "      <th>geometry</th>\n",
       "      <th>assets</th>\n",
       "      <th>links</th>\n",
       "      <th>repository</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Feature</td>\n",
       "      <td>1.0.0</td>\n",
       "      <td>[]</td>\n",
       "      <td>2025-04-02 15:57:04.637389</td>\n",
       "      <td>https://link1.com</td>\n",
       "      <td>{'xmax': 0.0, 'xmin': 0.0, 'ymax': 0.0, 'ymin'...</td>\n",
       "      <td>POLYGON EMPTY</td>\n",
       "      <td>{'asset': {'checksum': None, 'href': 'https://...</td>\n",
       "      <td>[]</td>\n",
       "      <td>eotdl</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Feature</td>\n",
       "      <td>1.0.0</td>\n",
       "      <td>[]</td>\n",
       "      <td>2025-04-02 15:57:04.637449</td>\n",
       "      <td>https://link2.com</td>\n",
       "      <td>{'xmax': 0.0, 'xmin': 0.0, 'ymax': 0.0, 'ymin'...</td>\n",
       "      <td>POLYGON EMPTY</td>\n",
       "      <td>{'asset': {'checksum': None, 'href': 'https://...</td>\n",
       "      <td>[]</td>\n",
       "      <td>eotdl</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Feature</td>\n",
       "      <td>1.0.0</td>\n",
       "      <td>[]</td>\n",
       "      <td>2025-04-02 15:57:04.637455</td>\n",
       "      <td>https://link3.com</td>\n",
       "      <td>{'xmax': 0.0, 'xmin': 0.0, 'ymax': 0.0, 'ymin'...</td>\n",
       "      <td>POLYGON EMPTY</td>\n",
       "      <td>{'asset': {'checksum': None, 'href': 'https://...</td>\n",
       "      <td>[]</td>\n",
       "      <td>eotdl</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Feature</td>\n",
       "      <td>1.0.0</td>\n",
       "      <td>[]</td>\n",
       "      <td>2025-04-02 15:57:04.637465</td>\n",
       "      <td>README.md</td>\n",
       "      <td>{'xmax': 0.0, 'xmin': 0.0, 'ymax': 0.0, 'ymin'...</td>\n",
       "      <td>POLYGON EMPTY</td>\n",
       "      <td>{'asset': {'checksum': '197885c67d6fca3c301d8e...</td>\n",
       "      <td>[]</td>\n",
       "      <td>eotdl</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      type stac_version stac_extensions                   datetime  \\\n",
       "0  Feature        1.0.0              [] 2025-04-02 15:57:04.637389   \n",
       "1  Feature        1.0.0              [] 2025-04-02 15:57:04.637449   \n",
       "2  Feature        1.0.0              [] 2025-04-02 15:57:04.637455   \n",
       "3  Feature        1.0.0              [] 2025-04-02 15:57:04.637465   \n",
       "\n",
       "                  id                                               bbox  \\\n",
       "0  https://link1.com  {'xmax': 0.0, 'xmin': 0.0, 'ymax': 0.0, 'ymin'...   \n",
       "1  https://link2.com  {'xmax': 0.0, 'xmin': 0.0, 'ymax': 0.0, 'ymin'...   \n",
       "2  https://link3.com  {'xmax': 0.0, 'xmin': 0.0, 'ymax': 0.0, 'ymin'...   \n",
       "3          README.md  {'xmax': 0.0, 'xmin': 0.0, 'ymax': 0.0, 'ymin'...   \n",
       "\n",
       "        geometry                                             assets links  \\\n",
       "0  POLYGON EMPTY  {'asset': {'checksum': None, 'href': 'https://...    []   \n",
       "1  POLYGON EMPTY  {'asset': {'checksum': None, 'href': 'https://...    []   \n",
       "2  POLYGON EMPTY  {'asset': {'checksum': None, 'href': 'https://...    []   \n",
       "3  POLYGON EMPTY  {'asset': {'checksum': '197885c67d6fca3c301d8e...    []   \n",
       "\n",
       "  repository  \n",
       "0      eotdl  \n",
       "1      eotdl  \n",
       "2      eotdl  \n",
       "3      eotdl  "
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import geopandas as gpd\n",
    "\n",
    "catalog = f\"{path}/catalog.parquet\"\n",
    "\n",
    "gdf = gpd.read_parquet(catalog)\n",
    "gdf.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ingesting models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can ingest a model exactly in the same way"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m                                                                                \u001b[0m\n",
      "\u001b[1m \u001b[0m\u001b[1;33mUsage: \u001b[0m\u001b[1meotdl models ingest [OPTIONS]\u001b[0m\u001b[1m                                          \u001b[0m\u001b[1m \u001b[0m\n",
      "\u001b[1m                                                                                \u001b[0m\n",
      " Ingest a model to the EOTDL.                                                   \n",
      " \u001b[2mThis command ingests the model to the EOTDL. The model must be a folder with \u001b[0m  \n",
      " \u001b[2mthe model files, and at least a metadata.yml file or a catalog.json file. If \u001b[0m  \n",
      " \u001b[2mthere are not these files, the ingestion will not work. All the files in the \u001b[0m  \n",
      " \u001b[2mfolder will be uploaded to the EOTDL.\u001b[0m                                          \n",
      "                                                                                \n",
      " \u001b[2mThe following constraints apply to the model name:\u001b[0m                             \n",
      " \u001b[2m- It must be unique\u001b[0m                                                            \n",
      " \u001b[2m- It must be between 3 and 45 characters long\u001b[0m                                  \n",
      " \u001b[2m- It can only contain alphanumeric characters and dashes.\u001b[0m                      \n",
      "                                                                                \n",
      " \u001b[2mThe metadata.yml file must contain the following fields:\u001b[0m                       \n",
      " \u001b[2m- name: the name of the model\u001b[0m                                                  \n",
      " \u001b[2m- authors: the author or authors of the model\u001b[0m                                  \n",
      " \u001b[2m- license: the license of the model\u001b[0m                                            \n",
      " \u001b[2m- source: the source of the model\u001b[0m                                              \n",
      " \u001b[2m- thumbnail: an image to use as the thumbnail of the dataset in the website\u001b[0m    \n",
      "                                                                                \n",
      " \u001b[2mIf using \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-verbose\u001b[0m\u001b[2m, it will print the progress of the ingestion.\u001b[0m               \n",
      "                                                                                \n",
      " \u001b[2mExamples\u001b[0m                                                                       \n",
      " \u001b[1;2;36m--------\u001b[0m                                                                       \n",
      " \u001b[2m$ eotdl models ingest \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-path\u001b[0m\u001b[2m /path/to/folder-with-model \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-verbose\u001b[0m\u001b[2m True\u001b[0m         \n",
      "                                                                                \n",
      "\u001b[2m╭─\u001b[0m\u001b[2m Options \u001b[0m\u001b[2m───────────────────────────────────────────────────────────────────\u001b[0m\u001b[2m─╮\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[31m*\u001b[0m  \u001b[1;36m-\u001b[0m\u001b[1;36m-path\u001b[0m     \u001b[1;32m-p\u001b[0m      \u001b[1;33mPATH\u001b[0m  Path to the model to ingest \u001b[2m[default: None]\u001b[0m      \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m                             \u001b[2;31m[required]                 \u001b[0m                      \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m    \u001b[1;36m-\u001b[0m\u001b[1;36m-verbose\u001b[0m          \u001b[1;33m    \u001b[0m  Verbose output. This will print the progress of  \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m                             the ingestion                                    \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m    \u001b[1;36m-\u001b[0m\u001b[1;36m-force\u001b[0m    \u001b[1;32m-f\u001b[0m      \u001b[1;33m    \u001b[0m  Force metadata update even if it already exists. \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m                             Will overwrite the current metadata in EOTDL     \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m    \u001b[1;36m-\u001b[0m\u001b[1;36m-sync\u001b[0m     \u001b[1;32m-s\u001b[0m      \u001b[1;33m    \u001b[0m  Sync local metadata with the EOTDL. Will         \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m                             overwrite the local metadata                     \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m    \u001b[1;36m-\u001b[0m\u001b[1;36m-help\u001b[0m             \u001b[1;33m    \u001b[0m  Show this message and exit.                      \u001b[2m│\u001b[0m\n",
      "\u001b[2m╰──────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!eotdl models ingest --help"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['example_data/RoadSegmentation/catalog.parquet',\n",
       " 'example_data/RoadSegmentation/README.md',\n",
       " 'example_data/RoadSegmentation/model.onnx']"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from glob import glob \n",
    "\n",
    "path = \"example_data/RoadSegmentation\"\n",
    "files = glob(f'{path}/**/*.*', recursive=True)\n",
    "files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create README.md\n",
    "\n",
    "text = \"\"\"---\n",
    "name: RoadSegmentation\n",
    "authors: \n",
    "  - Juan B. Pedro\n",
    "license: free\n",
    "source: https://github.com/earthpulse/eotdl/blob/develop/tutorials/workshops/philab24/02_prototype_ingesting.ipynb\n",
    "---\n",
    "\n",
    "# RoadSegmentation\n",
    "\n",
    "This is an ONNX model for road segmentation.\n",
    "\"\"\"\n",
    "\n",
    "with open(f\"{path}/README.md\", \"w\") as outfile:\n",
    "    outfile.write(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ingesting directory: example_data/RoadSegmentation\n",
      "Ingesting files: 100%|████████████████████████████| 2/2 [00:05<00:00,  2.65s/it]\n",
      "A new version was created, your dataset has changed.\n",
      "Num changes: 2\n"
     ]
    }
   ],
   "source": [
    "!eotdl models ingest -p example_data/RoadSegmentation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Versioning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, every time you re-upload a dataset or model a new version is created if any changes are detected (new files, modified files, removed files).\n",
    "\n",
    "When you stage a dataset, the latest version is used by default."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset `EuroSAT-small` already exists at /Users/juan/.cache/eotdl/datasets/EuroSAT-small. To force download, use force=True or -f in the CLI.\n"
     ]
    }
   ],
   "source": [
    "!eotdl datasets get EuroSAT-small"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "README.md          catalog.v1.parquet catalog.v2.parquet\n"
     ]
    }
   ],
   "source": [
    "!ls $HOME/.cache/eotdl/datasets/EuroSAT-small"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, you can specify the version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Version 2 not found\n"
     ]
    }
   ],
   "source": [
    "!eotdl datasets get EuroSAT-small -v 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's make some changes and reingest the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm -rf data/EuroSAT-small-modified\n",
    "!cp -r example_data/EuroSAT-small data/EuroSAT-small-modified"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['data/EuroSAT-small-modified/catalog.parquet',\n",
       " 'data/EuroSAT-small-modified/README.md',\n",
       " 'data/EuroSAT-small-modified/Forest/Forest_1.tif',\n",
       " 'data/EuroSAT-small-modified/Forest/Forest_2.tif',\n",
       " 'data/EuroSAT-small-modified/Forest/Forest_3.tif',\n",
       " 'data/EuroSAT-small-modified/AnnualCrop/AnnualCrop_2.tif',\n",
       " 'data/EuroSAT-small-modified/AnnualCrop/AnnualCrop_3.tif',\n",
       " 'data/EuroSAT-small-modified/AnnualCrop/AnnualCrop_1.tif']"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from glob import glob \n",
    "\n",
    "path = \"data/EuroSAT-small-modified\"\n",
    "files = glob(f'{path}/**/*.*', recursive=True)\n",
    "files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "# mofidy README.md\n",
    "\n",
    "text = \"\"\"---\n",
    "name: EuroSAT-small\n",
    "authors: \n",
    "  - Juan B. Pedro\n",
    "license: free\n",
    "source: https://github.com/earthpulse/eotdl/blob/main/tutorials/notebooks/02_ingesting.ipynb\n",
    "---\n",
    "\n",
    "# EuroSAT-small\n",
    "\n",
    "This is a small subet of the EuroSAT dataset.\n",
    "\"\"\"\n",
    "\n",
    "with open(f\"{path}/README.md\", \"w\") as outfile:\n",
    "    outfile.write(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['data/EuroSAT-small-modified/catalog.parquet',\n",
       " 'data/EuroSAT-small-modified/README.md',\n",
       " 'data/EuroSAT-small-modified/test.txt',\n",
       " 'data/EuroSAT-small-modified/Forest/Forest_1.tif',\n",
       " 'data/EuroSAT-small-modified/Forest/Forest_2.tif',\n",
       " 'data/EuroSAT-small-modified/Forest/Forest_3.tif',\n",
       " 'data/EuroSAT-small-modified/AnnualCrop/AnnualCrop_2.tif',\n",
       " 'data/EuroSAT-small-modified/AnnualCrop/AnnualCrop_3.tif',\n",
       " 'data/EuroSAT-small-modified/AnnualCrop/AnnualCrop_1.tif']"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# add a new file\n",
    "\n",
    "with open(f\"{path}/test.txt\", \"w\") as outfile:\n",
    "    outfile.write(\"This is a new file!\")\n",
    "    \n",
    "files = glob(f'{path}/**/*.*', recursive=True)\n",
    "files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ingesting directory: data/EuroSAT-small-modified\n",
      "Ingesting files: 100%|████████████████████████████| 8/8 [00:03<00:00,  2.28it/s]\n",
      "A new version was created, your dataset has changed.\n",
      "Num changes: 1\n"
     ]
    }
   ],
   "source": [
    "!eotdl datasets ingest -p data/EuroSAT-small-modified"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data available at /Users/juan/.cache/eotdl/datasets/EuroSAT-small\n"
     ]
    }
   ],
   "source": [
    "!eotdl datasets get EuroSAT-small -f -v 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "README.md          catalog.v1.parquet catalog.v2.parquet\n"
     ]
    }
   ],
   "source": [
    "!ls $HOME/.cache/eotdl/datasets/EuroSAT-small"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We apply versioning at dataset/model and file level, meaning only new or modified files will be uploaded in future re-uploads, downloading the appropriate files for each version.\n",
    "\n",
    "You can explore the different versions in the user interface."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}