tutorials/workshops/philab24/03_ingesting.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ingest an existing Dataset or Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook we are going to showcase how to ingest an existing dataset or model into EOTDL.\n",
    "\n",
    "Once it is ingested, you can use it in the same way as any other dataset or model in EOTDL (exploring, downloading, etc.)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ingesting through the CLI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The recommended way to ingest a dataset is using the CLI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m                                                                                \u001b[0m\n",
      "\u001b[1m \u001b[0m\u001b[1;33mUsage: \u001b[0m\u001b[1meotdl datasets ingest [OPTIONS]\u001b[0m\u001b[1m                                        \u001b[0m\u001b[1m \u001b[0m\n",
      "\u001b[1m                                                                                \u001b[0m\n",
      " Ingest a dataset to the EOTDL.                                                 \n",
      " \u001b[2mThis command ingests the dataset to the EOTDL. The dataset must be a folder \u001b[0m   \n",
      " \u001b[2mwith the dataset files, and at least a README.md file (and a catalog.json file\u001b[0m \n",
      " \u001b[2mfor Q1+). If these files are missing, the ingestion will not work. All the \u001b[0m    \n",
      " \u001b[2mfiles in the folder will be uploaded to the EOTDL.\u001b[0m                             \n",
      "                                                                                \n",
      " \u001b[2mThe following constraints apply to the dataset name:\u001b[0m                           \n",
      " \u001b[2m- It must be unique\u001b[0m                                                            \n",
      " \u001b[2m- It must be between 3 and 45 characters long\u001b[0m                                  \n",
      " \u001b[2m- It can only contain alphanumeric characters and dashes.\u001b[0m                      \n",
      "                                                                                \n",
      " \u001b[2mThe README.md file must contain the following fields in the metadata header:\u001b[0m   \n",
      " \u001b[2m- name: the name of the dataset\u001b[0m                                                \n",
      " \u001b[2m- authors: the author or authors of the dataset\u001b[0m                                \n",
      " \u001b[2m- license: the license of the dataset\u001b[0m                                          \n",
      " \u001b[2m- source: the source of the dataset\u001b[0m                                            \n",
      " \u001b[2m- thumbnail: an image to use as the thumbnail of the dataset in the website\u001b[0m    \n",
      " \u001b[2mThe rest of the content in the README.md file will be used as the description \u001b[0m \n",
      " \u001b[2mof the dataset in the website. If using \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-verbose\u001b[0m\u001b[2m, it will print the progress \u001b[0m \n",
      " \u001b[2mof the ingestion.\u001b[0m                                                              \n",
      "                                                                                \n",
      " \u001b[2mExamples\u001b[0m                                                                       \n",
      " \u001b[1;2;36m--------\u001b[0m                                                                       \n",
      " \u001b[2m$ eotdl dataset ingest \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-path\u001b[0m\u001b[2m /path/to/folder-with-dataset \u001b[0m\u001b[1;2;36m-\u001b[0m\u001b[1;2;36m-verbose\u001b[0m\u001b[2m True\u001b[0m      \n",
      "                                                                                \n",
      "\u001b[2m╭─\u001b[0m\u001b[2m Options \u001b[0m\u001b[2m───────────────────────────────────────────────────────────────────\u001b[0m\u001b[2m─╮\u001b[0m\n",
      "\u001b[2m│\u001b[0m \u001b[31m*\u001b[0m  \u001b[1;36m-\u001b[0m\u001b[1;36m-path\u001b[0m     \u001b[1;32m-p\u001b[0m      \u001b[1;33mPATH\u001b[0m  Path to the dataset to ingest \u001b[2m[default: None]\u001b[0m    \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m                             \u001b[2;31m[required]                   \u001b[0m                    \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m    \u001b[1;36m-\u001b[0m\u001b[1;36m-verbose\u001b[0m  \u001b[1;32m-v\u001b[0m      \u001b[1;33m    \u001b[0m  Verbose output. This will print the progress of  \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m                             the ingestion                                    \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m    \u001b[1;36m-\u001b[0m\u001b[1;36m-force\u001b[0m    \u001b[1;32m-f\u001b[0m      \u001b[1;33m    \u001b[0m  Force metadata update even if it already exists. \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m                             Will overwrite the current metadata in EOTDL     \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m    \u001b[1;36m-\u001b[0m\u001b[1;36m-sync\u001b[0m     \u001b[1;32m-s\u001b[0m      \u001b[1;33m    \u001b[0m  Sync local metadata with the EOTDL. Will         \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m                             overwrite the local metadata                     \u001b[2m│\u001b[0m\n",
      "\u001b[2m│\u001b[0m    \u001b[1;36m-\u001b[0m\u001b[1;36m-help\u001b[0m             \u001b[1;33m    \u001b[0m  Show this message and exit.                      \u001b[2m│\u001b[0m\n",
      "\u001b[2m╰──────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!eotdl datasets ingest --help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to ingest a dataset you will need a folder in your system with the data you want to upload."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EuroSAT-small\n"
     ]
    }
   ],
   "source": [
    "!ls workshop_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this tutorial we are going to work with a subsample of the [EuroSAT](https://www.eotdl.com/datasets/EuroSAT-RGB) dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['workshop_data/EuroSAT-small/README.md',\n",
       " 'workshop_data/EuroSAT-small/Forest/Forest_1.tif',\n",
       " 'workshop_data/EuroSAT-small/Forest/Forest_2.tif',\n",
       " 'workshop_data/EuroSAT-small/Forest/Forest_3.tif',\n",
       " 'workshop_data/EuroSAT-small/AnnualCrop/AnnualCrop_1.tif',\n",
       " 'workshop_data/EuroSAT-small/AnnualCrop/AnnualCrop_3.tif',\n",
       " 'workshop_data/EuroSAT-small/AnnualCrop/AnnualCrop_2.tif']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from glob import glob \n",
    "\n",
    "files = glob('workshop_data/EuroSAT-small/**/*.*', recursive=True)\n",
    "files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A `README.md` file is required for Q0 datasets and models, containing some basic required information (dataset authors, licens, link to source and dataset name)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---\n",
      "authors:\n",
      "- Patrick Helber\n",
      "license: open\n",
      "source: http://madm.dfki.de/downloads\n",
      "name: EuroSAT-small\n",
      "---\n",
      "\n",
      "# EuroSAT small\n",
      "\n",
      "This is a test datasets used for the PhiLab24 workshop. It is a subset of the EuroSAT dataset."
     ]
    }
   ],
   "source": [
    "!cat workshop_data/EuroSAT-small/README.md"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The chosen name is the one that will appear in the repository, hence it must be unique, between 3 and 45 characters long and can only contain alphanumeric characters and dashes (learn more at [https://www.eotdl.com/docs/datasets/ingest](https://www.eotdl.com/docs/datasets/ingest)).\n",
    "\n",
    "Trying to ingest a dataset without a `README.md` file will fail.\n",
    "\n",
    "If everything is correct, the ingestion process should suceed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Uploading directory workshop_data/EuroSAT-small...\n",
      "generating list of files to upload...\n",
      "100%|███████████████████████████████████████████| 6/6 [00:00<00:00, 4246.68it/s]\n",
      "6 new files will be ingested\n",
      "0 files already exist in dataset\n",
      "0 large files will be ingested separately\n",
      "New version created, version: 1\n",
      "generating batches...\n",
      "100%|█████████████████████████████████████████| 6/6 [00:00<00:00, 162360.15it/s]\n",
      "Uploading 6 small files in 1 batches...\n",
      "Uploading batches: 100%|█████████████████████| 1/1 [00:00<00:00,  1.47batches/s]\n"
     ]
    }
   ],
   "source": [
    "!eotdl datasets ingest -p workshop_data/EuroSAT-small/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now your dataset is avilable at EOTDL"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['EuroSAT-small']\n"
     ]
    }
   ],
   "source": [
    "!eotdl datasets list -n eurosat-small"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Since the `EuroSAT-small` name is already taken, this process should fail for you. To solve it, just upload the dataset with a different name. However, this will polute the EOTDL with test datasets so we encourage you to try the ingestion process with a real dataset that you want to ingest (or overwrite your test dataset in the future with useful data)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to ingest Q1+ datasets, a valid STAC catalog is required instead of the `README.md` file. We will explore this in the [data curation](tutorials/workshops/bids23/05_STAC_metadata.ipynb) notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now ingest the model that we trained in the previous notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "data/model.onnx\n"
     ]
    }
   ],
   "source": [
    "!ls data/*.onnx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir -p data/EuroSAT-RGB-PhiLab24-model\n",
    "!cp data/model.onnx data/EuroSAT-RGB-PhiLab24-model/model.onnx"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we need to create a folder with the model and the `README.md` file (and any other file that you want)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "import yaml\n",
    "\n",
    "metadata = {\n",
    "\t\"name\": \"EuroSAT-RGB-BiDS23\",\n",
    "\t\"authors\": [\"Juan B. Pedro\"],\n",
    "\t\"license\": \"open\",\n",
    "\t\"source\": \"https://github.com/earthpulse/eotdl/blob/develop/tutorials/workshops/bids23/02_training.ipynb\"\n",
    "}\n",
    "\n",
    "with open('data/EuroSAT-RGB-BiDS-model/metadata.yml', 'w') as outfile:\n",
    "\tyaml.dump(metadata, outfile, default_flow_style=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = \"\"\"---\n",
    "name: EuroSAT-RGB-PhiLab24\n",
    "authors: \n",
    "  - Juan B. Pedro\n",
    "license: open\n",
    "source: https://github.com/earthpulse/eotdl/blob/develop/tutorials/workshops/philab24/02_training.ipynb\n",
    "---\n",
    "\n",
    "# EuroSAT-RGB-PhiLab24\n",
    "\n",
    "This is a toy model trained with the EuroSAT dataset for the PhiLab24 workshop.\n",
    "\"\"\"\n",
    "\n",
    "with open('data/EuroSAT-RGB-PhiLab24-model/README.md', 'w') as outfile:\n",
    "\toutfile.write(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can ingest the model to EOTDL"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Uploading directory data/EuroSAT-RGB-PhiLab24-model...\n",
      "generating list of files to upload...\n",
      "100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 17.36it/s]\n",
      "1 new files will be ingested\n",
      "0 files already exist in dataset\n",
      "1 large files will be ingested separately\n",
      "New version created, version: 1\n",
      "ingesting large files...\n",
      "ingesting file: model.onnx\n",
      "models\n",
      "42.65/42.65 MB: : 5it [00:03,  1.26it/s]                                        \n"
     ]
    }
   ],
   "source": [
    "!eotdl models ingest -p data/EuroSAT-RGB-PhiLab24-model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Versioning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, every time you re-upload a dataset or model a new version is created. We apply versioning at dataset/model and file level, meaning only new or modified files will be uploaded in future re-uploads, downloading the appropriate files for each version."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Uploading directory workshop_data/EuroSAT-small...\n",
      "generating list of files to upload...\n",
      "100%|███████████████████████████████████████████| 7/7 [00:00<00:00, 3436.75it/s]\n",
      "1 new files will be ingested\n",
      "6 files already exist in dataset\n",
      "0 large files will be ingested separately\n",
      "New version created, version: 2\n",
      "generating batches...\n",
      "100%|██████████████████████████████████████████| 1/1 [00:00<00:00, 14563.56it/s]\n",
      "Uploading 1 small files in 1 batches...\n",
      "Uploading batches: 100%|█████████████████████| 1/1 [00:00<00:00,  2.30batches/s]\n",
      "100%|██████████████████████████████████████████| 6/6 [00:00<00:00, 88612.06it/s]\n",
      "Ingesting existing files: 100%|██████████████| 1/1 [00:00<00:00,  3.61batches/s]\n"
     ]
    }
   ],
   "source": [
    "!echo \"hello\" > workshop_data/EuroSAT-small/hello.txt\n",
    "!eotdl datasets ingest -p workshop_data/EuroSAT-small/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you download a dataset, the latest version is used by default."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████████| 7/7 [00:02<00:00,  2.46file/s]\n",
      "Data available at /home/juan/.cache/eotdl/datasets/EuroSAT-small/v2\n"
     ]
    }
   ],
   "source": [
    "!eotdl datasets get EuroSAT-small"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, you can specify the version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset `EuroSAT-small v1` already exists at /home/juan/.cache/eotdl/datasets/EuroSAT-small/v1. To force download, use force=True or -f in the CLI.\n"
     ]
    }
   ],
   "source": [
    "!eotdl datasets get EuroSAT-small -v 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "v1  v2\n"
     ]
    }
   ],
   "source": [
    "!ls $HOME/.cache/eotdl/datasets/EuroSAT-small"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can explore the different versions in the user interface."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ingesting through the Library"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can ingest datasets and models using the library"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Uploading directory workshop_data/EuroSAT-small...\n",
      "generating list of files to upload...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 7/7 [00:00<00:00, 3520.82it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No new files to upload\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "from eotdl.datasets import ingest_dataset\n",
    "\n",
    "try:\n",
    "    ingest_dataset(\"workshop_data/EuroSAT-small\")\n",
    "except Exception as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Uploading directory data/EuroSAT-RGB-PhiLab24-model...\n",
      "generating list of files to upload...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1/1 [00:00<00:00, 16.56it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No new files to upload\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "from eotdl.models import ingest_model\n",
    "\n",
    "try:\n",
    "    ingest_model(\"data/EuroSAT-RGB-PhiLab24-model\")\n",
    "except Exception as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Discussion and Contribution opportunities"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Feel free to ask questions now (live or through Discord) and make suggestions for future improvements.\n",
    "\n",
    "- What features concerning ingestion would you like to see?\n",
    "- What other features concerning versioning would you like to see?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "eotdl",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}