tutorials/workshops/bids23/04_creating.ipynb

{
   "cells": [
      {
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "# Creating datasets"
         ]
      },
      {
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "This is probably the most interesting part of the tutorial, where you get to leverage EOTDL tools to create a brand new dataset. Here we cover:\n",
            "\n",
            "1. **Data exploration**: given an area of interest, query available sentinel data for your dataset.\n",
            "2. **Data access**: download your data for creating the dataset.\n",
            "3. **Data preparation**: clean your data, perform feature engineering, data analysis, labelling, etc.\n",
            "\n",
            "Once your dataset is ready, you can ingest it to the EOTDL like we have seen in the previous notebook and start working with it as any other dataset in the repository."
         ]
      },
      {
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "## Exploration"
         ]
      },
      {
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "First of all, let's explore the area of interest that we have selected for this workshop. In this case we have chosen the [Boadella reservoir](https://es.wikipedia.org/wiki/Embalse_de_Darnius_Boadella) in Catalonia, Spain, which geometry is in the data folder as `workshop_data/boadella.geojson`. Here we use [leafmap](https://leafmap.org/) for visualizing it, but feel free to use your preferred solution."
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 1,
         "metadata": {},
         "outputs": [],
         "source": [
            "# !pip install leafmap"
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 4,
         "metadata": {},
         "outputs": [
            {
               "data": {
                  "application/vnd.jupyter.widget-view+json": {
                     "model_id": "fa68875bacdd421cbdad01c018c33dc0",
                     "version_major": 2,
                     "version_minor": 0
                  },
                  "text/plain": [
                     "Map(center=[42.347577325903515, 2.815024677909404], controls=(ZoomControl(options=['position', 'zoom_in_text',…"
                  ]
               },
               "execution_count": 4,
               "metadata": {},
               "output_type": "execute_result"
            }
         ],
         "source": [
            "import leafmap\n",
            "import geopandas as gpd\n",
            "\n",
            "in_geojson = 'workshop_data/boadella.geojson'\n",
            "gdf = gpd.read_file(in_geojson)\n",
            "\n",
            "centroid_coords = gdf['geometry'].centroid\n",
            "centroid = [centroid_coords.y.values[0], centroid_coords.x.values[0]]   # We are going to use the centroid later\n",
            "\n",
            "m = leafmap.Map(center=centroid, zoom=13)\n",
            "m.add_geojson(in_geojson, layer_name=\"Boadella reservoir\")\n",
            "m"
         ]
      },
      {
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "When creating AI-Ready datasets it is usual to work at a fixed resolution. You can either retrieve full scenes and cut patches, or use EOTDL functionality to generate appropriate bounding boxes. With the aim that all the images in the dataset have 512x512 pixels, we are going to use the centroid that we extracted before from the geoJSON and generate a bounding box that will result in a 512x512 pixels image at 10m resolution since we are going to use Sentinel data."
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 5,
         "metadata": {},
         "outputs": [
            {
               "data": {
                  "text/plain": [
                     "[2.7920278066359443, 42.330578684998784, 2.8380215491828635, 42.36457137143557]"
                  ]
               },
               "execution_count": 5,
               "metadata": {},
               "output_type": "execute_result"
            }
         ],
         "source": [
            "from eotdl.tools import bbox_from_centroid\n",
            "\n",
            "boadella_bbox = bbox_from_centroid(x=centroid[0], y=centroid[1], pixel_size=10, width=512, height=512)\n",
            "boadella_bbox"
         ]
      },
      {
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "Let's visualize the bounding box on a map!"
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 3,
         "metadata": {},
         "outputs": [
            {
               "data": {
                  "application/vnd.jupyter.widget-view+json": {
                     "model_id": "f9f137e1c85748f491af95d81104d307",
                     "version_major": 2,
                     "version_minor": 0
                  },
                  "text/plain": [
                     "Map(bottom=776013.0, center=[42.347992661753544, 2.814509843722251], controls=(ZoomControl(options=['position'…"
                  ]
               },
               "execution_count": 3,
               "metadata": {},
               "output_type": "execute_result"
            }
         ],
         "source": [
            "from eotdl.tools import bbox_to_polygon\n",
            "\n",
            "# Create a polygon from the bbox\n",
            "boadella_polygon = bbox_to_polygon(boadella_bbox)\n",
            "# Create a GeoDataFrame from the polygon\n",
            "gdf = gpd.GeoDataFrame(geometry=[boadella_polygon])\n",
            "# Save the bounding box as a geoJSON file, if needed\n",
            "gdf.to_file('workshop_data/boadella_bbox.geojson', driver='GeoJSON')   # Uncomment to save the bbox as a GeoJSON file\n",
            "\n",
            "m.add_geojson('workshop_data/boadella_bbox.geojson', layer_name=\"Boadella bbox\")\n",
            "m"
         ]
      },
      {
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "Now that he have our desired bounding box we can look for available Sentinel-2 imagery on it. This can be done through the EOTDL. \n",
            "\n",
            "First, we can look for which Sentinel sensors are supported in the EOTDL"
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 5,
         "metadata": {},
         "outputs": [
            {
               "data": {
                  "text/plain": [
                     "('sentinel-1-grd', 'sentinel-2-l1c', 'sentinel-2-l2a', 'dem')"
                  ]
               },
               "execution_count": 5,
               "metadata": {},
               "output_type": "execute_result"
            }
         ],
         "source": [
            "from eotdl.access import SUPPORTED_SENSORS\n",
            "\n",
            "SUPPORTED_SENSORS"
         ]
      },
      {
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "If we want to look for available Sentinel-2 imagery in our AoI, we must define a range of dates in which to search for the images. We have already defined a time interval for this workshop, which is in the `workshop_data/dates.csv` file."
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 6,
         "metadata": {},
         "outputs": [
            {
               "data": {
                  "text/plain": [
                     "['2020-01-13', '2020-01-28', '2020-02-02', '2020-06-21', '2020-09-14']"
                  ]
               },
               "execution_count": 6,
               "metadata": {},
               "output_type": "execute_result"
            }
         ],
         "source": [
            "import csv\n",
            "\n",
            "dates = list()\n",
            "with open(\"workshop_data/dates.csv\", \"r\") as file:\n",
            "    reader = csv.reader(file)\n",
            "    for row in reader:\n",
            "        dates.append(row[0])\n",
            "dates.sort()\n",
            "\n",
            "dates[:5]"
         ]
      },
      {
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "Although we have the specific dates, we are going to search for the entire time interval, just as a demonstrator."
         ]
      },
      {
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "> We use Sentinle Hub under the hood, so you will need appropriate credentials. You can generate them automatically from your user [profile](https://www.eotdl.com/profile) by accepting the terms and conditions. When you login to the EOTDL, via the library or CLI, we retrieve and store this information for you, so you don't need to worry about it. However there are a couple of gotchas: <br><br> 1. If you already have a Sentinel HUB account with the same email as your EOTDL account, you will need to retrieve the credentials from Sentinel Hub Dashboard and set them as env variables. <br> 2. The credentials generated via EOTDL may expire after some time (we are working on this). If this happens, let us know in Discord to fix the issue. <br><br> In any case, you can provide your own credentials by setting the appropriate environment variables: `SH_CLIENT_ID` and `SH_CLIENT_SECRET`."
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 7,
         "metadata": {},
         "outputs": [
            {
               "data": {
                  "text/plain": [
                     "[{'id': 'S2B_MSIL2A_20220601T103629_N0400_R008_T31TDG_20220601T135543',\n",
                     "  'properties': {'datetime': '2022-06-01T10:49:26Z', 'eo:cloud_cover': 0.23}},\n",
                     " {'id': 'S2B_MSIL2A_20220601T103629_N0400_R008_T31TDH_20220601T135543',\n",
                     "  'properties': {'datetime': '2022-06-01T10:49:14Z', 'eo:cloud_cover': 12.82}},\n",
                     " {'id': 'S2A_MSIL2A_20220527T103631_N0400_R008_T31TDG_20220527T183616',\n",
                     "  'properties': {'datetime': '2022-05-27T10:49:34Z', 'eo:cloud_cover': 85.6}},\n",
                     " {'id': 'S2A_MSIL2A_20220527T103631_N0400_R008_T31TDH_20220527T183616',\n",
                     "  'properties': {'datetime': '2022-05-27T10:49:19Z', 'eo:cloud_cover': 30.42}},\n",
                     " {'id': 'S2B_MSIL2A_20220522T103629_N0400_R008_T31TDG_20220522T124154',\n",
                     "  'properties': {'datetime': '2022-05-22T10:49:27Z', 'eo:cloud_cover': 12.99}}]"
                  ]
               },
               "execution_count": 7,
               "metadata": {},
               "output_type": "execute_result"
            }
         ],
         "source": [
            "from eotdl.access import search_sentinel_imagery\n",
            "\n",
            "time_interval = (dates[0], dates[-1])\n",
            "\n",
            "r = search_sentinel_imagery(time_interval, boadella_bbox, 'sentinel-2-l2a')\n",
            "response = list(r)\n",
            "response[:5]"
         ]
      },
      {
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "They make sense, as the [revisit time](https://docs.sentinel-hub.com/api/latest/data/sentinel-2-l2a/#basic-facts) for Sentinel-2 is 5 days.\n",
            "\n",
            "As a final step, let's check the number of dates with available images."
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 8,
         "metadata": {},
         "outputs": [
            {
               "name": "stdout",
               "output_type": "stream",
               "text": [
                  "342\n"
               ]
            }
         ],
         "source": [
            "print(len(response))"
         ]
      },
      {
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "To sum up this section, we have explored our AoI, generated a bounding box and a time interval in which to look for imagery and searched for Sentinel-2 imagery."
         ]
      },
      {
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "## Download"
         ]
      },
      {
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "The next step is to download the images. On the one hand, we can download image by image, as follows."
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 9,
         "metadata": {},
         "outputs": [],
         "source": [
            "from eotdl.access import download_sentinel_imagery\n",
            "\n",
            "first_date = dates[0]\n",
            "\n",
            "# Uncomment to demonstrate\n",
            "# download_sentinel_imagery('workshop_data/sentinel_2', first_date, boadella_bbox, 'sentinel-2-l2a')"
         ]
      },
      {
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "On the other hand, we can search and download all available images within a time interval, as follows. This is the recommended way for a bulk download, but it has the drawback that we cannot control the quality of the images, as for example know their cloud cover."
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 10,
         "metadata": {},
         "outputs": [],
         "source": [
            "from eotdl.access import search_and_download_sentinel_imagery\n",
            "\n",
            "# Uncomment to demonstrate\n",
            "demostration_dates = (dates[0], dates[1])\n",
            "search_and_download_sentinel_imagery(\n",
            "    output='workshop_data/sentinel_2',\n",
            "    time_interval=demostration_dates,\n",
            "    bounding_box=boadella_bbox,\n",
            "    sensor='sentinel-2-l2a'\n",
            ")"
         ]
      },
      {
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "Despite what we have seen, in the `workshop_data/dates.csv` file we already have a list with the acquisition dates of valid, cloud-free and good quality images. This is a slower but safer solution. So, let's download them!"
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 11,
         "metadata": {},
         "outputs": [],
         "source": [
            "for date in dates:\n",
            "    download_sentinel_imagery('workshop_data/sentinel_2', date, boadella_bbox, 'sentinel-2-l2a')"
         ]
      },
      {
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "That's all! We have downloaded the images for our dataset. Let's check them!"
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 12,
         "metadata": {},
         "outputs": [
            {
               "data": {
                  "text/plain": [
                     "['workshop_data/sentinel_2/Boadella_2021-07-16.tif',\n",
                     " 'workshop_data/sentinel_2/Boadella_2020-06-21.tif',\n",
                     " 'workshop_data/sentinel_2/sentinel-2-l2a_2021-06-26.tif',\n",
                     " 'workshop_data/sentinel_2/Boadella_2020-02-02.tif',\n",
                     " 'workshop_data/sentinel_2/sentinel-2-l2a_2022-06-01.tif']"
                  ]
               },
               "execution_count": 12,
               "metadata": {},
               "output_type": "execute_result"
            }
         ],
         "source": [
            "from glob import glob\n",
            "\n",
            "rasters = glob('workshop_data/sentinel_2/*.tif')\n",
            "rasters[:5]"
         ]
      },
      {
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "We can look for them metadata files, too."
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 13,
         "metadata": {},
         "outputs": [
            {
               "data": {
                  "text/plain": [
                     "['workshop_data/sentinel_2/sentinel-2-l2a_2020-01-23.json',\n",
                     " 'workshop_data/sentinel_2/sentinel-2-l2a_2021-07-21.json',\n",
                     " 'workshop_data/sentinel_2/sentinel-2-l2a_2020-09-14.json',\n",
                     " 'workshop_data/sentinel_2/sentinel-2-l2a_2020-01-28.json',\n",
                     " 'workshop_data/sentinel_2/sentinel-2-l2a_2020-06-21.json']"
                  ]
               },
               "execution_count": 13,
               "metadata": {},
               "output_type": "execute_result"
            }
         ],
         "source": [
            "jsons = glob('workshop_data/sentinel_2/*.json')\n",
            "jsons[:5]"
         ]
      },
      {
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "It looks amazing! One last step, in order to easily ingest the images generate STAC metadata in next steps, we will rename the images, maintaining the acquisiton date but replacing the sensor type in the filename by `Boadella`, and delete the metadata files. This is not mandatory, but it will be useful for our usecase."
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 14,
         "metadata": {},
         "outputs": [
            {
               "name": "stdout",
               "output_type": "stream",
               "text": [
                  "mv: 'workshop_data/sentinel_2/Boadella_2021-07-16.tif' and 'workshop_data/sentinel_2/Boadella_2021-07-16.tif' are the same file\n",
                  "mv: 'workshop_data/sentinel_2/Boadella_2020-06-21.tif' and 'workshop_data/sentinel_2/Boadella_2020-06-21.tif' are the same file\n",
                  "mv: 'workshop_data/sentinel_2/Boadella_2020-02-02.tif' and 'workshop_data/sentinel_2/Boadella_2020-02-02.tif' are the same file\n",
                  "mv: 'workshop_data/sentinel_2/metadata.yml' and 'workshop_data/sentinel_2/metadata.yml' are the same file\n",
                  "mv: 'workshop_data/sentinel_2/Boadella_2020-01-28.tif' and 'workshop_data/sentinel_2/Boadella_2020-01-28.tif' are the same file\n",
                  "mv: 'workshop_data/sentinel_2/Boadella_2022-03-08.tif' and 'workshop_data/sentinel_2/Boadella_2022-03-08.tif' are the same file\n",
                  "mv: 'workshop_data/sentinel_2/Boadella_2020-01-13.tif' and 'workshop_data/sentinel_2/Boadella_2020-01-13.tif' are the same file\n",
                  "mv: 'workshop_data/sentinel_2/Boadella_2022-06-01.tif' and 'workshop_data/sentinel_2/Boadella_2022-06-01.tif' are the same file\n",
                  "mv: 'workshop_data/sentinel_2/labels.json' and 'workshop_data/sentinel_2/labels.json' are the same file\n",
                  "mv: 'workshop_data/sentinel_2/Boadella_2020-09-14.tif' and 'workshop_data/sentinel_2/Boadella_2020-09-14.tif' are the same file\n",
                  "mv: 'workshop_data/sentinel_2/Boadella_2021-06-26.tif' and 'workshop_data/sentinel_2/Boadella_2021-06-26.tif' are the same file\n",
                  "mv: 'workshop_data/sentinel_2/Boadella_2020-01-13_labels.geojson' and 'workshop_data/sentinel_2/Boadella_2020-01-13_labels.geojson' are the same file\n",
                  "mv: 'workshop_data/sentinel_2/Boadella_2021-07-21.tif' and 'workshop_data/sentinel_2/Boadella_2021-07-21.tif' are the same file\n",
                  "mv: 'workshop_data/sentinel_2/Boadella_2020-02-02_labels.geojson' and 'workshop_data/sentinel_2/Boadella_2020-02-02_labels.geojson' are the same file\n"
               ]
            }
         ],
         "source": [
            "files = glob('workshop_data/sentinel_2/*')\n",
            "for file in files:\n",
            "    new_file_name = file.replace('sentinel-2-l2a', 'Boadella')\n",
            "    ! mv $file $new_file_name\n",
            "\n",
            "!rm -r workshop_data/sentinel_2/*.json"
         ]
      },
      {
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "## Data Preparation"
         ]
      },
      {
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "As the final step towards creating our training dataset, we need to make the data AI-Ready. There are multitude of tasks that can be performed here, such as:\n",
            "\n",
            "- **Data cleaning**: remove corrupted images, remove images with too much cloud cover, etc.\n",
            "- **Feature engineering**: calculate vegetation indices, calculate statistics, etc.\n",
            "- **Data analysis**: plot time series, plot histograms, etc.\n",
            "- **Labelling**: create labels for the images, etc.\n",
            "\n",
            "For each one, feel free to use your favourite tools. Here we are going to demonstrate labelling using SCANEO.\n",
            "\n",
            "SCANEO is a labelling web application that allows tagging satellite images (to identify, e.g., objects present, terrain types, etc.) in an easy and fast way. The service provided by SCANEO is vital since it is necessary to prepare the satellite data so that it can be processed by neural networks, enabling active learning. \n",
            "\n",
            "Before running the web interface, we need to make sure we have the `scaneo` package installed in our machine and, if not, install it."
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 15,
         "metadata": {},
         "outputs": [],
         "source": [
            "# !pip install scaneo  "
         ]
      },
      {
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "You can run `scaneo` with the following options"
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 16,
         "metadata": {},
         "outputs": [
            {
               "name": "stdout",
               "output_type": "stream",
               "text": [
                  "\u001b[1m                                                                                \u001b[0m\n",
                  "\u001b[1m \u001b[0m\u001b[1;33mUsage: \u001b[0m\u001b[1mscaneo [OPTIONS]\u001b[0m\u001b[1m                                                       \u001b[0m\u001b[1m \u001b[0m\n",
                  "\u001b[1m                                                                                \u001b[0m\n",
                  "\u001b[2m╭─\u001b[0m\u001b[2m Options \u001b[0m\u001b[2m───────────────────────────────────────────────────────────────────\u001b[0m\u001b[2m─╮\u001b[0m\n",
                  "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-port\u001b[0m                \u001b[1;32m-p\u001b[0m      \u001b[1;33mINTEGER\u001b[0m  Port to run the server on             \u001b[2m│\u001b[0m\n",
                  "\u001b[2m│\u001b[0m                                        \u001b[2m[default: 8000]          \u001b[0m             \u001b[2m│\u001b[0m\n",
                  "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-host\u001b[0m                \u001b[1;32m-h\u001b[0m      \u001b[1;33mTEXT   \u001b[0m  Host to run the server on             \u001b[2m│\u001b[0m\n",
                  "\u001b[2m│\u001b[0m                                        \u001b[2m[default: localhost]     \u001b[0m             \u001b[2m│\u001b[0m\n",
                  "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-data\u001b[0m                \u001b[1;32m-d\u001b[0m      \u001b[1;33mPATH   \u001b[0m  Path to data directory                \u001b[2m│\u001b[0m\n",
                  "\u001b[2m│\u001b[0m                                        \u001b[2m[default: None]       \u001b[0m                \u001b[2m│\u001b[0m\n",
                  "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-image\u001b[0m               \u001b[1;32m-i\u001b[0m      \u001b[1;33mPATH   \u001b[0m  Save masks as vector or raster        \u001b[2m│\u001b[0m\n",
                  "\u001b[2m│\u001b[0m                                        \u001b[2m[default: vector]             \u001b[0m        \u001b[2m│\u001b[0m\n",
                  "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-env\u001b[0m                 \u001b[1;32m-e\u001b[0m      \u001b[1;33mPATH   \u001b[0m  Path to environment file with         \u001b[2m│\u001b[0m\n",
                  "\u001b[2m│\u001b[0m                                        credentials to cloud bucket: URL,     \u001b[2m│\u001b[0m\n",
                  "\u001b[2m│\u001b[0m                                        ACCESS_KEY, SECRET_KEY, BUCKET,       \u001b[2m│\u001b[0m\n",
                  "\u001b[2m│\u001b[0m                                        REGION                                \u001b[2m│\u001b[0m\n",
                  "\u001b[2m│\u001b[0m                                        \u001b[2m[default: .env]                      \u001b[0m \u001b[2m│\u001b[0m\n",
                  "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-version\u001b[0m             \u001b[1;32m-v\u001b[0m      \u001b[1;33m       \u001b[0m  Print the version and exit            \u001b[2m│\u001b[0m\n",
                  "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-install\u001b[0m\u001b[1;36m-completion\u001b[0m          \u001b[1;33m       \u001b[0m  Install completion for the current    \u001b[2m│\u001b[0m\n",
                  "\u001b[2m│\u001b[0m                                        shell.                                \u001b[2m│\u001b[0m\n",
                  "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-show\u001b[0m\u001b[1;36m-completion\u001b[0m             \u001b[1;33m       \u001b[0m  Show completion for the current       \u001b[2m│\u001b[0m\n",
                  "\u001b[2m│\u001b[0m                                        shell, to copy it or customize the    \u001b[2m│\u001b[0m\n",
                  "\u001b[2m│\u001b[0m                                        installation.                         \u001b[2m│\u001b[0m\n",
                  "\u001b[2m│\u001b[0m \u001b[1;36m-\u001b[0m\u001b[1;36m-help\u001b[0m                        \u001b[1;33m       \u001b[0m  Show this message and exit.           \u001b[2m│\u001b[0m\n",
                  "\u001b[2m╰──────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n",
                  "\n"
               ]
            }
         ],
         "source": [
            "!scaneo --help"
         ]
      },
      {
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "As seen, we have several options in `scaneo` usage, such as selecting the default port to run the server, the host, environment parameters, and so on. In this workshop, what we need is as simple as give the path of our dataset. You can launch `scaneo` with the following command: `scaneo --data workshop_data/sentinel_2`"
         ]
      },
      {
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "![scaneo](./images/scaneo.png)"
         ]
      },
      {
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "Your annotations will be stored alongside the images as GeoJSON files containig the segmentation masks as multipolygons, bounding boxes for detection tasks or classification labels."
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 17,
         "metadata": {},
         "outputs": [
            {
               "name": "stdout",
               "output_type": "stream",
               "text": [
                  "workshop_data/sentinel_2/Boadella_2020-01-13_labels.geojson\n",
                  "workshop_data/sentinel_2/Boadella_2020-02-02_labels.geojson\n"
               ]
            }
         ],
         "source": [
            "!ls workshop_data/sentinel_2/*.geojson"
         ]
      },
      {
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "Once your data is ready you can ingest it to EOTDL like we have seen in the previous notebook and start working with it as any other dataset in the repository."
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 18,
         "metadata": {},
         "outputs": [],
         "source": [
            "import yaml\n",
            "\n",
            "metadata = {\n",
            "\t'authors': ['Fran Martin', 'Juan B. Pedro'],\n",
            "\t'license': 'free',\n",
            "\t'source': 'https://github.com/earthpulse/eotdl/blob/develop/tutorials/workshops/bids23/04_creating.ipynb',\n",
            "\t'name': 'Boadella-BiDS23',\n",
            "}\n",
            "\n",
            "with open('workshop_data/sentinel_2/metadata.yml', 'w') as outfile:\n",
            "\tyaml.dump(metadata, outfile, default_flow_style=False)"
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 19,
         "metadata": {},
         "outputs": [
            {
               "name": "stdout",
               "output_type": "stream",
               "text": [
                  "Uploading directory workshop_data/sentinel_2...\n",
                  "generating list of files to upload...\n"
               ]
            },
            {
               "name": "stderr",
               "output_type": "stream",
               "text": [
                  "100%|██████████| 15/15 [00:00<00:00, 761.59it/s]"
               ]
            },
            {
               "name": "stdout",
               "output_type": "stream",
               "text": [
                  "2 new files will be ingested\n",
                  "13 files already exist in dataset\n",
                  "0 large files will be ingested separately\n"
               ]
            },
            {
               "name": "stderr",
               "output_type": "stream",
               "text": [
                  "\n"
               ]
            },
            {
               "name": "stdout",
               "output_type": "stream",
               "text": [
                  "New version created, version: 3\n",
                  "generating batches...\n"
               ]
            },
            {
               "name": "stderr",
               "output_type": "stream",
               "text": [
                  "100%|██████████| 2/2 [00:00<00:00, 45343.83it/s]\n"
               ]
            },
            {
               "name": "stdout",
               "output_type": "stream",
               "text": [
                  "Uploading 2 small files in 1 batches...\n"
               ]
            },
            {
               "name": "stderr",
               "output_type": "stream",
               "text": [
                  "Uploading batches: 100%|██████████| 1/1 [00:00<00:00,  1.30batches/s]\n",
                  "100%|██████████| 13/13 [00:00<00:00, 228142.06it/s]\n",
                  "Ingesting existing files: 100%|██████████| 2/2 [00:00<00:00,  6.89batches/s]\n"
               ]
            }
         ],
         "source": [
            "from eotdl.datasets import ingest_dataset\n",
            "\n",
            "ingest_dataset('workshop_data/sentinel_2');"
         ]
      },
      {
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "If you add more images or labels to the dataset, you can re-upload and a new version will be automatically generated.\n",
            "\n",
            "However, you might want to wait for the next tutorial where you will find how to generate STAC metadata for this dataset in order to ingest it to the EOTDL as Q1 or Q2 datasets, leveraging advanced functionality."
         ]
      },
      {
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "## Roadmap"
         ]
      },
      {
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "In the future, we plan to add other data sources as well as provide more functionality for data exploration and access. \n",
            "\n",
            "We are also wotking on SCANEO, adding new features and improving the user experience."
         ]
      },
      {
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "## Discussion and Contribution opportunities"
         ]
      },
      {
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "Feel free to ask questions now (live or through Discord) and make suggestions for future improvements.\n",
            "\n",
            "- What features concerning data exploration would you like to see?\n",
            "- What other features concerning data download would you like to see?\n",
            "- What features and tools concerning data preparation would you like to see?\n",
            "- What does your typical workflow look like?\n",
            "- Do you already use any labelling tool?\n",
            "- What does you ideal labelling tool looks like?"
         ]
      }
   ],
   "metadata": {
      "kernelspec": {
         "display_name": "Python 3",
         "language": "python",
         "name": "python3"
      },
      "language_info": {
         "codemirror_mode": {
            "name": "ipython",
            "version": 3
         },
         "file_extension": ".py",
         "mimetype": "text/x-python",
         "name": "python",
         "nbconvert_exporter": "python",
         "pygments_lexer": "ipython3",
         "version": "3.8.18"
      },
      "orig_nbformat": 4
   },
   "nbformat": 4,
   "nbformat_minor": 2
}