soils_revealed_get_baseline_soc.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "soils_revealed_get_baseline_soc.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "mount_file_id": "1zLNCD2knFHppkx6nUWhGOw2hD-DpIL6z",
      "authorship_tag": "ABX9TyMgr6uxTiktViEqKq7umyZo",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Vizzuality/soils-revealed-data/blob/master/soils_revealed_get_baseline_soc.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RTLRdjsHkjtl",
        "colab_type": "text"
      },
      "source": [
        "# Prepare data for the soils-revealed project\n",
        "\n",
        "https://github.com/Vizzuality/soils-revealed-data\n",
        "\n",
        "`Edward P. Morris (vizzuality.)`\n",
        "\n",
        "## Description\n",
        "This notebook downloads the baseline (2000) soils layer for WCS source. \n",
        "\n",
        "```\n",
        "MIT License\n",
        "\n",
        "Copyright (c) 2020 Vizzuality\n",
        "\n",
        "Permission is hereby granted, free of charge, to any person obtaining a copy\n",
        "of this software and associated documentation files (the \"Software\"), to deal\n",
        "in the Software without restriction, including without limitation the rights\n",
        "to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n",
        "copies of the Software, and to permit persons to whom the Software is\n",
        "furnished to do so, subject to the following conditions:\n",
        "\n",
        "The above copyright notice and this permission notice shall be included in all\n",
        "copies or substantial portions of the Software.\n",
        "\n",
        "THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n",
        "IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n",
        "FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n",
        "AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n",
        "LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n",
        "OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n",
        "SOFTWARE.\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uKARM1_f3Ccg",
        "colab_type": "text"
      },
      "source": [
        "# Setup\n",
        "\n",
        "Instructions for setting up the computing environment."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5f327xg8JujU",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%%bash\n",
        "# Remove sample_data\n",
        "rm -r sample_data"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FjdjX-Vc3LVK",
        "colab_type": "text"
      },
      "source": [
        "## Linux dependencies\n",
        "\n",
        "Instructions for adding linux (including node, ect.) system packages. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-5S4mxFswFUs",
        "colab_type": "code",
        "outputId": "07459ce3-f6f4-4748-e1e8-b37e7073c01b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 884
        }
      },
      "source": [
        "# Packages for projections and geospatial processing\n",
        "!apt install -q -y libspatialindex-dev libproj-dev proj-data proj-bin libgeos-dev"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Reading package lists...\n",
            "Building dependency tree...\n",
            "Reading state information...\n",
            "proj-data is already the newest version (4.9.3-2).\n",
            "proj-data set to manually installed.\n",
            "The following additional packages will be installed:\n",
            "  libspatialindex-c4v5 libspatialindex4v5\n",
            "Suggested packages:\n",
            "  libgdal-doc\n",
            "The following NEW packages will be installed:\n",
            "  libgeos-dev libproj-dev libspatialindex-c4v5 libspatialindex-dev\n",
            "  libspatialindex4v5 proj-bin\n",
            "0 upgraded, 6 newly installed, 0 to remove and 31 not upgraded.\n",
            "Need to get 860 kB of archives.\n",
            "After this operation, 5,014 kB of additional disk space will be used.\n",
            "Get:1 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libgeos-dev amd64 3.6.2-1build2 [73.1 kB]\n",
            "Get:2 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libspatialindex4v5 amd64 1.8.5-5 [219 kB]\n",
            "Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libspatialindex-c4v5 amd64 1.8.5-5 [51.7 kB]\n",
            "Get:4 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libproj-dev amd64 4.9.3-2 [199 kB]\n",
            "Get:5 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libspatialindex-dev amd64 1.8.5-5 [285 kB]\n",
            "Get:6 http://archive.ubuntu.com/ubuntu bionic/universe amd64 proj-bin amd64 4.9.3-2 [32.3 kB]\n",
            "Fetched 860 kB in 1s (1,057 kB/s)\n",
            "Selecting previously unselected package libgeos-dev.\n",
            "(Reading database ... 144433 files and directories currently installed.)\n",
            "Preparing to unpack .../0-libgeos-dev_3.6.2-1build2_amd64.deb ...\n",
            "Unpacking libgeos-dev (3.6.2-1build2) ...\n",
            "Selecting previously unselected package libspatialindex4v5:amd64.\n",
            "Preparing to unpack .../1-libspatialindex4v5_1.8.5-5_amd64.deb ...\n",
            "Unpacking libspatialindex4v5:amd64 (1.8.5-5) ...\n",
            "Selecting previously unselected package libspatialindex-c4v5:amd64.\n",
            "Preparing to unpack .../2-libspatialindex-c4v5_1.8.5-5_amd64.deb ...\n",
            "Unpacking libspatialindex-c4v5:amd64 (1.8.5-5) ...\n",
            "Selecting previously unselected package libproj-dev:amd64.\n",
            "Preparing to unpack .../3-libproj-dev_4.9.3-2_amd64.deb ...\n",
            "Unpacking libproj-dev:amd64 (4.9.3-2) ...\n",
            "Selecting previously unselected package libspatialindex-dev:amd64.\n",
            "Preparing to unpack .../4-libspatialindex-dev_1.8.5-5_amd64.deb ...\n",
            "Unpacking libspatialindex-dev:amd64 (1.8.5-5) ...\n",
            "Selecting previously unselected package proj-bin.\n",
            "Preparing to unpack .../5-proj-bin_4.9.3-2_amd64.deb ...\n",
            "Unpacking proj-bin (4.9.3-2) ...\n",
            "Setting up libspatialindex4v5:amd64 (1.8.5-5) ...\n",
            "Setting up libproj-dev:amd64 (4.9.3-2) ...\n",
            "Setting up libgeos-dev (3.6.2-1build2) ...\n",
            "Setting up libspatialindex-c4v5:amd64 (1.8.5-5) ...\n",
            "Setting up proj-bin (4.9.3-2) ...\n",
            "Setting up libspatialindex-dev:amd64 (1.8.5-5) ...\n",
            "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...\n",
            "Processing triggers for libc-bin (2.27-3ubuntu1) ...\n",
            "/sbin/ldconfig.real: /usr/local/lib/python3.6/dist-packages/ideep4py/lib/libmkldnn.so.0 is not a symbolic link\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CO3fFGR4Cw_g",
        "colab_type": "code",
        "outputId": "b46cca61-1c90-42e6-fb39-1afe32b89740",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 119
        }
      },
      "source": [
        "# Fix for curl certificates (rasterio virtual connectors)\n",
        "# RasterioIOError: CURL error: error setting certificate verify locations:   CAfile: /etc/pki/tls/certs/ca-bundle.crt   CApath: none\n",
        "!apt install ca-certificates\n",
        "#!export CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt\n",
        "!mkdir -p /etc/pki/tls/certs\n",
        "!cp /etc/ssl/certs/ca-certificates.crt /etc/pki/tls/certs/ca-bundle.crt\n",
        "!export CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt\n",
        "!ls /etc/pki/tls/certs"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Reading package lists... Done\n",
            "Building dependency tree       \n",
            "Reading state information... Done\n",
            "ca-certificates is already the newest version (20180409).\n",
            "0 upgraded, 0 newly installed, 0 to remove and 31 not upgraded.\n",
            "ca-bundle.crt\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LGWeJje13Tcw",
        "colab_type": "text"
      },
      "source": [
        "## Python packages\n",
        "\n",
        "Consider using package versions to ensure nothing changes.\n",
        "\n",
        "`!pip install -q <package-name>`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2BNK1kif6VXu",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "58739cbe-8054-4b89-fcc8-4d4b97f51434"
      },
      "source": [
        "# connect to Google cloud storage, WebDAV\n",
        "!pip install -q gcsfs webdavclient3"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "  Building wheel for webdavclient3 (setup.py) ... \u001b[?25l\u001b[?25hdone\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2MelOBk55M8T",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 119
        },
        "outputId": "7fd51bb0-2ed7-4537-964b-535e413173fa"
      },
      "source": [
        "# geospatial tools\n",
        "!pip install -q country-converter geopandas owslib"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\u001b[K     |████████████████████████████████| 51kB 1.6MB/s \n",
            "\u001b[K     |████████████████████████████████| 931kB 6.9MB/s \n",
            "\u001b[K     |████████████████████████████████| 204kB 29.1MB/s \n",
            "\u001b[K     |████████████████████████████████| 14.7MB 299kB/s \n",
            "\u001b[K     |████████████████████████████████| 10.9MB 43.1MB/s \n",
            "\u001b[?25h  Building wheel for country-converter (setup.py) ... \u001b[?25l\u001b[?25hdone\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "i-PyPT2y57gc",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 493
        },
        "outputId": "950d8941-a12d-4e24-b3b7-df933bf801bd"
      },
      "source": [
        "# xarray and Zarr tools\n",
        "!pip install -q cftime netcdf4 nc-time-axis zarr xarray xclim rioxarray regionmask sparse xarray-extras"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\u001b[K     |████████████████████████████████| 327kB 2.8MB/s \n",
            "\u001b[K     |████████████████████████████████| 4.1MB 54.3MB/s \n",
            "\u001b[K     |████████████████████████████████| 3.3MB 42.1MB/s \n",
            "\u001b[K     |████████████████████████████████| 143kB 51.3MB/s \n",
            "\u001b[K     |████████████████████████████████| 3.7MB 40.9MB/s \n",
            "\u001b[K     |████████████████████████████████| 122kB 51.6MB/s \n",
            "\u001b[K     |████████████████████████████████| 71kB 10.0MB/s \n",
            "\u001b[K     |████████████████████████████████| 3.8MB 38.4MB/s \n",
            "\u001b[K     |████████████████████████████████| 174kB 50.4MB/s \n",
            "\u001b[K     |████████████████████████████████| 194kB 50.0MB/s \n",
            "\u001b[K     |████████████████████████████████| 71kB 8.2MB/s \n",
            "\u001b[K     |████████████████████████████████| 18.2MB 251kB/s \n",
            "\u001b[K     |████████████████████████████████| 14.4MB 337kB/s \n",
            "\u001b[K     |████████████████████████████████| 645kB 50.5MB/s \n",
            "\u001b[K     |████████████████████████████████| 225kB 50.1MB/s \n",
            "\u001b[K     |████████████████████████████████| 102kB 11.9MB/s \n",
            "\u001b[?25h  Building wheel for zarr (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Building wheel for rioxarray (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Building wheel for xarray-extras (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Building wheel for asciitree (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Building wheel for numcodecs (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Building wheel for rtree (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Building wheel for cartopy (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Building wheel for pyshp (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Building wheel for locket (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Building wheel for contextvars (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "\u001b[31mERROR: distributed 2.17.0 has requirement tornado>=5; python_version < \"3.8\", but you'll have tornado 4.5.3 which is incompatible.\u001b[0m\n",
            "\u001b[31mERROR: sparse 0.10.0 has requirement numba>=0.49, but you'll have numba 0.48.0 which is incompatible.\u001b[0m\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8s7-DYbxjesM",
        "colab_type": "code",
        "outputId": "45b1705f-1e78-4336-fa15-801e467f17a5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "# Show python package versions\n",
        "!pip list"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Package                  Version        \n",
            "------------------------ ---------------\n",
            "absl-py                  0.9.0          \n",
            "affine                   2.3.0          \n",
            "alabaster                0.7.12         \n",
            "albumentations           0.1.12         \n",
            "altair                   4.1.0          \n",
            "asciitree                0.3.3          \n",
            "asgiref                  3.2.7          \n",
            "astor                    0.8.1          \n",
            "astropy                  4.0.1.post1    \n",
            "astunparse               1.6.3          \n",
            "atari-py                 0.2.6          \n",
            "atomicwrites             1.4.0          \n",
            "attrs                    19.3.0         \n",
            "audioread                2.1.8          \n",
            "autograd                 1.3            \n",
            "Babel                    2.8.0          \n",
            "backcall                 0.1.0          \n",
            "beautifulsoup4           4.6.3          \n",
            "bleach                   3.1.5          \n",
            "blis                     0.4.1          \n",
            "bokeh                    1.4.0          \n",
            "boltons                  20.1.0         \n",
            "boto                     2.49.0         \n",
            "boto3                    1.13.13        \n",
            "botocore                 1.16.13        \n",
            "Bottleneck               1.3.2          \n",
            "branca                   0.4.1          \n",
            "bs4                      0.0.1          \n",
            "CacheControl             0.12.6         \n",
            "cachetools               3.1.1          \n",
            "Cartopy                  0.18.0         \n",
            "catalogue                1.0.0          \n",
            "certifi                  2020.4.5.1     \n",
            "cffi                     1.14.0         \n",
            "cftime                   1.1.3          \n",
            "chainer                  6.5.0          \n",
            "chardet                  3.0.4          \n",
            "click                    7.1.2          \n",
            "click-plugins            1.1.1          \n",
            "cligj                    0.5.0          \n",
            "cloudpickle              1.3.0          \n",
            "cmake                    3.12.0         \n",
            "cmdstanpy                0.4.0          \n",
            "colorlover               0.3.0          \n",
            "community                1.0.0b1        \n",
            "contextlib2              0.5.5          \n",
            "contextvars              2.4            \n",
            "convertdate              2.2.1          \n",
            "country-converter        0.6.7          \n",
            "coverage                 3.7.1          \n",
            "coveralls                0.5            \n",
            "crcmod                   1.7            \n",
            "cufflinks                0.17.3         \n",
            "cvxopt                   1.2.5          \n",
            "cvxpy                    1.0.31         \n",
            "cycler                   0.10.0         \n",
            "cymem                    2.0.3          \n",
            "Cython                   0.29.18        \n",
            "daft                     0.0.4          \n",
            "dask                     2.12.0         \n",
            "dataclasses              0.7            \n",
            "datascience              0.10.6         \n",
            "decorator                4.4.2          \n",
            "defusedxml               0.6.0          \n",
            "descartes                1.1.0          \n",
            "dill                     0.3.1.1        \n",
            "distributed              2.17.0         \n",
            "Django                   3.0.6          \n",
            "dlib                     19.18.0        \n",
            "docopt                   0.6.2          \n",
            "docutils                 0.15.2         \n",
            "dopamine-rl              1.0.5          \n",
            "earthengine-api          0.1.221        \n",
            "easydict                 1.9            \n",
            "ecos                     2.0.7.post1    \n",
            "editdistance             0.5.3          \n",
            "en-core-web-sm           2.2.5          \n",
            "entrypoints              0.3            \n",
            "ephem                    3.7.7.1        \n",
            "et-xmlfile               1.0.1          \n",
            "fa2                      0.3.5          \n",
            "fancyimpute              0.4.3          \n",
            "fastai                   1.0.61         \n",
            "fastdtw                  0.3.4          \n",
            "fasteners                0.15           \n",
            "fastprogress             0.2.3          \n",
            "fastrlock                0.4            \n",
            "fbprophet                0.6            \n",
            "feather-format           0.4.1          \n",
            "featuretools             0.4.1          \n",
            "filelock                 3.0.12         \n",
            "Fiona                    1.8.13.post1   \n",
            "firebase-admin           4.1.0          \n",
            "fix-yahoo-finance        0.0.22         \n",
            "Flask                    1.1.2          \n",
            "folium                   0.8.3          \n",
            "fsspec                   0.7.4          \n",
            "future                   0.16.0         \n",
            "gast                     0.3.3          \n",
            "gcsfs                    0.6.2          \n",
            "GDAL                     2.2.2          \n",
            "gdown                    3.6.4          \n",
            "gensim                   3.6.0          \n",
            "geographiclib            1.50           \n",
            "geojson                  2.5.0          \n",
            "geopandas                0.7.0          \n",
            "geopy                    1.17.0         \n",
            "gin-config               0.3.0          \n",
            "glob2                    0.7            \n",
            "google                   2.0.3          \n",
            "google-api-core          1.16.0         \n",
            "google-api-python-client 1.7.12         \n",
            "google-auth              1.7.2          \n",
            "google-auth-httplib2     0.0.3          \n",
            "google-auth-oauthlib     0.4.1          \n",
            "google-cloud-bigquery    1.21.0         \n",
            "google-cloud-core        1.0.3          \n",
            "google-cloud-datastore   1.8.0          \n",
            "google-cloud-firestore   1.7.0          \n",
            "google-cloud-language    1.2.0          \n",
            "google-cloud-storage     1.18.1         \n",
            "google-cloud-translate   1.5.0          \n",
            "google-colab             1.0.0          \n",
            "google-pasta             0.2.0          \n",
            "google-resumable-media   0.4.1          \n",
            "googleapis-common-protos 1.51.0         \n",
            "googledrivedownloader    0.4            \n",
            "graphviz                 0.10.1         \n",
            "grpcio                   1.29.0         \n",
            "gspread                  3.0.1          \n",
            "gspread-dataframe        3.0.6          \n",
            "gym                      0.17.2         \n",
            "h5py                     2.10.0         \n",
            "HeapDict                 1.0.1          \n",
            "holidays                 0.9.12         \n",
            "html5lib                 1.0.1          \n",
            "httpimport               0.5.18         \n",
            "httplib2                 0.17.3         \n",
            "httplib2shim             0.0.3          \n",
            "humanize                 0.5.1          \n",
            "hyperopt                 0.1.2          \n",
            "ideep4py                 2.0.0.post3    \n",
            "idna                     2.9            \n",
            "image                    1.5.31         \n",
            "imagecodecs              2020.2.18      \n",
            "imageio                  2.4.1          \n",
            "imagesize                1.2.0          \n",
            "imbalanced-learn         0.4.3          \n",
            "imblearn                 0.0            \n",
            "imgaug                   0.2.9          \n",
            "immutables               0.14           \n",
            "importlib-metadata       1.6.0          \n",
            "imutils                  0.5.3          \n",
            "inflect                  2.1.0          \n",
            "intel-openmp             2020.0.133     \n",
            "intervaltree             2.1.0          \n",
            "ipykernel                4.10.1         \n",
            "ipython                  5.5.0          \n",
            "ipython-genutils         0.2.0          \n",
            "ipython-sql              0.3.9          \n",
            "ipywidgets               7.5.1          \n",
            "itsdangerous             1.1.0          \n",
            "jax                      0.1.67         \n",
            "jaxlib                   0.1.47         \n",
            "jdcal                    1.4.1          \n",
            "jedi                     0.17.0         \n",
            "jieba                    0.42.1         \n",
            "Jinja2                   2.11.2         \n",
            "jmespath                 0.10.0         \n",
            "joblib                   0.15.1         \n",
            "jpeg4py                  0.1.4          \n",
            "jsonschema               2.6.0          \n",
            "jupyter                  1.0.0          \n",
            "jupyter-client           5.3.4          \n",
            "jupyter-console          5.2.0          \n",
            "jupyter-core             4.6.3          \n",
            "kaggle                   1.5.6          \n",
            "kapre                    0.1.3.1        \n",
            "Keras                    2.3.1          \n",
            "Keras-Applications       1.0.8          \n",
            "Keras-Preprocessing      1.1.2          \n",
            "keras-vis                0.4.1          \n",
            "kiwisolver               1.2.0          \n",
            "knnimpute                0.1.0          \n",
            "librosa                  0.6.3          \n",
            "lightgbm                 2.2.3          \n",
            "llvmlite                 0.31.0         \n",
            "lmdb                     0.98           \n",
            "locket                   0.2.0          \n",
            "lucid                    0.3.8          \n",
            "LunarCalendar            0.0.9          \n",
            "lxml                     4.2.6          \n",
            "Markdown                 3.2.2          \n",
            "MarkupSafe               1.1.1          \n",
            "matplotlib               3.2.1          \n",
            "matplotlib-venn          0.11.5         \n",
            "missingno                0.4.2          \n",
            "mistune                  0.8.4          \n",
            "mizani                   0.6.0          \n",
            "mkl                      2019.0         \n",
            "mlxtend                  0.14.0         \n",
            "monotonic                1.5            \n",
            "more-itertools           8.3.0          \n",
            "moviepy                  0.2.3.5        \n",
            "mpmath                   1.1.0          \n",
            "msgpack                  1.0.0          \n",
            "multiprocess             0.70.9         \n",
            "multitasking             0.0.9          \n",
            "munch                    2.5.0          \n",
            "murmurhash               1.0.2          \n",
            "music21                  5.5.0          \n",
            "natsort                  5.5.0          \n",
            "nbconvert                5.6.1          \n",
            "nbformat                 5.0.6          \n",
            "nc-time-axis             1.2.0          \n",
            "netCDF4                  1.5.3          \n",
            "networkx                 2.4            \n",
            "nibabel                  3.0.2          \n",
            "nltk                     3.2.5          \n",
            "notebook                 5.2.2          \n",
            "np-utils                 0.5.12.1       \n",
            "numba                    0.48.0         \n",
            "numcodecs                0.6.4          \n",
            "numexpr                  2.7.1          \n",
            "numpy                    1.18.4         \n",
            "nvidia-ml-py3            7.352.0        \n",
            "oauth2client             4.1.3          \n",
            "oauthlib                 3.1.0          \n",
            "okgrade                  0.4.3          \n",
            "opencv-contrib-python    4.1.2.30       \n",
            "opencv-python            4.1.2.30       \n",
            "openpyxl                 2.5.9          \n",
            "opt-einsum               3.2.1          \n",
            "osqp                     0.6.1          \n",
            "OWSLib                   0.19.2         \n",
            "packaging                20.4           \n",
            "palettable               3.3.0          \n",
            "pandas                   1.0.3          \n",
            "pandas-datareader        0.8.1          \n",
            "pandas-gbq               0.11.0         \n",
            "pandas-profiling         1.4.1          \n",
            "pandocfilters            1.4.2          \n",
            "parso                    0.7.0          \n",
            "partd                    1.1.0          \n",
            "pathlib                  1.0.1          \n",
            "patsy                    0.5.1          \n",
            "pexpect                  4.8.0          \n",
            "pickleshare              0.7.5          \n",
            "Pillow                   7.0.0          \n",
            "Pint                     0.11           \n",
            "pip                      19.3.1         \n",
            "pip-tools                4.5.1          \n",
            "plac                     1.1.3          \n",
            "plotly                   4.4.1          \n",
            "plotnine                 0.6.0          \n",
            "pluggy                   0.7.1          \n",
            "portpicker               1.3.1          \n",
            "prefetch-generator       1.0.1          \n",
            "preshed                  3.0.2          \n",
            "prettytable              0.7.2          \n",
            "progressbar2             3.38.0         \n",
            "prometheus-client        0.7.1          \n",
            "promise                  2.3            \n",
            "prompt-toolkit           1.0.18         \n",
            "protobuf                 3.10.0         \n",
            "psutil                   5.4.8          \n",
            "psycopg2                 2.7.6.1        \n",
            "ptvsd                    5.0.0a12       \n",
            "ptyprocess               0.6.0          \n",
            "py                       1.8.1          \n",
            "pyarrow                  0.14.1         \n",
            "pyasn1                   0.4.8          \n",
            "pyasn1-modules           0.2.8          \n",
            "pycocotools              2.0.0          \n",
            "pycparser                2.20           \n",
            "pydata-google-auth       1.1.0          \n",
            "pydot                    1.3.0          \n",
            "pydot-ng                 2.0.0          \n",
            "pydotplus                2.0.2          \n",
            "PyDrive                  1.3.1          \n",
            "pyemd                    0.5.1          \n",
            "pyglet                   1.5.0          \n",
            "Pygments                 2.1.3          \n",
            "pygobject                3.26.1         \n",
            "pymc3                    3.7            \n",
            "PyMeeus                  0.3.7          \n",
            "pymongo                  3.10.1         \n",
            "pymystem3                0.2.0          \n",
            "PyOpenGL                 3.1.5          \n",
            "pyparsing                2.4.7          \n",
            "pyproj                   2.6.1.post1    \n",
            "pyrsistent               0.16.0         \n",
            "pyshp                    2.1.0          \n",
            "pysndfile                1.3.8          \n",
            "PySocks                  1.7.1          \n",
            "pystan                   2.19.1.1       \n",
            "pytest                   3.6.4          \n",
            "python-apt               1.6.5+ubuntu0.2\n",
            "python-chess             0.23.11        \n",
            "python-dateutil          2.8.1          \n",
            "python-louvain           0.14           \n",
            "python-slugify           4.0.0          \n",
            "python-utils             2.4.0          \n",
            "pytz                     2018.9         \n",
            "PyWavelets               1.1.1          \n",
            "PyYAML                   3.13           \n",
            "pyzmq                    19.0.1         \n",
            "qtconsole                4.7.4          \n",
            "QtPy                     1.9.0          \n",
            "rasterio                 1.1.4          \n",
            "regex                    2019.12.20     \n",
            "regionmask               0.5.0          \n",
            "requests                 2.23.0         \n",
            "requests-oauthlib        1.3.0          \n",
            "resampy                  0.2.2          \n",
            "retrying                 1.3.3          \n",
            "rioxarray                0.0.26         \n",
            "rpy2                     3.2.7          \n",
            "rsa                      4.0            \n",
            "Rtree                    0.9.4          \n",
            "s3fs                     0.4.2          \n",
            "s3transfer               0.3.3          \n",
            "scikit-image             0.16.2         \n",
            "scikit-learn             0.22.2.post1   \n",
            "scipy                    1.4.1          \n",
            "screen-resolution-extra  0.0.0          \n",
            "scs                      2.1.2          \n",
            "seaborn                  0.10.1         \n",
            "Send2Trash               1.5.0          \n",
            "setuptools               46.3.0         \n",
            "setuptools-git           1.2            \n",
            "Shapely                  1.7.0          \n",
            "simplegeneric            0.8.1          \n",
            "six                      1.12.0         \n",
            "sklearn                  0.0            \n",
            "sklearn-pandas           1.8.0          \n",
            "smart-open               2.0.0          \n",
            "snowballstemmer          2.0.0          \n",
            "snuggs                   1.4.7          \n",
            "sortedcontainers         2.1.0          \n",
            "spacy                    2.2.4          \n",
            "sparse                   0.10.0         \n",
            "Sphinx                   1.8.5          \n",
            "sphinxcontrib-websupport 1.2.2          \n",
            "SQLAlchemy               1.3.17         \n",
            "sqlparse                 0.3.1          \n",
            "srsly                    1.0.2          \n",
            "statsmodels              0.10.2         \n",
            "sympy                    1.1.1          \n",
            "tables                   3.4.4          \n",
            "tabulate                 0.8.7          \n",
            "tbb                      2020.0.133     \n",
            "tblib                    1.6.0          \n",
            "tensorboard              2.2.1          \n",
            "tensorboard-plugin-wit   1.6.0.post3    \n",
            "tensorboardcolab         0.0.22         \n",
            "tensorflow               2.2.0          \n",
            "tensorflow-addons        0.8.3          \n",
            "tensorflow-datasets      2.1.0          \n",
            "tensorflow-estimator     2.2.0          \n",
            "tensorflow-gcs-config    2.1.8          \n",
            "tensorflow-hub           0.8.0          \n",
            "tensorflow-metadata      0.22.0         \n",
            "tensorflow-privacy       0.2.2          \n",
            "tensorflow-probability   0.10.0         \n",
            "termcolor                1.1.0          \n",
            "terminado                0.8.3          \n",
            "testpath                 0.4.4          \n",
            "text-unidecode           1.3            \n",
            "textblob                 0.15.3         \n",
            "textgenrnn               1.4.1          \n",
            "Theano                   1.0.4          \n",
            "thinc                    7.4.0          \n",
            "tifffile                 2020.5.11      \n",
            "toolz                    0.10.0         \n",
            "torch                    1.5.0+cu101    \n",
            "torchsummary             1.5.1          \n",
            "torchtext                0.3.1          \n",
            "torchvision              0.6.0+cu101    \n",
            "tornado                  4.5.3          \n",
            "tqdm                     4.41.1         \n",
            "traitlets                4.3.3          \n",
            "tweepy                   3.6.0          \n",
            "typeguard                2.7.1          \n",
            "typing                   3.6.6          \n",
            "typing-extensions        3.6.6          \n",
            "tzlocal                  1.5.1          \n",
            "umap-learn               0.4.3          \n",
            "uritemplate              3.0.1          \n",
            "urllib3                  1.24.3         \n",
            "vega-datasets            0.8.0          \n",
            "wasabi                   0.6.0          \n",
            "wcwidth                  0.1.9          \n",
            "webdavclient3            3.14.4         \n",
            "webencodings             0.5.1          \n",
            "Werkzeug                 1.0.1          \n",
            "wheel                    0.34.2         \n",
            "widgetsnbextension       3.5.1          \n",
            "wordcloud                1.5.0          \n",
            "wrapt                    1.12.1         \n",
            "xarray                   0.15.1         \n",
            "xarray-extras            0.4.2          \n",
            "xclim                    0.17.0         \n",
            "xgboost                  0.90           \n",
            "xkit                     0.0.0          \n",
            "xlrd                     1.1.0          \n",
            "xlwt                     1.3.0          \n",
            "yellowbrick              0.9.1          \n",
            "zarr                     2.4.0          \n",
            "zict                     2.0.0          \n",
            "zipp                     3.1.0          \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IK4kYzml3bDD",
        "colab_type": "text"
      },
      "source": [
        "## Authorisation\n",
        "\n",
        "Setting up connections and authorisation to cloud services."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T7jp5JdFrxGm",
        "colab_type": "text"
      },
      "source": [
        "### Google Cloud\n",
        "\n",
        "This can be done in the URL or via adding service account credentials.\n",
        "\n",
        "If you do not share the notebook, you can mount your Drive and and transfer credentials to disk. Note if the notebook is shared you always need to authenticate via URL.  "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pIeVojTV21Nr",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Set the Google Cloud project id\n",
        "project_id = 'soc-platform'\n",
        "gc_creds = \"soc-platform-6a9bf204638c.json\"\n",
        "username = \"edward-morris-vizzuality@soc-platform.iam.gserviceaccount.com\"\n",
        "gcs_prefix = \"gs://vizz-data-transfer\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "oY_ymKY_oIY8",
        "colab": {}
      },
      "source": [
        "# For auth WITHOUT service account\n",
        "# https://cloud.google.com/resource-manager/docs/creating-managing-projects\n",
        "#from google.colab import auth\n",
        "#auth.authenticate_user()\n",
        "#!gcloud config set project {project_id}"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7akT-7lZ9x3R",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# If the notebook is shared\n",
        "#from google.colab import drive\n",
        "#drive.mount('/content/drive')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "dCFS8FOnzScr",
        "colab": {}
      },
      "source": [
        "# If Drive is mounted, copy GC credentials to home (place in your GDrive, and connect Drive)\n",
        "!cp \"/content/drive/My Drive/{gc_creds}\" \"/root/.{gc_creds}\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "03Tbqeq9zSc0",
        "outputId": "b8b06d6a-2e84-4d1c-eae2-ed076bff705f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "# Auth WITH service account\n",
        "!gcloud auth activate-service-account {username} --key-file=/root/.{gc_creds} --project={project_id}\n"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Activated service account credentials for: [edward-morris-vizzuality@soc-platform.iam.gserviceaccount.com]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "PA4tPZ4-zSc9",
        "outputId": "faac2653-f9aa-4f71-c37a-047072e3fb11",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        }
      },
      "source": [
        "# Test GC auth\n",
        "!gsutil ls {gcs_prefix}"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "gs://vizz-data-transfer/test-cog.tif\n",
            "gs://vizz-data-transfer/SOC_maps/\n",
            "gs://vizz-data-transfer/land-cover/\n",
            "gs://vizz-data-transfer/movie-tiles/\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vb3bFXTp0JFz",
        "colab_type": "text"
      },
      "source": [
        "# Utils\n",
        "\n",
        "Generic helper functions used in the subsequent processing. For easy navigation each function seperated into a section with the function name."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-l-NVtcE8KdV",
        "colab_type": "text"
      },
      "source": [
        "## copy_gcs"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "h1KygWllicHk",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import os\n",
        "import subprocess\n",
        "\n",
        "def copy_gcs(source_list, dest_list, opts=\"\"):\n",
        "  \"\"\"\n",
        "  Use gsutil to copy each corresponding item in source_list\n",
        "  to dest_list.\n",
        "\n",
        "  Example:\n",
        "  copy_gcs([\"gs://my-bucket/data-file.csv\"], [\".\"])\n",
        "\n",
        "  \"\"\"\n",
        "  for s, d  in zip(source_list, dest_list):\n",
        "    cmd = f\"gsutil -m cp -r {opts} {s} {d}\"\n",
        "    print(f\"Processing: {cmd}\")\n",
        "    r = subprocess.call(cmd, shell=True)\n",
        "    if r == 0:\n",
        "        print(\"Task created\")\n",
        "    else:\n",
        "        print(\"Task failed\")\n",
        "  print(\"Finished copy\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9PZ2mMKo8hYW",
        "colab_type": "text"
      },
      "source": [
        "## get_image_meta"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-uuBXl2wYnDw",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Get image metadata\n",
        "# FIXME is MultiPolygon the correct way to store footprint?\n",
        "# FIXME can the footprint be simplified?\n",
        "# FIXME what happens with complex NA image structures?\n",
        "import rasterio\n",
        "import os\n",
        "from collections import namedtuple\n",
        "\n",
        "def get_image_meta(uri_prefix, file_path, gcs_creds=False):\n",
        "  \"\"\"\n",
        "  Get image metadata\n",
        "\n",
        "  Uses rasterio, which can handle virtual files systems\n",
        "  \n",
        "  Arguments\n",
        "  ---------\n",
        "  uri_prefix : str\n",
        "      The uri prefix string.  \n",
        "  file_path : str\n",
        "      The file path of the asset.\n",
        "  gcs_creds : str\n",
        "      Path to GCS credentials JSON. #FIXME! \n",
        "\n",
        "  Returns\n",
        "  ------- \n",
        "  A named tuple of image properties formatted for earthengine manifest\n",
        "  \"\"\"\n",
        "  # Set GCS creds\n",
        "  if gcs_creds:\n",
        "    # FIXME!\n",
        "    with open(\"adc.json\", \"r\") as f:\n",
        "            gcs_creds = json.load(f)\n",
        "    #print(gcs_creds)\n",
        "    os.environ['GS_SECRET_ACCESS_KEY'] = gcs_creds.get('client_secret')\n",
        "    os.environ['GS_ACCESS_KEY_ID'] = gcs_creds.get('client_id')\n",
        "    #print(os.environ['GS_ACCESS_KEY_ID'])\n",
        "  \n",
        "    # hack for ubuntu/debian\n",
        "    os.environ['CURL_CA_BUNDLE'] = \"/etc/ssl/certs/ca-certificates.crt\"\n",
        "\n",
        "  # Make path\n",
        "  p = os.path.join(uri_prefix,file_path)\n",
        "  #print(p)\n",
        "  \n",
        "  # Get metadata\n",
        "  with rasterio.open(p, 'r') as src:\n",
        "    \n",
        "    # band info\n",
        "    Band = namedtuple('Band', ['idx', 'dtype', 'description', 'units', 'nodata'])\n",
        "    bands = [Band(idx=tmp1, dtype=tmp2.swapcase(), description=tmp3, units=tmp4, nodata=tmp5) for tmp1, tmp2, tmp3, tmp4, tmp5 in zip(\n",
        "    src.indexes, src.dtypes, src.descriptions, src.units, src.nodatavals)]\n",
        "\n",
        "    # affine transform\n",
        "    def get_affine_transform(tmp2):\n",
        "      # FIXME check mapping!\n",
        "      AffineTransform = namedtuple('AffineTransform', [\"scale_x\", \"shear_x\", \"translate_x\", \"shear_y\", \"scale_y\", \"translate_y\"])\n",
        "      return AffineTransform(**{\n",
        "          \"scale_x\":tmp2[0],\n",
        "           \"shear_x\":tmp2[1],\n",
        "            \"translate_x\":tmp2[2],\n",
        "             \"shear_y\":tmp2[3],\n",
        "              \"scale_y\":tmp2[4],\n",
        "               \"translate_y\":tmp2[5]})\n",
        "    aft = get_affine_transform([tmp2 for tmp2 in src.transform.to_gdal()])\n",
        "    \n",
        "    # crs\n",
        "    CRS = namedtuple('CRS', ['epsg'])\n",
        "    crs = CRS(epsg=src.crs.to_epsg())\n",
        "    \n",
        "    # tags\n",
        "    tags = src.tags()\n",
        "\n",
        "    # media type (driver)\n",
        "    mt = src.driver\n",
        "    \n",
        "  src.close()\n",
        "  imageMeta = namedtuple('imageMeta', [\"bands\", \"affine_transform\", \"crs\", \"properties\", \"mtype\"])\n",
        "  return(imageMeta(**{\"bands\":bands, \"affine_transform\":aft, \"crs\":crs, \"properties\":tags, \"mtype\":mt}))\n",
        "\n",
        "# Examples\n",
        "# --------\n",
        "# Singleband image\n",
        "#uri_prefix = \"https://storage.googleapis.com/skydipper-water-quality\"\n",
        "#file_path = \"/cloud-masks/SENTINEL_2_reference_cloud_masks_Baetens_Hagolle/S2A_MSIL1C_20161221T085352_N0204_R107_T33KWP_20161221T091140.tif\"\n",
        "#print(get_image_meta(uri_prefix, file_path))\n",
        "\n",
        "# Multiban#d image\n",
        "#uri_prefix = \"https://storage.googleapis.com/skydipper_materials\"\n",
        "#file_path = \"/gee_data/L7_AOI_00.tif\"\n",
        "#print(get_image_meta(uri_prefix, file_path))\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ikHLJlY76f1F",
        "colab_type": "text"
      },
      "source": [
        "## print_ds_size"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "s8SH12rl6me9",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def print_ds_size(ds):\n",
        "  print('ds size in GB {:0.2f}\\n'.format(ds.nbytes / 1e9))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GBPS3DALtie0",
        "colab_type": "text"
      },
      "source": [
        "# Processing\n",
        "\n",
        "Data processing organised into sections."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jMBZ7SwItthj",
        "colab_type": "text"
      },
      "source": [
        "## Get datasets"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dsbSbhGZrnsE",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import os\n",
        "import subprocess\n",
        "\n",
        "# Examine datasets\n",
        "uri_prefix = \"https://files.isric.org/soilgrids/data/recent/ocs\"\n",
        "file_paths = [\"ocs_0-30cm_mean.vrt\", \"ocs_0-30cm_Q0.5.vrt\", \"ocs_0-30cm_Q0.05.vrt\", \"ocs_0-30cm_Q0.95.vrt\", \"ocs_0-30cm_uncertainty.vrt\"]  \n",
        "\n",
        "def get_meta(file_paths):\n",
        "  return [(get_image_meta(uri_prefix, fp, gcs_creds=False),\\\n",
        "        subprocess.getoutput(f\"rio info {os.path.join(uri_prefix, fp)}\"),\\\n",
        "        subprocess.getoutput(f\"gdalinfo {os.path.join(uri_prefix, fp)}\")) for fp in file_paths]\n",
        "\n",
        "meta = get_meta(file_paths)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "34Kn0PgTEWeV",
        "colab_type": "code",
        "outputId": "d2dfb696-7a1d-42e8-d7f5-f74ef26d6f89",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 88
        }
      },
      "source": [
        "meta[0]"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(imageMeta(bands=[Band(idx=1, dtype='INT16', description=None, units=None, nodata=-32768.0)], affine_transform=AffineTransform(scale_x=-19949750.0, shear_x=250.0, translate_x=0.0, shear_y=8361000.0, scale_y=0.0, translate_y=-250.0), crs=CRS(epsg=None), properties={'Code_version': 'v2.0.0', 'Covariates': 'clm_wcl_srdyrsum,clm_wcl_s04rad,clm_wcl_p10tot,veg_mod_eviyravg,clm_wcl_p09tot,veg_mod_nppy15,mor_env_demm,clm_wcl_s08rad,clm_wcl_srdyrstd,clm_wcl_s07rad,clm_wcl_bio19,clm_wcl_p05tot,clm_wcl_bio17,mor_mrg_twi,clm_mod_lstd03std,clm_mod_lstd04std,clm_wcl_bio16,clm_wcl_bio08,clm_mod_lstd09std,clm_mod_lstd11std,clm_wcl_bio18,mor_mrg_vdp,clm_wcl_p04tot,clm_mod_lstd05std,clm_mod_lstd12std,clm_mod_lstd10std,clm_mod_lstd02std,clm_wcl_p12tot,clm_mod_lstd01std,veg_mod_evient,veg_mod_evirng,veg_mod_evimax,veg_mod_evievn,luc_gfc_trely10,mor_mrg_vbf', 'Litter_layers': 'FALSE', 'Model': 'Quantile Regression Forests', 'Model_type': 'ranger', 'Mtry': '15', 'Number_trees': '200', 'Other_synth_profiles': 'the_map.tiff', 'Outputs_version': 'RUN03', 'WoSIS_version': 'Data stream 7'}, mtype='VRT'),\n",
              " '{\"blockxsize\": 128, \"blockysize\": 128, \"bounds\": [-19949750.0, -6147500.0, 19861750.0, 8361000.0], \"colorinterp\": [\"gray\"], \"count\": 1, \"crs\": \"PROJCS[\\\\\"Interrupted_Goode_Homolosine\\\\\",GEOGCS[\\\\\"GCS_unnamed ellipse\\\\\",DATUM[\\\\\"D_unknown\\\\\",SPHEROID[\\\\\"Unknown\\\\\",6378137,298.257223563]],PRIMEM[\\\\\"Greenwich\\\\\",0],UNIT[\\\\\"Degree\\\\\",0.017453292519943295]],PROJECTION[\\\\\"Interrupted_Goode_Homolosine\\\\\"],UNIT[\\\\\"Meter\\\\\",1]]\", \"descriptions\": [null], \"driver\": \"VRT\", \"dtype\": \"int16\", \"height\": 58034, \"indexes\": [1], \"lnglat\": [-0.8586721604776795, 9.9421044069928], \"mask_flags\": [[\"nodata\"]], \"nodata\": -32768.0, \"res\": [250.0, 250.0], \"shape\": [58034, 159246], \"tiled\": true, \"transform\": [250.0, 0.0, -19949750.0, 0.0, -250.0, 8361000.0, 0.0, 0.0, 1.0], \"units\": [null], \"width\": 159246}',\n",
              " \"ERROR 1: An error occurred while creating a virtual connection to the DAP server:\\nError while reading the URL: https://files.isric.org/soilgrids/data/recent/ocs/ocs_0-30cm_mean.vrt.ver.\\nThe OPeNDAP server returned the following message:\\nNot Found: The data source or server could not be found.\\n        Often this means that the OPeNDAP server is missing or needs attention;\\n        Please contact the server administrator.\\ngdalinfo failed - unable to open 'https://files.isric.org/soilgrids/data/recent/ocs/ocs_0-30cm_mean.vrt'.\")"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sacs_CebssZg",
        "colab_type": "code",
        "outputId": "932debb2-7928-4b5f-f2e7-48bb18edb9f2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 442
        }
      },
      "source": [
        "# Create dataset (note this just creates the structure, ie., does not get data yet)\n",
        "import xarray as xr\n",
        "import rioxarray as rxr\n",
        "import os\n",
        "import pandas as pd\n",
        "\n",
        "# Time coordinate\n",
        "# check if this should be CF defined?\n",
        "time = pd.to_datetime(['2000-01-01'])\n",
        "\n",
        "# Variable attributes\n",
        "# see https://www.nodc.noaa.gov/data/formats/netcdf/v2.0/ for guide to attributes \n",
        "attrs_list = [\n",
        "{\n",
        "    \"standard_name\": \"mean_of_mass_content_of_organic_carbon_in_zero_to_thirty_centimeter_depth_soil_layer\",\n",
        "    \"long_name\": \"Soil organic content (mean, 0-30cm)\",\n",
        "    \"units\": \"t/ha\",\n",
        "    \"description\": \"The mean vertically integrated mass content (or stock) [M/A] of organic carbon in the 0 to 30 cm depth soil layer.\"\n",
        "},\n",
        "{\n",
        "    \"standard_name\": \"median_of_mass_content_of_organic_carbon_in_zero_to_thirty_centimeter_depth_soil_layer\",\n",
        "    \"long_name\": \"Soil organic content (median, 0-30cm)\",\n",
        "    \"units\": \"t/ha\",\n",
        "    \"description\": \"The median of vertically integrated mass content (or stock) [M/A] of organic carbon in the 0 to 30 cm depth soil layer.\"\n",
        "},\n",
        "{\n",
        "    \"standard_name\": \"five_percent_quantile_of_mass_content_of_organic_carbon_in_zero_to_thirty_centimeter_depth_soil_layer\",\n",
        "    \"long_name\": \"Soil organic content (Q0.05, 0-30cm)\",\n",
        "    \"units\": \"t/ha\",\n",
        "    \"description\": \"The 0.05 quantile of vertically integrated mass content (or stock) [M/A] of organic carbon in the 0 to 30 cm depth soil layer.\"\n",
        "},\n",
        "{\n",
        "    \"standard_name\": \"ninety_five_percent_quantile_of_mass_content_of_organic_carbon_in_zero_to_thirty_centimeter_depth_soil_layer\",\n",
        "    \"long_name\": \"Soil organic content (Q0.95, 0-30cm)\",\n",
        "    \"units\": \"t/ha\",\n",
        "    \"description\": \"The 0.95 quantile of vertically integrated mass content (or stock) [M/A] of organic carbon in the 0 to 30 cm depth soil layer.\"\n",
        "},\n",
        "{\n",
        "    \"standard_name\": \"uncertainity_of_mass_content_of_organic_carbon_in_zero_to_thirty_centimeter_depth_soil_layer\",\n",
        "    \"long_name\": \"Soil organic content (uncertainity, 0-30cm)\",\n",
        "    \"units\": \"t/ha\",\n",
        "    \"description\": \"The uncertainity of vertically integrated mass content (or stock) [M/A] of organic carbon in the 0 to 30 cm depth soil layer.\"\n",
        "}]\n",
        "\n",
        "# Dataset attributes\n",
        "attrs = {\n",
        "    \"Conventions\": \"CF-1.6\",\n",
        "    \"title\": \"Global gridded soil properties\",\n",
        "    \"description\": \"Global spatial distribution of soil properties mapped using state-of-the-art machine learning methods.\",\n",
        "    \"institution\": \"ISRIC\",\n",
        "    \"license\" : \"[CC-BY 4.0 License](https://creativecommons.org/licenses/by/4.0/)\",\n",
        "    \"comment\" : \"SoilGrids is a system for global digital soil mapping that uses state-of-the-art machine learning methods to map the spatial distribution of soil properties across the globe. SoilGrids prediction models are fitted using over 230 000 soil profile observations from the WoSIS database and a series of environmental covariates. Covariates were selected from a pool of over 400 environmental layers from Earth observation derived products and other environmental information including climate, land cover and terrain morphology. The outputs of SoilGrids are global soil property maps at six standard depth intervals (according to the GlobalSoilMap IUSS working group and its specifications) at a spatial resolution of 250 meters. Prediction uncertainty is quantified by the lower and upper limits of a 90% prediction interval. The SoilGrids maps are publicly available under the [CC-BY 4.0 License](https://creativecommons.org/licenses/by/4.0/).\",\n",
        "    \"history\": \"\",\n",
        "    \"references\" : \"[Article describing soil profiles](https://doi.org/10.5194/essd-12-299-2020) , [Soil grids platform](https://soilgrids.org/)\"\n",
        "\n",
        "}\n",
        "\n",
        "def assign_name(da, name):\n",
        "  da.name = name\n",
        "  return da \n",
        "\n",
        "# create dataset\n",
        "ds = xr.merge(\\\n",
        "[\\\n",
        " rxr.open_rasterio(os.path.join(uri_prefix, fp)).rename(os.path.splitext(fp)[0]).assign_attrs(at)\\\n",
        " for fp, at in zip(file_paths, attrs_list)\\\n",
        " ])\\\n",
        ".assign_coords({'time':time})\\\n",
        ".assign_attrs(attrs)\\\n",
        ".chunk({'x':512, 'y':512, 'band':-1})\n",
        "\n",
        "print_ds_size(ds)\n",
        "with xr.set_options(display_width=120):\n",
        "  print(ds)"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "ds size in GB 92.42\n",
            "\n",
            "<xarray.Dataset>\n",
            "Dimensions:                 (band: 1, time: 1, x: 159246, y: 58034)\n",
            "Coordinates:\n",
            "  * band                    (band) int64 1\n",
            "  * y                       (y) float64 8.361e+06 8.361e+06 8.36e+06 8.36e+06 ... -6.147e+06 -6.147e+06 -6.147e+06\n",
            "  * x                       (x) float64 -1.995e+07 -1.995e+07 -1.995e+07 -1.995e+07 ... 1.986e+07 1.986e+07 1.986e+07\n",
            "    spatial_ref             int64 0\n",
            "  * time                    (time) datetime64[ns] 2000-01-01\n",
            "Data variables:\n",
            "    ocs_0-30cm_mean         (band, y, x) int16 dask.array<chunksize=(1, 512, 512), meta=np.ndarray>\n",
            "    ocs_0-30cm_Q0.5         (band, y, x) int16 dask.array<chunksize=(1, 512, 512), meta=np.ndarray>\n",
            "    ocs_0-30cm_Q0.05        (band, y, x) int16 dask.array<chunksize=(1, 512, 512), meta=np.ndarray>\n",
            "    ocs_0-30cm_Q0.95        (band, y, x) int16 dask.array<chunksize=(1, 512, 512), meta=np.ndarray>\n",
            "    ocs_0-30cm_uncertainty  (band, y, x) uint16 dask.array<chunksize=(1, 512, 512), meta=np.ndarray>\n",
            "Attributes:\n",
            "    Conventions:  CF-1.6\n",
            "    title:        Global gridded soil properties\n",
            "    description:  Global spatial distribution of soil properties mapped using state-of-the-art machine learning methods.\n",
            "    institution:  ISRIC\n",
            "    license:      [CC-BY 4.0 License](https://creativecommons.org/licenses/by/4.0/)\n",
            "    comment:      SoilGrids is a system for global digital soil mapping that uses state-of-the-art machine learning m...\n",
            "    history:      \n",
            "    references:   [Article describing soil profiles](https://doi.org/10.5194/essd-12-299-2020) , [Soil grids platform...\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wb7nawxbwdOJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}