vj_Into_to_pipelines_kfp.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/hpylieva/vertex-ai-pipelines-workshop/blob/main/pipelines_intro_kfp.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "title:generic"
      },
      "source": [
        "# Vertex AI Pipelines: Pipelines introduction for KFP"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "overview:pipelines,intro"
      },
      "source": [
        "## Overview\n",
        "\n",
        "This notebook provides an introduction to using [Vertex AI Pipelines](https://cloud.google.com/vertex-ai/docs/pipelines) with [the Kubeflow Pipelines (KFP) SDK](https://www.kubeflow.org/docs/components/pipelines/).\n",
        "\n",
        "Learn more about [Vertex AI Pipelines](https://cloud.google.com/vertex-ai/docs/pipelines/introduction)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "objective:pipelines,intro"
      },
      "source": [
        "### Objective\n",
        "\n",
        "In this tutorial, you learn how to use the KFP SDK to build pipelines that generate evaluation metrics.\n",
        "\n",
        "This tutorial uses the following Google Cloud ML services:\n",
        "\n",
        "- `Vertex AI Pipelines`\n",
        "\n",
        "The steps performed include:\n",
        "\n",
        "- Define and compile a `Vertex AI` pipeline.\n",
        "- Specify which service account to use for a pipeline run."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "costs:functions,scheduler"
      },
      "source": [
        "### Costs\n",
        "\n",
        "This tutorial uses billable components of Google Cloud:\n",
        "\n",
        "* Vertex AI\n",
        "* Cloud Storage\n",
        "\n",
        "Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing),\n",
        "[Cloud Storage pricing](https://cloud.google.com/storage/pricing),\n",
        "and use the [Pricing\n",
        "Calculator](https://cloud.google.com/products/calculator/)\n",
        "to generate a cost estimate based on your projected usage."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "install_aip:mbsdk"
      },
      "source": [
        "## Installation\n",
        "\n",
        "Install the packages required for executing this notebook."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1fd00fa70a2a",
        "outputId": "fcf5b6ae-faef-415c-cb7e-e0ce821d9074",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting google-cloud-aiplatform\n",
            "  Downloading google_cloud_aiplatform-1.22.0-py2.py3-none-any.whl (2.4 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.4/2.4 MB\u001b[0m \u001b[31m16.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: google-cloud-storage in /usr/local/lib/python3.8/dist-packages (2.7.0)\n",
            "Collecting kfp\n",
            "  Downloading kfp-1.8.19.tar.gz (304 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m304.8/304.8 KB\u001b[0m \u001b[31m8.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting google-cloud-pipeline-components\n",
            "  Downloading google_cloud_pipeline_components-1.0.39-py3-none-any.whl (870 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m870.6/870.6 KB\u001b[0m \u001b[31m33.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: proto-plus<2.0.0dev,>=1.22.0 in /usr/local/lib/python3.8/dist-packages (from google-cloud-aiplatform) (1.22.2)\n",
            "Collecting shapely<2.0.0\n",
            "  Downloading Shapely-1.8.5.post1-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (2.1 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.1/2.1 MB\u001b[0m \u001b[31m46.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5 in /usr/local/lib/python3.8/dist-packages (from google-cloud-aiplatform) (3.19.6)\n",
            "Requirement already satisfied: google-cloud-bigquery<4.0.0dev,>=1.15.0 in /usr/local/lib/python3.8/dist-packages (from google-cloud-aiplatform) (3.4.2)\n",
            "Collecting google-cloud-resource-manager<3.0.0dev,>=1.3.3\n",
            "  Downloading google_cloud_resource_manager-1.8.1-py2.py3-none-any.whl (235 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m235.7/235.7 KB\u001b[0m \u001b[31m13.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0 in /usr/local/lib/python3.8/dist-packages (from google-cloud-aiplatform) (2.11.0)\n",
            "Collecting packaging<22.0.0dev,>=14.3\n",
            "  Downloading packaging-21.3-py3-none-any.whl (40 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m40.8/40.8 KB\u001b[0m \u001b[31m1.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: google-resumable-media>=2.3.2 in /usr/local/lib/python3.8/dist-packages (from google-cloud-storage) (2.4.1)\n",
            "Requirement already satisfied: google-auth<3.0dev,>=1.25.0 in /usr/local/lib/python3.8/dist-packages (from google-cloud-storage) (2.16.0)\n",
            "Requirement already satisfied: requests<3.0.0dev,>=2.18.0 in /usr/local/lib/python3.8/dist-packages (from google-cloud-storage) (2.25.1)\n",
            "Requirement already satisfied: google-cloud-core<3.0dev,>=2.3.0 in /usr/local/lib/python3.8/dist-packages (from google-cloud-storage) (2.3.2)\n",
            "Requirement already satisfied: absl-py<2,>=0.9 in /usr/local/lib/python3.8/dist-packages (from kfp) (1.4.0)\n",
            "Collecting PyYAML<6,>=5.3\n",
            "  Downloading PyYAML-5.4.1-cp38-cp38-manylinux1_x86_64.whl (662 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m662.4/662.4 KB\u001b[0m \u001b[31m22.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting kubernetes<20,>=8.0.0\n",
            "  Downloading kubernetes-19.15.0-py2.py3-none-any.whl (1.7 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.7/1.7 MB\u001b[0m \u001b[31m19.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting google-api-python-client<2,>=1.7.8\n",
            "  Downloading google_api_python_client-1.12.11-py2.py3-none-any.whl (62 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m62.1/62.1 KB\u001b[0m \u001b[31m2.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting requests-toolbelt<1,>=0.8.0\n",
            "  Downloading requests_toolbelt-0.10.1-py2.py3-none-any.whl (54 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m54.5/54.5 KB\u001b[0m \u001b[31m5.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: cloudpickle<3,>=2.0.0 in /usr/local/lib/python3.8/dist-packages (from kfp) (2.2.1)\n",
            "Collecting kfp-server-api<2.0.0,>=1.1.2\n",
            "  Downloading kfp-server-api-1.8.5.tar.gz (58 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m58.1/58.1 KB\u001b[0m \u001b[31m1.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: jsonschema<5,>=3.0.1 in /usr/local/lib/python3.8/dist-packages (from kfp) (4.3.3)\n",
            "Requirement already satisfied: tabulate<1,>=0.8.6 in /usr/local/lib/python3.8/dist-packages (from kfp) (0.8.10)\n",
            "Requirement already satisfied: click<9,>=7.1.2 in /usr/local/lib/python3.8/dist-packages (from kfp) (7.1.2)\n",
            "Collecting Deprecated<2,>=1.2.7\n",
            "  Downloading Deprecated-1.2.13-py2.py3-none-any.whl (9.6 kB)\n",
            "Collecting strip-hints<1,>=0.1.8\n",
            "  Downloading strip-hints-0.1.10.tar.gz (29 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting docstring-parser<1,>=0.7.3\n",
            "  Downloading docstring_parser-0.15-py3-none-any.whl (36 kB)\n",
            "Collecting kfp-pipeline-spec<0.2.0,>=0.1.16\n",
            "  Downloading kfp_pipeline_spec-0.1.16-py3-none-any.whl (19 kB)\n",
            "Collecting fire<1,>=0.3.1\n",
            "  Downloading fire-0.5.0.tar.gz (88 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m88.3/88.3 KB\u001b[0m \u001b[31m3.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting uritemplate<4,>=3.0.1\n",
            "  Downloading uritemplate-3.0.1-py2.py3-none-any.whl (15 kB)\n",
            "Requirement already satisfied: pydantic<2,>=1.8.2 in /usr/local/lib/python3.8/dist-packages (from kfp) (1.10.4)\n",
            "Requirement already satisfied: typer<1.0,>=0.3.2 in /usr/local/lib/python3.8/dist-packages (from kfp) (0.7.0)\n",
            "Requirement already satisfied: typing-extensions<5,>=3.7.4 in /usr/local/lib/python3.8/dist-packages (from kfp) (4.5.0)\n",
            "Collecting grpcio-status<=1.47.0\n",
            "  Downloading grpcio_status-1.47.0-py3-none-any.whl (10.0 kB)\n",
            "Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.56.2 in /usr/local/lib/python3.8/dist-packages (from google-cloud-pipeline-components) (1.58.0)\n",
            "Collecting google-cloud-notebooks>=0.4.0\n",
            "  Downloading google_cloud_notebooks-1.6.1-py2.py3-none-any.whl (190 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m190.1/190.1 KB\u001b[0m \u001b[31m9.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: wrapt<2,>=1.10 in /usr/local/lib/python3.8/dist-packages (from Deprecated<2,>=1.2.7->kfp) (1.14.1)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.8/dist-packages (from fire<1,>=0.3.1->kfp) (1.15.0)\n",
            "Requirement already satisfied: termcolor in /usr/local/lib/python3.8/dist-packages (from fire<1,>=0.3.1->kfp) (2.2.0)\n",
            "Requirement already satisfied: grpcio<2.0dev,>=1.33.2 in /usr/local/lib/python3.8/dist-packages (from google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-cloud-aiplatform) (1.51.1)\n",
            "Requirement already satisfied: google-auth-httplib2>=0.0.3 in /usr/local/lib/python3.8/dist-packages (from google-api-python-client<2,>=1.7.8->kfp) (0.1.0)\n",
            "Requirement already satisfied: httplib2<1dev,>=0.15.0 in /usr/local/lib/python3.8/dist-packages (from google-api-python-client<2,>=1.7.8->kfp) (0.17.4)\n",
            "Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.8/dist-packages (from google-auth<3.0dev,>=1.25.0->google-cloud-storage) (4.9)\n",
            "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.8/dist-packages (from google-auth<3.0dev,>=1.25.0->google-cloud-storage) (0.2.8)\n",
            "Requirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.8/dist-packages (from google-auth<3.0dev,>=1.25.0->google-cloud-storage) (5.3.0)\n",
            "Requirement already satisfied: python-dateutil<3.0dev,>=2.7.2 in /usr/local/lib/python3.8/dist-packages (from google-cloud-bigquery<4.0.0dev,>=1.15.0->google-cloud-aiplatform) (2.8.2)\n",
            "Collecting grpc-google-iam-v1<1.0.0dev,>=0.12.4\n",
            "  Downloading grpc_google_iam_v1-0.12.6-py2.py3-none-any.whl (26 kB)\n",
            "Requirement already satisfied: google-crc32c<2.0dev,>=1.0 in /usr/local/lib/python3.8/dist-packages (from google-resumable-media>=2.3.2->google-cloud-storage) (1.5.0)\n",
            "Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema<5,>=3.0.1->kfp) (0.19.3)\n",
            "Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema<5,>=3.0.1->kfp) (22.2.0)\n",
            "Requirement already satisfied: importlib-resources>=1.4.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema<5,>=3.0.1->kfp) (5.10.2)\n",
            "Requirement already satisfied: urllib3>=1.15 in /usr/local/lib/python3.8/dist-packages (from kfp-server-api<2.0.0,>=1.1.2->kfp) (1.24.3)\n",
            "Requirement already satisfied: certifi in /usr/local/lib/python3.8/dist-packages (from kfp-server-api<2.0.0,>=1.1.2->kfp) (2022.12.7)\n",
            "Requirement already satisfied: setuptools>=21.0.0 in /usr/local/lib/python3.8/dist-packages (from kubernetes<20,>=8.0.0->kfp) (57.4.0)\n",
            "Requirement already satisfied: requests-oauthlib in /usr/local/lib/python3.8/dist-packages (from kubernetes<20,>=8.0.0->kfp) (1.3.1)\n",
            "Collecting websocket-client!=0.40.0,!=0.41.*,!=0.42.*,>=0.32.0\n",
            "  Downloading websocket_client-1.5.1-py3-none-any.whl (55 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m55.9/55.9 KB\u001b[0m \u001b[31m6.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging<22.0.0dev,>=14.3->google-cloud-aiplatform) (3.0.9)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests<3.0.0dev,>=2.18.0->google-cloud-storage) (2.10)\n",
            "Requirement already satisfied: chardet<5,>=3.0.2 in /usr/local/lib/python3.8/dist-packages (from requests<3.0.0dev,>=2.18.0->google-cloud-storage) (4.0.0)\n",
            "Requirement already satisfied: wheel in /usr/local/lib/python3.8/dist-packages (from strip-hints<1,>=0.1.8->kfp) (0.38.4)\n",
            "Requirement already satisfied: zipp>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from importlib-resources>=1.4.0->jsonschema<5,>=3.0.1->kfp) (3.13.0)\n",
            "Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.8/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3.0dev,>=1.25.0->google-cloud-storage) (0.4.8)\n",
            "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from requests-oauthlib->kubernetes<20,>=8.0.0->kfp) (3.2.2)\n",
            "Building wheels for collected packages: kfp, fire, kfp-server-api, strip-hints\n",
            "  Building wheel for kfp (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for kfp: filename=kfp-1.8.19-py3-none-any.whl size=426983 sha256=3764d0c23498a741e5d214710c7f8ab6535bf7ad60f8c3d07372e3540bd03c38\n",
            "  Stored in directory: /root/.cache/pip/wheels/9f/48/49/09cd2b5fbd81315c167c7d58877eaaf825e8f9ff1d733ca618\n",
            "  Building wheel for fire (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for fire: filename=fire-0.5.0-py2.py3-none-any.whl size=116949 sha256=88db6de363af5408092f977e397ac3f71f3a72a097352aaaad8f6f2c179f363b\n",
            "  Stored in directory: /root/.cache/pip/wheels/5b/eb/43/7295e71293b218ddfd627f935229bf54af9018add7fbb5aac6\n",
            "  Building wheel for kfp-server-api (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for kfp-server-api: filename=kfp_server_api-1.8.5-py3-none-any.whl size=99714 sha256=4d6ab46dcb6e9b2717ca9ec63765d9c792f6d553b8ae9b0cbffc006a2c47d66a\n",
            "  Stored in directory: /root/.cache/pip/wheels/93/b7/87/8884b574029455610a5b99752115d2ac857f8cfe8b846a1225\n",
            "  Building wheel for strip-hints (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for strip-hints: filename=strip_hints-0.1.10-py2.py3-none-any.whl size=22301 sha256=7f62328e02364f5394ddf47d80603911e554067696160e9b2840fbe87c94194a\n",
            "  Stored in directory: /root/.cache/pip/wheels/28/16/9b/7c21f4d08b98d02819658600b738d3453b2ffee3c9b757629e\n",
            "Successfully built kfp fire kfp-server-api strip-hints\n",
            "Installing collected packages: websocket-client, uritemplate, strip-hints, shapely, PyYAML, packaging, kfp-pipeline-spec, fire, docstring-parser, Deprecated, requests-toolbelt, kfp-server-api, grpcio-status, kubernetes, grpc-google-iam-v1, google-api-python-client, google-cloud-resource-manager, google-cloud-notebooks, kfp, google-cloud-aiplatform, google-cloud-pipeline-components\n",
            "  Attempting uninstall: uritemplate\n",
            "    Found existing installation: uritemplate 4.1.1\n",
            "    Uninstalling uritemplate-4.1.1:\n",
            "      Successfully uninstalled uritemplate-4.1.1\n",
            "  Attempting uninstall: shapely\n",
            "    Found existing installation: shapely 2.0.1\n",
            "    Uninstalling shapely-2.0.1:\n",
            "      Successfully uninstalled shapely-2.0.1\n",
            "  Attempting uninstall: PyYAML\n",
            "    Found existing installation: PyYAML 6.0\n",
            "    Uninstalling PyYAML-6.0:\n",
            "      Successfully uninstalled PyYAML-6.0\n",
            "  Attempting uninstall: packaging\n",
            "    Found existing installation: packaging 23.0\n",
            "    Uninstalling packaging-23.0:\n",
            "      Successfully uninstalled packaging-23.0\n",
            "  Attempting uninstall: grpcio-status\n",
            "    Found existing installation: grpcio-status 1.48.2\n",
            "    Uninstalling grpcio-status-1.48.2:\n",
            "      Successfully uninstalled grpcio-status-1.48.2\n",
            "  Attempting uninstall: google-api-python-client\n",
            "    Found existing installation: google-api-python-client 2.70.0\n",
            "    Uninstalling google-api-python-client-2.70.0:\n",
            "      Successfully uninstalled google-api-python-client-2.70.0\n",
            "Successfully installed Deprecated-1.2.13 PyYAML-5.4.1 docstring-parser-0.15 fire-0.5.0 google-api-python-client-1.12.11 google-cloud-aiplatform-1.22.0 google-cloud-notebooks-1.6.1 google-cloud-pipeline-components-1.0.39 google-cloud-resource-manager-1.8.1 grpc-google-iam-v1-0.12.6 grpcio-status-1.47.0 kfp-1.8.19 kfp-pipeline-spec-0.1.16 kfp-server-api-1.8.5 kubernetes-19.15.0 packaging-21.3 requests-toolbelt-0.10.1 shapely-1.8.5.post1 strip-hints-0.1.10 uritemplate-3.0.1 websocket-client-1.5.1\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.colab-display-data+json": {
              "pip_warning": {
                "packages": [
                  "google"
                ]
              }
            }
          },
          "metadata": {}
        }
      ],
      "source": [
        "# Install the packages\n",
        "! pip3 install --upgrade google-cloud-aiplatform \\\n",
        "                        google-cloud-storage \\\n",
        "                        kfp \\\n",
        "                        google-cloud-pipeline-components"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "blGlVGFYW9Pt"
      },
      "source": [
        "### Colab only: Uncomment the following cell to restart the kernel."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0JrvuK6LUYnQ"
      },
      "outputs": [],
      "source": [
        "# Automatically restart kernel after installs so that your environment can access the new packages\n",
        "# import IPython\n",
        "\n",
        "# app = IPython.Application.instance()\n",
        "# app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "check_versions"
      },
      "source": [
        "Check the versions of the packages you installed.  The KFP SDK version should be >=1.6."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "check_versions:kfp",
        "outputId": "b9d26c92-63e3-448b-9562-38c5e41a4b48",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "KFP SDK version: 1.8.19\n"
          ]
        }
      ],
      "source": [
        "! python3 -c \"import kfp; print('KFP SDK version: {}'.format(kfp.__version__))\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a47846030fef"
      },
      "source": [
        "## Before you begin"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "project_id"
      },
      "source": [
        "#### Set your project ID\n",
        "\n",
        "**If you don't know your project ID**, try the following:\n",
        "* Run `gcloud config list`.\n",
        "* Run `gcloud projects list`.\n",
        "* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3c8049930470",
        "outputId": "ca3de2c6-5ab4-468e-f8da-f3436e4c84e8",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Updated property [core/project].\n"
          ]
        }
      ],
      "source": [
        "PROJECT_ID = \"tokyo-charge-378510\"  # @param {type:\"string\"}\n",
        "\n",
        "# Set the project id\n",
        "! gcloud config set project {PROJECT_ID}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a54f9d7c1876"
      },
      "source": [
        "#### Region\n",
        "\n",
        "You can also change the `REGION` variable used by Vertex AI. Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3aaadaaf9b30"
      },
      "outputs": [],
      "source": [
        "REGION = \"us-central1\"  # @param {type: \"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5c0404984792"
      },
      "source": [
        "### Authenticate your Google Cloud account\n",
        "\n",
        "Depending on your Jupyter environment, you may have to manually authenticate. Follow the relevant instructions below."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BaFKzJ_xXpvm"
      },
      "source": [
        "**1. Vertex AI Workbench**\n",
        "* Do nothing as you are already authenticated."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_fV-KyGAX4Xl"
      },
      "source": [
        "**2. Local JupyterLab instance, uncomment and run:**\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7uXB1HAPX6L_"
      },
      "outputs": [],
      "source": [
        "# ! gcloud auth login"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ab_TRMQIYCCX"
      },
      "source": [
        "**3. Colab, uncomment and run:**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vx25htmYYExI"
      },
      "outputs": [],
      "source": [
        "from google.colab import auth\n",
        "auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uZdA0-jBYGqt"
      },
      "source": [
        "**4. Service account or other**\n",
        "* See how to grant Cloud Storage permissions to your service account at https://cloud.google.com/storage/docs/gsutil/commands/iam#ch-examples."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bucket:custom"
      },
      "source": [
        "### Create a Cloud Storage bucket\n",
        "\n",
        "Create a storage bucket to store intermediate artifacts such as datasets."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bucket"
      },
      "outputs": [],
      "source": [
        "BUCKET_URI = \"gs://pipeline_root_wine/\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "create_bucket"
      },
      "source": [
        "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!gcloud auth login"
      ],
      "metadata": {
        "id": "HbZeaeHRlQJj",
        "outputId": "84d79386-b090-4fff-c3ef-dc7b09521842",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "You are running on a Google Compute Engine virtual machine.\n",
            "It is recommended that you use service accounts for authentication.\n",
            "\n",
            "You can run:\n",
            "\n",
            "  $ gcloud config set account `ACCOUNT`\n",
            "\n",
            "to switch accounts if necessary.\n",
            "\n",
            "Your credentials may be visible to others with access to this\n",
            "virtual machine. Are you sure you want to authenticate with\n",
            "your personal account?\n",
            "\n",
            "Do you want to continue (Y/n)?  Y\n",
            "\n",
            "Go to the following link in your browser:\n",
            "\n",
            "    https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=32555940559.apps.googleusercontent.com&redirect_uri=https%3A%2F%2Fsdk.cloud.google.com%2Fauthcode.html&scope=openid+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fappengine.admin+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fsqlservice.login+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcompute+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Faccounts.reauth&state=18ct5XjfLjx8c4zx5GVMMWOQWKQ1tG&prompt=consent&access_type=offline&code_challenge=KJGlDrhlI__mL6lRRWO6ML-UxX1Za-DaAcyDG6gKpAo&code_challenge_method=S256\n",
            "\n",
            "Enter authorization code: 4/0AWtgzh6zoWtMFumKRSPaQkj9_WBx8M3ls-GeDS2LtaecqSSyrAvf5mpqmVclsW53Tb8AIA\n",
            "\n",
            "You are now logged in as [hannap@2kgroup.com].\n",
            "Your current project is [andy-1234-221921].  You can change this setting by running:\n",
            "  $ gcloud config set project PROJECT_ID\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Oz8J0vmSlugt",
        "outputId": "34b27ffc-a03d-403a-ca60-1f90e4ee9616",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Creating gs://pipeline_root_wine/...\n"
          ]
        }
      ],
      "source": [
        "! gsutil mb -l $REGION -p $PROJECT_ID $BUCKET_URI"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "set_service_account:pipelines"
      },
      "source": [
        "#### Set service account access for Vertex AI Pipelines\n",
        "\n",
        "Run the following commands to grant your service account access to read and write pipeline artifacts in the bucket that you created in the previous step -- you only need to run these once per service account."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "shell_output = !gcloud auth list 2>/dev/null\n",
        "SERVICE_ACCOUNT = shell_output[2].strip()[8:]\n",
        "print(\"Service Account:\", SERVICE_ACCOUNT)"
      ],
      "metadata": {
        "id": "0QzqpIffk6TP",
        "outputId": "d3c0adad-4415-4d85-d960-05e2355add5f",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Service Account: hannap@2kgroup.com\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "shell_output"
      ],
      "metadata": {
        "id": "4k87FMJApCUJ",
        "outputId": "ed771714-5dc3-45ef-fa46-d7e82cbfa15c",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['  Credentialed Accounts', 'ACTIVE  ACCOUNT', '*       hannap@2kgroup.com']"
            ]
          },
          "metadata": {},
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "b556df542518"
      },
      "outputs": [],
      "source": [
        "SERVICE_ACCOUNT = \"24871937313-compute@developer.gserviceaccount.com\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0QNsyzEF2Ou4"
      },
      "outputs": [],
      "source": [
        "! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectCreator $BUCKET_URI\n",
        "\n",
        "! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectViewer $BUCKET_URI"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "setup_vars"
      },
      "source": [
        "### Set up variables\n",
        "\n",
        "Next, set up some variables used throughout the tutorial.\n",
        "### Import libraries and define constants"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "import_aip:mbsdk"
      },
      "outputs": [],
      "source": [
        "from typing import NamedTuple\n",
        "\n",
        "import google.cloud.aiplatform as aip\n",
        "from kfp import dsl\n",
        "from kfp.v2 import compiler\n",
        "from kfp.v2.dsl import component"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aip_constants:endpoint"
      },
      "source": [
        "#### Vertex AI constants\n",
        "\n",
        "Setup up the following constants for Vertex AI:\n",
        "\n",
        "- `API_ENDPOINT`: The Vertex AI API service endpoint for `Dataset`, `Model`, `Job`, `Pipeline` and `Endpoint` services."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TnO8gVBb2Ou4"
      },
      "outputs": [],
      "source": [
        "# API service endpoint\n",
        "API_ENDPOINT = \"{}-aiplatform.googleapis.com\".format(REGION)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pipeline_constants"
      },
      "source": [
        "#### Vertex AI Pipelines constants\n",
        "\n",
        "Setup up the following constants for Vertex AI Pipelines:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ONHtQDz32Ou5"
      },
      "outputs": [],
      "source": [
        "PIPELINE_ROOT = \"{}/pipeline_root/intro\".format(BUCKET_URI)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "init_aip:mbsdk"
      },
      "source": [
        "## Initialize Vertex AI SDK for Python\n",
        "\n",
        "Initialize the Vertex AI SDK for Python for your project and corresponding bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mbpw7oyM2Ou5"
      },
      "outputs": [],
      "source": [
        "aip.init(project=PROJECT_ID, staging_bucket=BUCKET_URI)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "define_component:hello_world"
      },
      "source": [
        "### Define Python function-based pipeline components\n",
        "\n",
        "In this tutorial, you define a simple pipeline that has three steps, where each step is defined as a component.\n",
        "\n",
        "#### Define hello_world component\n",
        "\n",
        "First, define a component based on a very simple Python function. It takes a string input parameter and returns that value as output.\n",
        "\n",
        "Note the use of the `@component` decorator, which compiles the function to a KFP component when evaluated.  For example purposes, this example specifies a base image to use for the component (`python:3.9`), and a component YAML file, `hw.yaml`. The compiled component specification is written to this file.  (The default base image is `python:3.7`, which would of course work just fine too)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GjJhJUID2Ou6"
      },
      "outputs": [],
      "source": [
        "@component(output_component_file=\"hw.yaml\", base_image=\"python:3.9\")\n",
        "def hello_world(text: str) -> str:\n",
        "    print(text)\n",
        "    return text"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SWcIXuxR2Ou6"
      },
      "source": [
        "As you'll see below, compilation of this component creates a [task factory function](https://www.kubeflow.org/docs/components/pipelines/sdk/python-function-components/)—called `hello_world`— that you can use in defining a pipeline step.\n",
        "\n",
        "While not shown here, if you want to share this component definition, or use it in another context, you could also load it from its yaml file like this:\n",
        "`hello_world_op = components.load_component_from_file('./hw.yaml')`.\n",
        "You can also use the `load_component_from_url` method, if your component yaml file is stored online. (For GitHub URLs, load the 'raw' file.)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "define_component:two_outputs"
      },
      "source": [
        "#### Define two_outputs component\n",
        "\n",
        "The first component below, `two_outputs`, demonstrates installing a package -- in this case the `google-cloud-storage` package. Alternatively, you can specify a base image that includes the necessary installations.\n",
        "\n",
        "*Note:* The component function won't actually use the package.\n",
        "\n",
        "Alternatively, you can specify a base image that includes the necessary installations.\n",
        "\n",
        "The `two_outputs` component returns two named outputs."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "U4Yv33su2Ou6"
      },
      "outputs": [],
      "source": [
        "@component(packages_to_install=[\"google-cloud-storage\"])\n",
        "def two_outputs(\n",
        "    text: str,\n",
        ") -> NamedTuple(\n",
        "    \"Outputs\",\n",
        "    [\n",
        "        (\"output_one\", str),  # Return parameters\n",
        "        (\"output_two\", str),\n",
        "    ],\n",
        "):\n",
        "    # the import is not actually used for this simple example, but the import\n",
        "    # is successful, as it was included in the `packages_to_install` list.\n",
        "    from google.cloud import storage  # noqa: F401\n",
        "\n",
        "    o1 = f\"output one from text: {text}\"\n",
        "    o2 = f\"output two from text: {text}\"\n",
        "    print(\"output one: {}; output_two: {}\".format(o1, o2))\n",
        "    return (o1, o2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "define_component:consumer"
      },
      "source": [
        "#### Define the consumer component\n",
        "\n",
        "The third component, `consumer`, takes three string inputs and prints them out."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bu8XvOj82Ou6"
      },
      "outputs": [],
      "source": [
        "@component\n",
        "def consumer(text1: str, text2: str, text3: str) -> str:\n",
        "    print(f\"text1: {text1}; text2: {text2}; text3: {text3}\")\n",
        "    return f\"text1: {text1}; text2: {text2}; text3: {text3}\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "define_pipeline:intro"
      },
      "source": [
        "### Define a pipeline that uses the components\n",
        "\n",
        "Next, define a pipeline that uses these three components.\n",
        "\n",
        "By evaluating the component definitions above, you've created task factory functions that are used in the pipeline definition to create the pipeline steps.\n",
        "\n",
        "The pipeline takes an input parameter, and passes that parameter as an argument to the first two pipeline steps (`hw_task` and `two_outputs_task`).\n",
        "\n",
        "Then, the third pipeline step (`consumer_task`) consumes the outputs of the first and second steps.  Because the `hello_world` component definition just returns one unnamed output, you refer to it as `hw_task.output`.  The `two_outputs` task returns two named outputs, which you access as `two_outputs_task.outputs[\"<output_name>\"]`.\n",
        "\n",
        "*Note:* In the `@dsl.pipeline` decorator, you're defining the `PIPELINE_ROOT` Cloud Storage path to use.  If you had not included that info here, it would be required to specify it when creating the pipeline run, as you'll see below."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CV5dRAeJ2Ou7"
      },
      "outputs": [],
      "source": [
        "@dsl.pipeline(\n",
        "    name=\"intro-pipeline-unique\",\n",
        "    description=\"A simple intro pipeline\",\n",
        "    pipeline_root=PIPELINE_ROOT,\n",
        ")\n",
        "def pipeline(text: str = \"hi there\"):\n",
        "    hw_task = hello_world(text)\n",
        "    two_outputs_task = two_outputs(text)\n",
        "    consumer_task = consumer(  # noqa: F841\n",
        "        hw_task.output,\n",
        "        two_outputs_task.outputs[\"output_one\"],\n",
        "        two_outputs_task.outputs[\"output_two\"],\n",
        "    )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "compile_pipeline"
      },
      "source": [
        "## Compile the pipeline\n",
        "\n",
        "Next, compile the pipeline."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VP_JJ9Oe2Ou7",
        "outputId": "fb15b7cb-5502-4a06-b4f3-c6487c8fe026",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.8/dist-packages/kfp/v2/compiler/compiler.py:1290: FutureWarning: APIs imported from the v1 namespace (e.g. kfp.dsl, kfp.components, etc) will not be supported by the v2 compiler since v2.0.0\n",
            "  warnings.warn(\n"
          ]
        }
      ],
      "source": [
        "from kfp.v2 import compiler  # noqa: F811\n",
        "\n",
        "compiler.Compiler().compile(pipeline_func=pipeline, package_path=\"intro_pipeline.json\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "run_pipeline:intro"
      },
      "source": [
        "## Run the pipeline\n",
        "\n",
        "Next, run the pipeline."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sjxaBix_2Ou7",
        "outputId": "0dc2f9e7-2e42-4aba-be96-c0894c8cb6c0",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Creating PipelineJob\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "INFO:google.cloud.aiplatform.pipeline_jobs:Creating PipelineJob\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "PipelineJob created. Resource name: projects/24871937313/locations/us-central1/pipelineJobs/intro-pipeline-unique-20230221110853\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "INFO:google.cloud.aiplatform.pipeline_jobs:PipelineJob created. Resource name: projects/24871937313/locations/us-central1/pipelineJobs/intro-pipeline-unique-20230221110853\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "To use this PipelineJob in another session:\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "INFO:google.cloud.aiplatform.pipeline_jobs:To use this PipelineJob in another session:\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "pipeline_job = aiplatform.PipelineJob.get('projects/24871937313/locations/us-central1/pipelineJobs/intro-pipeline-unique-20230221110853')\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "INFO:google.cloud.aiplatform.pipeline_jobs:pipeline_job = aiplatform.PipelineJob.get('projects/24871937313/locations/us-central1/pipelineJobs/intro-pipeline-unique-20230221110853')\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "View Pipeline Job:\n",
            "https://console.cloud.google.com/vertex-ai/locations/us-central1/pipelines/runs/intro-pipeline-unique-20230221110853?project=24871937313\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "INFO:google.cloud.aiplatform.pipeline_jobs:View Pipeline Job:\n",
            "https://console.cloud.google.com/vertex-ai/locations/us-central1/pipelines/runs/intro-pipeline-unique-20230221110853?project=24871937313\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "PipelineJob projects/24871937313/locations/us-central1/pipelineJobs/intro-pipeline-unique-20230221110853 current state:\n",
            "PipelineState.PIPELINE_STATE_RUNNING\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "INFO:google.cloud.aiplatform.pipeline_jobs:PipelineJob projects/24871937313/locations/us-central1/pipelineJobs/intro-pipeline-unique-20230221110853 current state:\n",
            "PipelineState.PIPELINE_STATE_RUNNING\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "PipelineJob projects/24871937313/locations/us-central1/pipelineJobs/intro-pipeline-unique-20230221110853 current state:\n",
            "PipelineState.PIPELINE_STATE_RUNNING\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "INFO:google.cloud.aiplatform.pipeline_jobs:PipelineJob projects/24871937313/locations/us-central1/pipelineJobs/intro-pipeline-unique-20230221110853 current state:\n",
            "PipelineState.PIPELINE_STATE_RUNNING\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "PipelineJob projects/24871937313/locations/us-central1/pipelineJobs/intro-pipeline-unique-20230221110853 current state:\n",
            "PipelineState.PIPELINE_STATE_RUNNING\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "INFO:google.cloud.aiplatform.pipeline_jobs:PipelineJob projects/24871937313/locations/us-central1/pipelineJobs/intro-pipeline-unique-20230221110853 current state:\n",
            "PipelineState.PIPELINE_STATE_RUNNING\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "PipelineJob projects/24871937313/locations/us-central1/pipelineJobs/intro-pipeline-unique-20230221110853 current state:\n",
            "PipelineState.PIPELINE_STATE_RUNNING\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "INFO:google.cloud.aiplatform.pipeline_jobs:PipelineJob projects/24871937313/locations/us-central1/pipelineJobs/intro-pipeline-unique-20230221110853 current state:\n",
            "PipelineState.PIPELINE_STATE_RUNNING\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "PipelineJob projects/24871937313/locations/us-central1/pipelineJobs/intro-pipeline-unique-20230221110853 current state:\n",
            "PipelineState.PIPELINE_STATE_RUNNING\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "INFO:google.cloud.aiplatform.pipeline_jobs:PipelineJob projects/24871937313/locations/us-central1/pipelineJobs/intro-pipeline-unique-20230221110853 current state:\n",
            "PipelineState.PIPELINE_STATE_RUNNING\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "PipelineJob run completed. Resource name: projects/24871937313/locations/us-central1/pipelineJobs/intro-pipeline-unique-20230221110853\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "INFO:google.cloud.aiplatform.pipeline_jobs:PipelineJob run completed. Resource name: projects/24871937313/locations/us-central1/pipelineJobs/intro-pipeline-unique-20230221110853\n"
          ]
        }
      ],
      "source": [
        "DISPLAY_NAME = \"intro_pipeline_job_unique\"\n",
        "\n",
        "job = aip.PipelineJob(\n",
        "    display_name=DISPLAY_NAME,\n",
        "    template_path=\"intro_pipeline.json\",\n",
        "    pipeline_root=PIPELINE_ROOT,\n",
        ")\n",
        "\n",
        "job.run()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view_pipeline_run:intro"
      },
      "source": [
        "Click on the generated link to see your run in the Cloud Console.\n",
        "\n",
        "<!-- It should look something like this as it is running:\n",
        "\n",
        "<a href=\"https://storage.googleapis.com/amy-jo/images/mp/automl_tabular_classif.png\" target=\"_blank\"><img src=\"https://storage.googleapis.com/amy-jo/images/mp/automl_tabular_classif.png\" width=\"40%\"/></a> -->\n",
        "\n",
        "In the UI, many of the pipeline DAG nodes will expand or collapse when you click on them. Here is a partially-expanded view of the DAG (click image to see larger version).\n",
        "\n",
        "<a href=\"https://storage.googleapis.com/amy-jo/images/mp/intro_pipeline.png\" target=\"_blank\"><img src=\"https://storage.googleapis.com/amy-jo/images/mp/intro_pipeline.png\" width=\"60%\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d428cc8803e7"
      },
      "source": [
        "### Delete the pipeline job\n",
        "\n",
        "You can delete the pipeline job with the method `delete()`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "97802d9432e7",
        "outputId": "649185b7-ed99-4e73-84e5-35d0a54139ca",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Deleting PipelineJob : projects/24871937313/locations/us-central1/pipelineJobs/intro-pipeline-unique-20230221110853\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "INFO:google.cloud.aiplatform.base:Deleting PipelineJob : projects/24871937313/locations/us-central1/pipelineJobs/intro-pipeline-unique-20230221110853\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Delete PipelineJob  backing LRO: projects/24871937313/locations/us-central1/operations/2036813198459404288\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "INFO:google.cloud.aiplatform.base:Delete PipelineJob  backing LRO: projects/24871937313/locations/us-central1/operations/2036813198459404288\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "PipelineJob deleted. . Resource name: projects/24871937313/locations/us-central1/pipelineJobs/intro-pipeline-unique-20230221110853\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "INFO:google.cloud.aiplatform.base:PipelineJob deleted. . Resource name: projects/24871937313/locations/us-central1/pipelineJobs/intro-pipeline-unique-20230221110853\n"
          ]
        }
      ],
      "source": [
        "job.delete()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "run_pipeline_service_account"
      },
      "source": [
        "## Specifying a service account to use for a pipeline run\n",
        "\n",
        "By default, the [service account](https://cloud.google.com/iam/docs/service-accounts) used for your pipeline run is your [default compute engine service account](https://cloud.google.com/compute/docs/access/service-accounts#default_service_account).\n",
        "However, you might want to run pipelines with permissions to access different roles than those configured for your default SA (e.g. perhaps using a more restricted set of permissions).\n",
        "\n",
        "If you want to execute your pipeline using a different service account, this is straightforward to do.  You just need to give the new service account the correct permissions.\n",
        "\n",
        "### Create a service account\n",
        "\n",
        "Once your service account is configured, you pass it as an argument to the `create_run_from_job_spec` method. The pipeline job runs with the permissions of the given service account.\n",
        "\n",
        "Learn about [creating and configuring a service account to work with Vertex AI Pipelines](https://cloud.google.com/vertex-ai/docs/pipelines/configure-project#service-account)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CHcFxq742Ou9"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "if not os.getenv(\"IS_TESTING\"):\n",
        "    response = api_client.create_run_from_job_spec(\n",
        "        job_spec_path=\"intro_pipeline.json\",\n",
        "        pipeline_root=PIPELINE_ROOT,\n",
        "        service_account=SERVICE_ACCOUNT,  # <-- CHANGE to use non-default service account\n",
        "    )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "run_pipeline_caching"
      },
      "source": [
        "## Pipeline step caching\n",
        "\n",
        "By default, pipeline step caching is enabled. This means that the results of previous step executions are reused when possible.\n",
        "\n",
        "if you want to disable caching for a pipeline run, you can pass the `enable_caching=False` argument to the `PipelineJob` constructor when you submit the pipeline job, as shown below.  Try submitting the example pipeline job again, first with and then without caching enabled."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VrM5wqjh2Ou9",
        "outputId": "08d96c6d-cfe8-4334-9f54-53e5df3d75a0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 873
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Creating PipelineJob\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "INFO:google.cloud.aiplatform.pipeline_jobs:Creating PipelineJob\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "PipelineJob created. Resource name: projects/24871937313/locations/us-central1/pipelineJobs/intro-pipeline-unique-20230221111402\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "INFO:google.cloud.aiplatform.pipeline_jobs:PipelineJob created. Resource name: projects/24871937313/locations/us-central1/pipelineJobs/intro-pipeline-unique-20230221111402\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "To use this PipelineJob in another session:\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "INFO:google.cloud.aiplatform.pipeline_jobs:To use this PipelineJob in another session:\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "pipeline_job = aiplatform.PipelineJob.get('projects/24871937313/locations/us-central1/pipelineJobs/intro-pipeline-unique-20230221111402')\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "INFO:google.cloud.aiplatform.pipeline_jobs:pipeline_job = aiplatform.PipelineJob.get('projects/24871937313/locations/us-central1/pipelineJobs/intro-pipeline-unique-20230221111402')\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "View Pipeline Job:\n",
            "https://console.cloud.google.com/vertex-ai/locations/us-central1/pipelines/runs/intro-pipeline-unique-20230221111402?project=24871937313\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "INFO:google.cloud.aiplatform.pipeline_jobs:View Pipeline Job:\n",
            "https://console.cloud.google.com/vertex-ai/locations/us-central1/pipelines/runs/intro-pipeline-unique-20230221111402?project=24871937313\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "PipelineJob projects/24871937313/locations/us-central1/pipelineJobs/intro-pipeline-unique-20230221111402 current state:\n",
            "PipelineState.PIPELINE_STATE_RUNNING\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "INFO:google.cloud.aiplatform.pipeline_jobs:PipelineJob projects/24871937313/locations/us-central1/pipelineJobs/intro-pipeline-unique-20230221111402 current state:\n",
            "PipelineState.PIPELINE_STATE_RUNNING\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "PipelineJob projects/24871937313/locations/us-central1/pipelineJobs/intro-pipeline-unique-20230221111402 current state:\n",
            "PipelineState.PIPELINE_STATE_RUNNING\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "INFO:google.cloud.aiplatform.pipeline_jobs:PipelineJob projects/24871937313/locations/us-central1/pipelineJobs/intro-pipeline-unique-20230221111402 current state:\n",
            "PipelineState.PIPELINE_STATE_RUNNING\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "PipelineJob projects/24871937313/locations/us-central1/pipelineJobs/intro-pipeline-unique-20230221111402 current state:\n",
            "PipelineState.PIPELINE_STATE_RUNNING\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "INFO:google.cloud.aiplatform.pipeline_jobs:PipelineJob projects/24871937313/locations/us-central1/pipelineJobs/intro-pipeline-unique-20230221111402 current state:\n",
            "PipelineState.PIPELINE_STATE_RUNNING\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "PipelineJob projects/24871937313/locations/us-central1/pipelineJobs/intro-pipeline-unique-20230221111402 current state:\n",
            "PipelineState.PIPELINE_STATE_RUNNING\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "INFO:google.cloud.aiplatform.pipeline_jobs:PipelineJob projects/24871937313/locations/us-central1/pipelineJobs/intro-pipeline-unique-20230221111402 current state:\n",
            "PipelineState.PIPELINE_STATE_RUNNING\n"
          ]
        },
        {
          "output_type": "error",
          "ename": "KeyboardInterrupt",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-17-8e81568f831c>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      5\u001b[0m )\n\u001b[1;32m      6\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 7\u001b[0;31m \u001b[0mjob\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;32m/usr/local/lib/python3.8/dist-packages/google/cloud/aiplatform/pipeline_jobs.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(self, service_account, network, sync, create_request_timeout)\u001b[0m\n\u001b[1;32m    312\u001b[0m         \u001b[0mnetwork\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnetwork\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0minitializer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mglobal_config\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnetwork\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    313\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 314\u001b[0;31m         self._run(\n\u001b[0m\u001b[1;32m    315\u001b[0m             \u001b[0mservice_account\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mservice_account\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    316\u001b[0m             \u001b[0mnetwork\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mnetwork\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.8/dist-packages/google/cloud/aiplatform/base.py\u001b[0m in \u001b[0;36mwrapper\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m    808\u001b[0m                 \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    809\u001b[0m                     \u001b[0mVertexAiResourceNounWithFutureManager\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwait\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 810\u001b[0;31m                 \u001b[0;32mreturn\u001b[0m \u001b[0mmethod\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    811\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    812\u001b[0m             \u001b[0;31m# callbacks to call within the Future (in same Thread)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.8/dist-packages/google/cloud/aiplatform/pipeline_jobs.py\u001b[0m in \u001b[0;36m_run\u001b[0;34m(self, service_account, network, sync, create_request_timeout)\u001b[0m\n\u001b[1;32m    349\u001b[0m         )\n\u001b[1;32m    350\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 351\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_block_until_complete\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    352\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    353\u001b[0m     def submit(\n",
            "\u001b[0;32m/usr/local/lib/python3.8/dist-packages/google/cloud/aiplatform/pipeline_jobs.py\u001b[0m in \u001b[0;36m_block_until_complete\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    492\u001b[0m                 \u001b[0mlog_wait\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlog_wait\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0mmultiplier\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmax_wait\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    493\u001b[0m                 \u001b[0mprevious_time\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcurrent_time\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 494\u001b[0;31m             \u001b[0mtime\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msleep\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mwait\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    495\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    496\u001b[0m         \u001b[0;31m# Error is only populated when the job state is\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
          ]
        }
      ],
      "source": [
        "job = aip.PipelineJob(\n",
        "    display_name=\"intro_pipeline_job_cached_unique\",\n",
        "    template_path=\"intro_pipeline.json\",\n",
        "    enable_caching=False,\n",
        ")\n",
        "\n",
        "job.run()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "37e2b84a0c0d"
      },
      "source": [
        "### Delete the pipeline job\n",
        "\n",
        "You can delete the pipeline job with the method `delete()`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "c5926e7eaad8",
        "outputId": "f9ea3bc3-f728-405d-fff3-eb8e0710e561",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Deleting PipelineJob : projects/24871937313/locations/us-central1/pipelineJobs/intro-pipeline-unique-20230221111402\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "INFO:google.cloud.aiplatform.base:Deleting PipelineJob : projects/24871937313/locations/us-central1/pipelineJobs/intro-pipeline-unique-20230221111402\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Delete PipelineJob  backing LRO: projects/24871937313/locations/us-central1/operations/9014014921163145216\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "INFO:google.cloud.aiplatform.base:Delete PipelineJob  backing LRO: projects/24871937313/locations/us-central1/operations/9014014921163145216\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "PipelineJob deleted. . Resource name: projects/24871937313/locations/us-central1/pipelineJobs/intro-pipeline-unique-20230221111402\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "INFO:google.cloud.aiplatform.base:PipelineJob deleted. . Resource name: projects/24871937313/locations/us-central1/pipelineJobs/intro-pipeline-unique-20230221111402\n"
          ]
        }
      ],
      "source": [
        "job.delete()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cleanup:pipelines"
      },
      "source": [
        "# Cleaning up\n",
        "\n",
        "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
        "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
        "\n",
        "Otherwise, you can delete the individual resources you created in this tutorial."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-eXdbQCo2Ou_",
        "outputId": "9ff69827-71ee-4e5f-ac1a-2a2eeda7ae19",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "list index out of range\n",
            "Removing gs://pipeline_root_wine/pipeline_root/#1676977819716732...\n",
            "Removing gs://pipeline_root_wine/pipeline_root/intro/#1676977819868427...\n",
            "Removing gs://pipeline_root_wine/pipeline_root/intro/24871937313/#1676977820044251...\n",
            "Removing gs://pipeline_root_wine/pipeline_root/intro/24871937313/intro-pipeline-unique-20230221110853/#1676977820218063...\n",
            "/ [4 objects]                                                                   \n",
            "==> NOTE: You are performing a sequence of gsutil operations that may\n",
            "run significantly faster if you instead use gsutil -m rm ... Please\n",
            "see the -m section under \"gsutil help options\" for further information\n",
            "about when gsutil -m can be advantageous.\n",
            "\n",
            "Removing gs://pipeline_root_wine/pipeline_root/intro/24871937313/intro-pipeline-unique-20230221110853/consumer_5702141243803303936/#1676977929973438...\n",
            "Removing gs://pipeline_root_wine/pipeline_root/intro/24871937313/intro-pipeline-unique-20230221110853/consumer_5702141243803303936/executor_output.json#1676977930255239...\n",
            "Removing gs://pipeline_root_wine/pipeline_root/intro/24871937313/intro-pipeline-unique-20230221110853/hello-world_-3521230793051471872/#1676977820415030...\n",
            "Removing gs://pipeline_root_wine/pipeline_root/intro/24871937313/intro-pipeline-unique-20230221110853/hello-world_-3521230793051471872/executor_output.json#1676977820830950...\n",
            "Removing gs://pipeline_root_wine/pipeline_root/intro/24871937313/intro-pipeline-unique-20230221110853/two-outputs_3396298234589609984/#1676977830121238...\n",
            "Removing gs://pipeline_root_wine/pipeline_root/intro/24871937313/intro-pipeline-unique-20230221110853/two-outputs_3396298234589609984/executor_output.json#1676977830425877...\n",
            "Removing gs://pipeline_root_wine/pipeline_root/intro/24871937313/intro-pipeline-unique-20230221111402/#1676978164754563...\n",
            "Removing gs://pipeline_root_wine/pipeline_root/intro/24871937313/intro-pipeline-unique-20230221111402/hello-world_-7250211284514242560/#1676978231209002...\n",
            "Removing gs://pipeline_root_wine/pipeline_root/intro/24871937313/intro-pipeline-unique-20230221111402/hello-world_-7250211284514242560/executor_output.json#1676978231541336...\n",
            "Removing gs://pipeline_root_wine/pipeline_root/intro/24871937313/intro-pipeline-unique-20230221111402/two-outputs_6584846770767921152/#1676978165027145...\n",
            "Removing gs://pipeline_root_wine/pipeline_root/intro/24871937313/intro-pipeline-unique-20230221111402/two-outputs_6584846770767921152/executor_output.json#1676978165332077...\n",
            "/ [15 objects]                                                                  \n",
            "Operation completed over 15 objects.                                             \n",
            "Removing gs://pipeline_root_wine/...\n"
          ]
        }
      ],
      "source": [
        "import os\n",
        "delete_pipeline = True\n",
        "delete_bucket = True\n",
        "\n",
        "try:\n",
        "    if delete_pipeline and \"DISPLAY_NAME\" in globals():\n",
        "        pipelines = aip.PipelineJob.list(\n",
        "            filter=f\"display_name={DISPLAY_NAME}\", order_by=\"create_time\"\n",
        "        )\n",
        "        pipeline = pipelines[0]\n",
        "        aip.PipelineJob.delete(pipeline.resource_name)\n",
        "        print(\"Deleted pipeline:\", pipeline)\n",
        "except Exception as e:\n",
        "    print(e)\n",
        "\n",
        "if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
        "    ! gsutil rm -r $BUCKET_URI"
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "U6HqFDrRssvN"
      },
      "execution_count": null,
      "outputs": []
    }
  ],
  "metadata": {
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}