Vertex_AI_Chat.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jbl5pojWzAD0"
      },
      "outputs": [],
      "source": [
        "# Copyright 2023-2024 Nils Knieling\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aVjTZHl4zKKi"
      },
      "source": [
        "# 💬 Chat with Vertex AI\n",
        "\n",
        "[![Open in Colab](https://img.shields.io/badge/Open%20in%20Colab-%23F9AB00.svg?logo=googlecolab&logoColor=white)](https://colab.research.google.com/github/Cyclenerd/google-cloud-gcp-openai-api/blob/master/Vertex_AI_Chat.ipynb)\n",
        "[![Open in Vertex AI Workbench](https://img.shields.io/badge/Open%20in%20Vertex%20AI%20Workbench-%234285F4.svg?logo=googlecloud&logoColor=white)](https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/Cyclenerd/google-cloud-gcp-openai-api/master/Vertex_AI_Chat.ipynb)\n",
        "[![View on GitHub](https://img.shields.io/badge/View%20on%20GitHub-181717.svg?logo=github&logoColor=white)](https://github.com/Cyclenerd/google-cloud-gcp-openai-api/blob/master/Vertex_AI_Chat.ipynb)\n",
        "\n",
        "This notebook describes how to deploy the [OpenAI API for Google Cloud Vertex AI](https://github.com/Cyclenerd/google-cloud-gcp-openai-api#readme) API backend and [Chatbot UI](https://github.com/mckaywrigley/chatbot-ui) frontend as Google Cloud Run service.\n",
        "\n",
        "> By default, [**Google Cloud does not use Customer Data to train its foundation models**](https://cloud.google.com/vertex-ai/docs/generative-ai/data-governance#foundation_model_development) as part of Google Cloud's AI/ML Privacy Commitment. More details about how Google processes data can also be found in [Google's Customer Data Processing Addendum (CDPA)](https://cloud.google.com/terms/data-processing-addendum)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ofbTwaIazX4Y"
      },
      "source": [
        "## Setup\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a3rLBQnqtAQv"
      },
      "source": [
        "### Configuration\n",
        "\n",
        "Regions: <https://cloud.google.com/vertex-ai/docs/general/locations#vertex-ai-regions>\n",
        "\n",
        "Model: One of the [foundation models](https://cloud.google.com/vertex-ai/docs/generative-ai/learn/models#foundation_models) that are available in Vertex AI.\n",
        "\n",
        "Key `openai_key` is used for authentication the chat frondend against the API backend application. It is not a real OpenAI API key."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "xZK1M-0OK6GD"
      },
      "outputs": [],
      "source": [
        "# @markdown ✏️ Replace the placeholder text below:\n",
        "\n",
        "# Please fill in these values.\n",
        "project_id = \"your-google-cloud-project\"  # @param {type:\"string\"}\n",
        "region = \"europe-west1\"  # @param {type:\"string\"}\n",
        "artifact_repository = \"docker-openai-api\"  # @param {type:\"string\"}\n",
        "model = \"codechat-bison\"  # @param {type:\"string\"}\n",
        "openai_key = \"sk-XYZ-replace-with-good-key\" # @param {type:\"string\"}\n",
        "\n",
        "# Quick input validations.\n",
        "assert project_id, \"⚠️ Please provide a Google Cloud project ID\"\n",
        "assert region, \"⚠️ Please provide a Google Cloud region\"\n",
        "assert artifact_repository, \"⚠️ Please provide a Google Cloud Artifact repository name\"\n",
        "assert model, \"⚠️ Please provide a valid model\"\n",
        "assert openai_key, \"⚠️ Please provide a valid key for authentication\"\n",
        "\n",
        "# Configure gcloud.\n",
        "!gcloud config set project \"{project_id}\"\n",
        "!gcloud config set storage/parallel_composite_upload_enabled \"True\"\n",
        "\n",
        "print(f\"\\n☁️ Google Cloud project ID: {project_id}\")\n",
        "print(\"☑️ Done\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yq1eutyctGO9"
      },
      "source": [
        "### Authenticate"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "rriTKU3R1PdS"
      },
      "outputs": [],
      "source": [
        "#@markdown #### (Colab only!) Authenticate your Google Cloud Account\n",
        "\n",
        "# Authenticate gcloud.\n",
        "from google.colab import auth\n",
        "auth.authenticate_user()\n",
        "\n",
        "print(\"☑️ OK\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "iv07-mmUKrNJ"
      },
      "outputs": [],
      "source": [
        "#@markdown ####  Check authenticated user and project\n",
        "current_user = !gcloud auth list \\\n",
        "  --filter=\"status:ACTIVE\" \\\n",
        "  --format=\"value(account)\" \\\n",
        "  --quiet\n",
        "\n",
        "current_user = current_user[0]\n",
        "print(f\"Current user: {current_user}\")\n",
        "\n",
        "project_number = !gcloud projects list \\\n",
        "  --filter=\"projectId:{project_id}\" \\\n",
        "  --format=\"value(PROJECT_NUMBER)\" \\\n",
        "  --quiet\n",
        "\n",
        "project_number = project_number[0]\n",
        "print(f\"Project number: {project_number}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "#@markdown ####  Add IAM policy binding for default compute service account\n",
        "\n",
        "!gcloud projects add-iam-policy-binding \"{project_id}\" \\\n",
        "  --member=\"serviceAccount:{project_number}-compute@developer.gserviceaccount.com\" \\\n",
        "  --role=\"roles/aiplatform.user\" \\\n",
        "  --quiet\n",
        "\n",
        "print(\"☑️ Done\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RNdTod9pspSz"
      },
      "source": [
        "### APIs"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "ODuKfAD6Ku2M"
      },
      "outputs": [],
      "source": [
        "#@markdown #### Enable Google Cloud APIs\n",
        "#@markdown > Only necessary if the APIs are not yet activated in the project!\n",
        "\n",
        "# Enable APIs\n",
        "my_google_apis = [\n",
        "    \"aiplatform.googleapis.com\",\n",
        "    \"run.googleapis.com\",\n",
        "    \"artifactregistry.googleapis.com\",\n",
        "    \"cloudbuild.googleapis.com\",\n",
        "    \"containeranalysis.googleapis.com\",\n",
        "    \"containerscanning.googleapis.com\",\n",
        "]\n",
        "\n",
        "for api in my_google_apis :\n",
        "  print(f\"Enable API: {api}\")\n",
        "  !gcloud services enable \"{api}\" \\\n",
        "    --project=\"{project_id}\" \\\n",
        "    --quiet\n",
        "\n",
        "print(\"☑️ OK\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3trFNcHdz2JS"
      },
      "source": [
        "### Registry"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "kASHGe6Vz5vR"
      },
      "outputs": [],
      "source": [
        "#@markdown #### Create Artifact Registry repositoriy for Docker cointainer images\n",
        "#@markdown > Only necessary if the repositoriy does not already exist in the project and region!\n",
        "\n",
        "!gcloud artifacts repositories create \"{artifact_repository}\" \\\n",
        "  --repository-format=\"docker\"\\\n",
        "  --description=\"Docker contrainer registry\" \\\n",
        "  --location=\"{region}\" \\\n",
        "  --project=\"{project_id}\" \\\n",
        "  --quiet\n",
        "\n",
        "print(\"☑️ Done\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rPv9RMgPn5fu"
      },
      "source": [
        "## Deploy"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8GPSgec-n-rO"
      },
      "source": [
        "### Backend"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "2d6E978LnTFv"
      },
      "outputs": [],
      "source": [
        "#@markdown #### Build Docker cointainer for API backend\n",
        "\n",
        "backend_git = \"https://github.com/Cyclenerd/google-cloud-gcp-openai-api.git\"  # @param {type:\"string\"}\n",
        "backend_git_rev = \"master\"  # @param {type:\"string\"}\n",
        "assert backend_git, \"⚠️ Please provide a Git repo\"\n",
        "assert backend_git_rev, \"⚠️ Please provide a Git revision\"\n",
        "\n",
        "!gcloud builds submit \"{backend_git}\" \\\n",
        "    --git-source-revision=\"{backend_git_rev}\" \\\n",
        "    --tag=\"{region}-docker.pkg.dev/{project_id}/{artifact_repository}/vertex:latest\" \\\n",
        "    --timeout=\"1h\" \\\n",
        "    --region=\"{region}\" \\\n",
        "    --default-buckets-behavior=\"regional-user-owned-bucket\" \\\n",
        "    --quiet"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "tJTs9GDNnU_X"
      },
      "outputs": [],
      "source": [
        "#@markdown #### Deploy Cloud Run service with API backend\n",
        "\n",
        "!gcloud run deploy \"openai-api-vertex\" \\\n",
        "    --image=\"{region}-docker.pkg.dev/{project_id}/{artifact_repository}/vertex:latest\" \\\n",
        "    --description=\"OpenAI API for Google Cloud Vertex AI\" \\\n",
        "    --region=\"{region}\" \\\n",
        "    --set-env-vars=\"OPENAI_API_KEY={openai_key},GOOGLE_CLOUD_LOCATION={region},MODEL_NAME={model}\" \\\n",
        "    --max-instances=4 \\\n",
        "    --allow-unauthenticated \\\n",
        "    --quiet"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TnCpVBv0qPRq"
      },
      "source": [
        "### Frontend\n",
        "\n",
        "* [Chatbot UI](https://github.com/mckaywrigley/chatbot-ui):\n",
        "  * Git: `https://github.com/mckaywrigley/chatbot-ui.git`\n",
        "  * Rev: `main`\n",
        "* [Chatbot UI fork](https://github.com/Cyclenerd/chatbot-ui) with less OpenAI branding:\n",
        "  * Git: `https://github.com/Cyclenerd/chatbot-ui.git`\n",
        "  * Rev: `google`\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "2L1IKPAHqRgM"
      },
      "outputs": [],
      "source": [
        "#@markdown #### Build Docker cointainer for chat frontend\n",
        "\n",
        "frontend_git = \"https://github.com/Cyclenerd/chatbot-ui.git\"  # @param {type:\"string\"}\n",
        "frontend_git_rev = \"google\"  # @param {type:\"string\"}\n",
        "\n",
        "assert frontend_git, \"⚠️ Please provide a Git repo\"\n",
        "assert frontend_git_rev, \"⚠️ Please provide a Git revision\"\n",
        "\n",
        "!gcloud builds submit \"{frontend_git}\" \\\n",
        "    --git-source-revision=\"{frontend_git_rev}\" \\\n",
        "    --tag=\"{region}-docker.pkg.dev/{project_id}/{artifact_repository}/chatbot-ui:main\" \\\n",
        "    --timeout=\"1h\" \\\n",
        "    --region=\"{region}\" \\\n",
        "    --default-buckets-behavior=\"regional-user-owned-bucket\" \\\n",
        "    --quiet"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "X1cMzomWqxju"
      },
      "outputs": [],
      "source": [
        "#@markdown #### Deploy Cloud Run service with chat frontend\n",
        "\n",
        "# @markdown ✏️ Replace the Cloud Run service URL from the backend below:\n",
        "\n",
        "backend_url = \"https://openai-api-vertex-XYZ-AB.a.run.app\"  # @param {type:\"string\"}\n",
        "assert backend_url, \"⚠️ Please provide a Cloud Run backend URL\"\n",
        "\n",
        "!gcloud run deploy \"chatbot-ui\" \\\n",
        "    --image=\"{region}-docker.pkg.dev/{project_id}/{artifact_repository}/chatbot-ui:main\" \\\n",
        "    --description=\"Chatbot UI\" \\\n",
        "    --region=\"{region}\" \\\n",
        "    --set-env-vars=\"OPENAI_API_KEY={openai_key},OPENAI_API_HOST={backend_url}\" \\\n",
        "    --max-instances=2 \\\n",
        "    --allow-unauthenticated \\\n",
        "    --quiet"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z403ny7WnmaZ"
      },
      "source": [
        "## 🗑️ Clean up\n",
        "\n",
        "If you don't need the infrastructure anymore, you can delete it with the following snippets."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "jnpf_oOerltB"
      },
      "outputs": [],
      "source": [
        "#@markdown ### Delete Cloud Run service with chat frontend\n",
        "\n",
        "!gcloud run services delete \"chatbot-ui\" \\\n",
        "  --region=\"{region}\" \\\n",
        "  --project=\"{project_id}\" \\\n",
        "  --quiet"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "rZNIMYbrsr8L"
      },
      "outputs": [],
      "source": [
        "#@markdown ### Delete Cloud Run service with API backend\n",
        "\n",
        "!gcloud run services delete \"openai-api-vertex\" \\\n",
        "  --region=\"{region}\" \\\n",
        "  --project=\"{project_id}\" \\\n",
        "  --quiet"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "XxOVb5NZjT4y"
      },
      "outputs": [],
      "source": [
        "#@markdown ### Delete Artifact Registry repositoriy\n",
        "!gcloud artifacts repositories delete \"{artifact_repository}\" \\\n",
        "  --location=\"{region}\" \\\n",
        "  --project=\"{project_id}\" \\\n",
        "  --quiet"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}