server_examples.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Example Model Servers with Seldon\n",
    "\n",
    "Follow [docs](https://docs.seldon.io/projects/seldon-core/en/latest/) to install Seldon Core."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Error from server (AlreadyExists): namespaces \"seldon\" already exists\n"
     ]
    }
   ],
   "source": [
    "!kubectl create namespace seldon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Context \"kind-seldon\" modified.\n"
     ]
    }
   ],
   "source": [
    "!kubectl config set-context $(kubectl config current-context) --namespace=seldon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Serve SKLearn Iris Model\n",
    "\n",
    "In order to deploy SKLearn artifacts, we can leverage the [pre-packaged SKLearn inference server](https://docs.seldon.io/projects/seldon-core/en/latest/servers/sklearn.html).\n",
    "The exposed API can follow either:\n",
    "\n",
    "- The default Seldon protocol. \n",
    "- The V2 protocol.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "fatal: destination path 'seldon-core' already exists and is not an empty directory.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'1.15.0-dev'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## clone the seldon-core repo to parent directory\n",
    "!git clone https://github.com/SeldonIO/seldon-core\n",
    "ISTIO_GATEWAY = \"localhost:8004\"\n",
    "VERSION = !cat seldon-core/version.txt\n",
    "VERSION = VERSION[0]\n",
    "VERSION"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Default Seldon protocol\n",
    "\n",
    "To deploy and start serving an SKLearn artifact using Seldon's default protocol, we can use a config like the one below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting ./servers/sklearnserver/samples/iris.yaml\n"
     ]
    }
   ],
   "source": [
    "%%writefile ./servers/sklearnserver/samples/iris.yaml\n",
    "apiVersion: machinelearning.seldon.io/v1alpha2\n",
    "kind: SeldonDeployment\n",
    "metadata:\n",
    "  name: sklearn\n",
    "spec:\n",
    "  predictors:\n",
    "  - graph:\n",
    "      name: classifier\n",
    "      implementation: SKLEARN_SERVER\n",
    "      modelUri: gs://seldon-models/v1.15.0-dev/sklearn/iris\n",
    "    name: default\n",
    "    replicas: 1\n",
    "    svcOrchSpec: \n",
    "      env: \n",
    "      - name: SELDON_LOG_LEVEL\n",
    "        value: DEBUG"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then apply it to deploy it to our Kubernetes cluster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io \"sklearn\" deleted\n",
      "seldondeployment.machinelearning.seldon.io/sklearn created\n"
     ]
    }
   ],
   "source": [
    "!kubectl delete -f ./servers/sklearnserver/samples/iris.yaml\n",
    "!kubectl apply -f ./servers/sklearnserver/samples/iris.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting for deployment \"sklearn-default-0-classifier\" rollout to finish: 0 of 1 updated replicas are available...\n",
      "^C\n"
     ]
    }
   ],
   "source": [
    "!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=sklearn -o jsonpath='{.items[0].metadata.name}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once it's deployed we can send our sklearn model requests.\n",
    "Make sure istio port forward is done.\n",
    "`kubectl port-forward $(kubectl get pods -l istio=ingressgateway -n istio-system -o jsonpath='{.items[0].metadata.name}') -n istio-system 8004:8080`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### REST Requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "ename": "IndexError",
     "evalue": "list index out of range",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mIndexError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[1;32m/Users/dileep.gadiraju/projects/AI4BHARAT/try-seldon-core/server_examples.ipynb Cell 14\u001b[0m in \u001b[0;36m<cell line: 2>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      <a href='vscode-notebook-cell:/Users/dileep.gadiraju/projects/AI4BHARAT/try-seldon-core/server_examples.ipynb#X16sZmlsZQ%3D%3D?line=0'>1</a>\u001b[0m X\u001b[39m=\u001b[39mget_ipython()\u001b[39m.\u001b[39mgetoutput(\u001b[39m'\u001b[39m\u001b[39mcurl -s -d \u001b[39m\u001b[39m\\'\u001b[39;00m\u001b[39m{\u001b[39m\u001b[39m\"\u001b[39m\u001b[39mdata\u001b[39m\u001b[39m\"\u001b[39m\u001b[39m: \u001b[39m\u001b[39m{\u001b[39m\u001b[39m\"\u001b[39m\u001b[39mndarray\u001b[39m\u001b[39m\"\u001b[39m\u001b[39m:[[1.0, 2.0, 5.0, 6.0]]}}\u001b[39m\u001b[39m\\'\u001b[39;00m\u001b[39m     -X POST http://localhost:8004/seldon/seldon/sklearn/api/v1.0/predictions     -H \u001b[39m\u001b[39m\"\u001b[39m\u001b[39mContent-Type: application/json\u001b[39m\u001b[39m\"\u001b[39m\u001b[39m'\u001b[39m)\n\u001b[0;32m----> <a href='vscode-notebook-cell:/Users/dileep.gadiraju/projects/AI4BHARAT/try-seldon-core/server_examples.ipynb#X16sZmlsZQ%3D%3D?line=1'>2</a>\u001b[0m d\u001b[39m=\u001b[39mjson\u001b[39m.\u001b[39mloads(X[\u001b[39m0\u001b[39;49m])\n\u001b[1;32m      <a href='vscode-notebook-cell:/Users/dileep.gadiraju/projects/AI4BHARAT/try-seldon-core/server_examples.ipynb#X16sZmlsZQ%3D%3D?line=2'>3</a>\u001b[0m \u001b[39mprint\u001b[39m(d)\n",
      "\u001b[0;31mIndexError\u001b[0m: list index out of range"
     ]
    }
   ],
   "source": [
    "X=!curl -s -d '{\"data\": {\"ndarray\":[[1.0, 2.0, 5.0, 6.0]]}}' \\\n",
    "   -X POST http://localhost:8004/seldon/seldon/sklearn/api/v1.0/predictions \\\n",
    "   -H \"Content-Type: application/json\"\n",
    "d=json.loads(X[0])\n",
    "print(d)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from seldon_core.seldon_client import SeldonClient\n",
    "\n",
    "sc = SeldonClient(deployment_name=\"sklearn\", namespace=\"seldon\",gateway_endpoint=ISTIO_GATEWAY)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Success:True message:\n",
      "Request:\n",
      "meta {\n",
      "}\n",
      "data {\n",
      "  tensor {\n",
      "    shape: 1\n",
      "    shape: 4\n",
      "    values: 0.6631433882135224\n",
      "    values: 0.09420686306171677\n",
      "    values: 0.4228270015431025\n",
      "    values: 0.6975775879782301\n",
      "  }\n",
      "}\n",
      "\n",
      "Response:\n",
      "{'data': {'names': ['t:0', 't:1', 't:2'], 'tensor': {'shape': [1, 3], 'values': [0.18342520713992527, 0.42067504473566336, 0.39589974812441137]}}, 'meta': {'requestPath': {'classifier': 'seldonio/sklearnserver:1.15.0'}}}\n"
     ]
    }
   ],
   "source": [
    "r = sc.predict(gateway=\"istio\", transport=\"rest\", shape=(1, 4))\n",
    "print(r)\n",
    "assert r.success == True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### gRPC Requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Success:True message:\n",
      "Request:\n",
      "{'meta': {}, 'data': {'tensor': {'shape': [1, 4], 'values': [0.7222072429970986, 0.5651491139269813, 0.2981192849265121, 0.14109657217323435]}}}\n",
      "Response:\n",
      "{'meta': {'requestPath': {'classifier': 'seldonio/sklearnserver:1.15.0'}}, 'data': {'names': ['t:0', 't:1', 't:2'], 'tensor': {'shape': [1, 3], 'values': [0.47278739561743405, 0.45455842695760823, 0.07265417742495756]}}}\n"
     ]
    }
   ],
   "source": [
    "r = sc.predict(gateway=\"istio\", transport=\"grpc\", shape=(1, 4))\n",
    "print(r)\n",
    "assert r.success == True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'meta': {'requestPath': {'classifier': 'seldonio/sklearnserver:1.15.0'}}, 'data': {'names': ['t:0', 't:1', 't:2'], 'ndarray': [[9.912315378486697e-07, 0.0007015931307746079, 0.9992974156376876]]}}\n"
     ]
    }
   ],
   "source": [
    "X=!cd ./seldon-core/executor/proto && grpcurl -d '{\"data\":{\"ndarray\":[[1.0,2.0,5.0,6.0]]}}' \\\n",
    "         -rpc-header seldon:sklearn -rpc-header namespace:seldon \\\n",
    "         -plaintext \\\n",
    "         -proto ./prediction.proto  0.0.0.0:8004 seldon.protos.Seldon/Predict\n",
    "d=json.loads(\"\".join(X))\n",
    "print(d)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And delete the model we deployed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io \"sklearn\" deleted\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl delete -f ./servers/sklearnserver/samples/iris.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### V2 protocol\n",
    "\n",
    "For example, we can consider the config below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting ./resources/iris-sklearn-v2.yaml\n"
     ]
    }
   ],
   "source": [
    "%%writefile ./resources/iris-sklearn-v2.yaml\n",
    "apiVersion: machinelearning.seldon.io/v1\n",
    "kind: SeldonDeployment\n",
    "metadata:\n",
    "  name: sklearn\n",
    "spec:\n",
    "  name: iris\n",
    "  protocol: v2\n",
    "  predictors:\n",
    "  - graph:\n",
    "      children: []\n",
    "      implementation: SKLEARN_SERVER\n",
    "      modelUri: gs://seldon-models/sklearn/iris-0.23.2/lr_model\n",
    "      name: classifier\n",
    "    name: default\n",
    "    replicas: 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then apply it to deploy our model to our Kubernetes cluster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io/sklearn created\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl apply -f resources/iris-sklearn-v2.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting for deployment \"sklearn-default-0-classifier\" rollout to finish: 0 of 1 updated replicas are available...\n",
      "deployment \"sklearn-default-0-classifier\" successfully rolled out\n"
     ]
    }
   ],
   "source": [
    "!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=sklearn -o jsonpath='{.items[0].metadata.name}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once it's deployed, we can send inference requests to our model.\n",
    "Note that, since it's using the V2 Protocol, these requests will be different to the ones using the default Seldon Protocol."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<Response [404]>\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "\n",
    "import requests\n",
    "\n",
    "inference_request = {\n",
    "    \"inputs\": [\n",
    "        {\"name\": \"predict\", \"shape\": [1, 4], \"datatype\": \"FP32\", \"data\": [[1, 2, 3, 4]]}\n",
    "    ]\n",
    "}\n",
    "\n",
    "endpoint = \"http://localhost:8004/seldon/seldon/sklearn/v2/models/infer\"\n",
    "response = requests.post(endpoint, json=inference_request)\n",
    "print(response)\n",
    "print(json.dumps(response.json(), indent=2))\n",
    "assert response.ok"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can delete the model we deployed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Error from server (NotFound): error when deleting \"resources/iris-sklearn-v2.yaml\": seldondeployments.machinelearning.seldon.io \"sklearn\" not found\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl delete -f resources/iris-sklearn-v2.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Serve XGBoost Iris Model\n",
    "\n",
    "In order to deploy XGBoost models, we can leverage the [pre-packaged XGBoost inference server](https://docs.seldon.io/projects/seldon-core/en/latest/servers/xgboost.html).\n",
    "The exposed API can follow either:\n",
    "\n",
    "- The default Seldon protocol. \n",
    "- The V2 protocol"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Default Seldon protocol\n",
    "\n",
    "We can deploy a XGBoost model uploaded to an object store by using the XGBoost model server implementation as shown in the config below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting resources/iris.yaml\n"
     ]
    }
   ],
   "source": [
    "%%writefile resources/iris.yaml\n",
    "apiVersion: machinelearning.seldon.io/v1\n",
    "kind: SeldonDeployment\n",
    "metadata:\n",
    "  name: xgboost\n",
    "spec:\n",
    "  name: iris\n",
    "  predictors:\n",
    "  - graph:\n",
    "      children: []\n",
    "      implementation: XGBOOST_SERVER\n",
    "      modelUri: gs://seldon-models/xgboost/iris\n",
    "      name: classifier\n",
    "    name: default\n",
    "    replicas: 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And then we apply it to deploy it to our kubernetes cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io/xgboost unchanged\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl apply -f resources/iris.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "deployment \"xgboost-default-0-classifier\" successfully rolled out\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=xgboost -o jsonpath='{.items[0].metadata.name}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Rest Requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'data': {'names': [], 'ndarray': [2.0]}, 'meta': {'requestPath': {'classifier': 'seldonio/xgboostserver:1.15.0'}}}\n"
     ]
    }
   ],
   "source": [
    "X=!curl -s -d '{\"data\": {\"ndarray\":[[1.0, 2.0, 5.0, 6.0]]}}' \\\n",
    "   -X POST http://localhost:8004/seldon/seldon/xgboost/api/v1.0/predictions \\\n",
    "   -H \"Content-Type: application/json\"\n",
    "d=json.loads(X[0])\n",
    "print(d)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "from seldon_core.seldon_client import SeldonClient\n",
    "\n",
    "sc = SeldonClient(deployment_name=\"xgboost\", namespace=\"seldon\",gateway_endpoint=ISTIO_GATEWAY)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Success:True message:\n",
      "Request:\n",
      "meta {\n",
      "}\n",
      "data {\n",
      "  tensor {\n",
      "    shape: 1\n",
      "    shape: 4\n",
      "    values: 0.5218017888418626\n",
      "    values: 0.7335763045185455\n",
      "    values: 0.9060568776905242\n",
      "    values: 0.9286064648595698\n",
      "  }\n",
      "}\n",
      "\n",
      "Response:\n",
      "{'data': {'names': [], 'tensor': {'shape': [1], 'values': [0.0]}}, 'meta': {'requestPath': {'classifier': 'seldonio/xgboostserver:1.15.0'}}}\n"
     ]
    }
   ],
   "source": [
    "r = sc.predict(gateway=\"istio\", transport=\"rest\", shape=(1, 4))\n",
    "print(r)\n",
    "assert r.success == True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### gRPC Requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Success:True message:\n",
      "Request:\n",
      "{'meta': {}, 'data': {'tensor': {'shape': [1, 4], 'values': [0.3433527754732981, 0.2959236682173807, 0.3176180264508235, 0.5409599645513264]}}}\n",
      "Response:\n",
      "{'meta': {'requestPath': {'classifier': 'seldonio/xgboostserver:1.15.0'}}, 'data': {'tensor': {'shape': [1], 'values': [0.0]}}}\n"
     ]
    }
   ],
   "source": [
    "r = sc.predict(gateway=\"istio\", transport=\"grpc\", shape=(1, 4))\n",
    "print(r)\n",
    "assert r.success == True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'meta': {'requestPath': {'classifier': 'seldonio/xgboostserver:1.15.0'}}, 'data': {'ndarray': [2]}}\n"
     ]
    }
   ],
   "source": [
    "X=!cd ./seldon-core/executor/proto && grpcurl -d '{\"data\":{\"ndarray\":[[1.0,2.0,5.0,6.0]]}}' \\\n",
    "         -rpc-header seldon:xgboost -rpc-header namespace:seldon \\\n",
    "         -plaintext \\\n",
    "         -proto ./prediction.proto  0.0.0.0:8004 seldon.protos.Seldon/Predict\n",
    "d=json.loads(\"\".join(X))\n",
    "print(d)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And delete the model we deployed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io \"xgboost\" deleted\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl delete -f resources/iris.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### V2 protocol\n",
    "\n",
    "We can deploy a XGBoost model, exposing an API compatible with v2 protocol by specifying the `protocol` of our `SeldonDeployment` as `v2`.\n",
    "For example, we can consider the config below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing ./resources/iris-xgboost-v2.yaml\n"
     ]
    }
   ],
   "source": [
    "%%writefile ./resources/iris-xgboost-v2.yaml\n",
    "apiVersion: machinelearning.seldon.io/v1\n",
    "kind: SeldonDeployment\n",
    "metadata:\n",
    "  name: xgboost\n",
    "spec:\n",
    "  name: iris\n",
    "  protocol: v2\n",
    "  predictors:\n",
    "  - graph:\n",
    "      children: []\n",
    "      implementation: XGBOOST_SERVER\n",
    "      modelUri: gs://seldon-models/xgboost/iris\n",
    "      name: classifier\n",
    "    name: default\n",
    "    replicas: 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then apply it to deploy our model to our Kubernetes cluster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io/xgboost created\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl apply -f ./resources/iris-xgboost-v2.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting for deployment \"xgboost-default-0-classifier\" rollout to finish: 0 of 1 updated replicas are available...\n",
      "deployment \"xgboost-default-0-classifier\" successfully rolled out\n"
     ]
    }
   ],
   "source": [
    "!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=xgboost -o jsonpath='{.items[0].metadata.name}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once it's deployed, we can send inference requests to our model.\n",
    "Note that, since it's using the V2 Protocol, these requests will be different to the ones using the default Seldon Protocol."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"model_name\": \"classifier\",\n",
      "  \"model_version\": \"v1\",\n",
      "  \"id\": \"8a27889d-87a2-4720-9b0c-c225b9d60c7f\",\n",
      "  \"parameters\": {\n",
      "    \"content_type\": null,\n",
      "    \"headers\": null\n",
      "  },\n",
      "  \"outputs\": [\n",
      "    {\n",
      "      \"name\": \"predict\",\n",
      "      \"shape\": [\n",
      "        1\n",
      "      ],\n",
      "      \"datatype\": \"FP32\",\n",
      "      \"parameters\": null,\n",
      "      \"data\": [\n",
      "        2.0\n",
      "      ]\n",
      "    }\n",
      "  ]\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "\n",
    "import requests\n",
    "\n",
    "inference_request = {\n",
    "    \"inputs\": [\n",
    "        {\"name\": \"predict\", \"shape\": [1, 4], \"datatype\": \"FP32\", \"data\": [[1, 2, 3, 4]]}\n",
    "    ]\n",
    "}\n",
    "\n",
    "endpoint = \"http://localhost:8004/seldon/seldon/xgboost/v2/models/infer\"\n",
    "response = requests.post(endpoint, json=inference_request)\n",
    "\n",
    "print(json.dumps(response.json(), indent=2))\n",
    "assert response.ok"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can delete the model we deployed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io \"xgboost\" deleted\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl delete -f ./resources/iris-xgboost-v2.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Serve Tensorflow MNIST Model\n",
    "We can deploy a tensorflow model uploaded to an object store by using the\n",
    "tensorflow model server implementation as the config below.\n",
    "\n",
    "This notebook contains two examples, one which shows how you can use the\n",
    "TFServing prepackaged serve with the Seldon Protocol, and a second one which\n",
    "shows how you can deploy it using the tensorlfow protocol (so you can send\n",
    "requests of the exact format as you would to a tfserving server)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Serve Tensorflow MNIST Model with Seldon Protocol\n",
    "\n",
    "The config file below shows how you can deploy your Tensorflow model which\n",
    "exposes the Seldon protocol."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing ./resources/mnist_rest.yaml\n"
     ]
    }
   ],
   "source": [
    "%%writefile ./resources/mnist_rest.yaml\n",
    "apiVersion: machinelearning.seldon.io/v1alpha2\n",
    "kind: SeldonDeployment\n",
    "metadata:\n",
    "  name: tfserving\n",
    "spec:\n",
    "  name: mnist\n",
    "  predictors:\n",
    "  - graph:\n",
    "      children: []\n",
    "      implementation: TENSORFLOW_SERVER\n",
    "      modelUri: gs://seldon-models/tfserving/mnist-model\n",
    "      name: mnist-model\n",
    "      parameters:\n",
    "        - name: signature_name\n",
    "          type: STRING\n",
    "          value: predict_images\n",
    "        - name: model_name\n",
    "          type: STRING\n",
    "          value: mnist-model\n",
    "        - name: model_input\n",
    "          type: STRING\n",
    "          value: images\n",
    "        - name: model_output\n",
    "          type: STRING\n",
    "          value: scores     \n",
    "    name: default\n",
    "    replicas: 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io/tfserving created\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl apply -f ./resources/mnist_rest.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting for deployment \"tfserving-default-0-mnist-model\" rollout to finish: 0 of 1 updated replicas are available...\n",
      "deployment \"tfserving-default-0-mnist-model\" successfully rolled out\n"
     ]
    }
   ],
   "source": [
    "!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=tfserving -o jsonpath='{.items[0].metadata.name}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "from seldon_core.seldon_client import SeldonClient\n",
    "\n",
    "sc = SeldonClient(deployment_name=\"tfserving\", namespace=\"seldon\",gateway_endpoint=ISTIO_GATEWAY)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### REST Request"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Success:True message:\n",
      "Request:\n",
      "meta {\n",
      "}\n",
      "data {\n",
      "  tensor {\n",
      "    shape: 1\n",
      "    shape: 784\n",
      "    values: 0.4689572966007861\n",
      "    values: 0.9660213976358323\n",
      "    values: 0.2439077409486442\n",
      "    values: 0.8575884865204007\n",
      "    values: 0.27970466773693103\n",
      "    values: 0.6353320891612885\n",
      "    values: 0.0971403754673349\n",
      "    values: 0.7164836350133393\n",
      "    values: 0.542073728393848\n",
      "    values: 0.11666104614030115\n",
      "    values: 0.4782253951392831\n",
      "    values: 0.7634452287588545\n",
      "    values: 0.3912767509737113\n",
      "    values: 0.2663996274182098\n",
      "    values: 0.6490030540413255\n",
      "    values: 0.19177236373841555\n",
      "    values: 0.7778343694560058\n",
      "    values: 0.012348665103772305\n",
      "    values: 0.6457729473685598\n",
      "    values: 0.360291654824512\n",
      "    values: 0.47249733414191286\n",
      "    values: 0.48503650958931044\n",
      "    values: 0.6901668112987778\n",
      "    values: 0.11909318221150922\n",
      "    values: 0.2110322062384462\n",
      "    values: 0.8754846146707363\n",
      "    values: 0.2406771324805147\n",
      "    values: 0.9638406936053503\n",
      "    values: 0.44412651138523596\n",
      "    values: 0.881529648323643\n",
      "    values: 0.6618790267488508\n",
      "    values: 0.8397443295516367\n",
      "    values: 0.3485875378767914\n",
      "    values: 0.06787703135983891\n",
      "    values: 0.6626849978621995\n",
      "    values: 0.8812856448637397\n",
      "    values: 0.16883922841917964\n",
      "    values: 0.8252057533978429\n",
      "    values: 0.7063852663334077\n",
      "    values: 0.755491275537365\n",
      "    values: 0.06189195461923569\n",
      "    values: 0.4974847199622653\n",
      "    values: 0.7126909547880684\n",
      "    values: 0.652842043062244\n",
      "    values: 0.8004882055837098\n",
      "    values: 0.4513876509393846\n",
      "    values: 0.03309579109301086\n",
      "    values: 0.00657342598264854\n",
      "    values: 0.07583134187796692\n",
      "    values: 0.5413680581099638\n",
      "    values: 0.5221281994737917\n",
      "    values: 0.12559187387102266\n",
      "    values: 0.6129595425026247\n",
      "    values: 0.6791503388302546\n",
      "    values: 0.45315490337643094\n",
      "    values: 0.43301498881702893\n",
      "    values: 0.9620258569120063\n",
      "    values: 0.4113222760303864\n",
      "    values: 0.662329966082312\n",
      "    values: 0.5052398862242359\n",
      "    values: 0.29098549391732864\n",
      "    values: 0.9776701121320753\n",
      "    values: 0.028477769091102\n",
      "    values: 0.35490682919640437\n",
      "    values: 0.7671959445872129\n",
      "    values: 0.5032956867244794\n",
      "    values: 0.8353945311082279\n",
      "    values: 0.02290912426050129\n",
      "    values: 0.1003999689225985\n",
      "    values: 0.214539428616232\n",
      "    values: 0.061551256159072754\n",
      "    values: 0.48262955720776135\n",
      "    values: 0.7372304511018402\n",
      "    values: 0.5344277607686541\n",
      "    values: 0.20091080490066726\n",
      "    values: 0.42068554421978144\n",
      "    values: 0.8266444994340592\n",
      "    values: 0.6517342839771983\n",
      "    values: 0.17680855666211215\n",
      "    values: 0.8722014078384503\n",
      "    values: 0.4077793865384568\n",
      "    values: 0.6252533986148548\n",
      "    values: 0.559603864452411\n",
      "    values: 0.24795741856587272\n",
      "    values: 0.7574444141977343\n",
      "    values: 0.5593664939479072\n",
      "    values: 0.3808921199407135\n",
      "    values: 0.6564733531039216\n",
      "    values: 0.9763953622035737\n",
      "    values: 0.5005143027457326\n",
      "    values: 0.7784046669043834\n",
      "    values: 0.0290215618799069\n",
      "    values: 0.48155543798934675\n",
      "    values: 0.20338522092177513\n",
      "    values: 0.5180937017600729\n",
      "    values: 0.007284218430685718\n",
      "    values: 0.12707899616322715\n",
      "    values: 0.882450551564275\n",
      "    values: 0.0016327184543928341\n",
      "    values: 0.9420675768867449\n",
      "    values: 0.5667828725493479\n",
      "    values: 0.20542091329662326\n",
      "    values: 0.7130984987481607\n",
      "    values: 0.5282963783368142\n",
      "    values: 0.9169456607694242\n",
      "    values: 0.47949176572498997\n",
      "    values: 0.1253143095083863\n",
      "    values: 0.47004577556093385\n",
      "    values: 0.0549514639638371\n",
      "    values: 0.24355883429707847\n",
      "    values: 0.9586101420616553\n",
      "    values: 0.9783037217179998\n",
      "    values: 0.576869248607957\n",
      "    values: 0.49676236888884373\n",
      "    values: 0.6663366822917971\n",
      "    values: 0.6394575824009932\n",
      "    values: 0.35460279721851895\n",
      "    values: 0.9153761644427794\n",
      "    values: 0.28039831357731726\n",
      "    values: 0.7171434496113136\n",
      "    values: 0.5078958025425561\n",
      "    values: 0.4969518205135266\n",
      "    values: 0.6130388810766317\n",
      "    values: 0.028565070862728525\n",
      "    values: 0.5539164872499862\n",
      "    values: 0.240060732702025\n",
      "    values: 0.5889812268587027\n",
      "    values: 0.199352065911806\n",
      "    values: 0.0597109833693783\n",
      "    values: 0.8824041919948532\n",
      "    values: 0.8805034495914273\n",
      "    values: 0.9921738517227549\n",
      "    values: 0.5357508981752857\n",
      "    values: 0.7677131619102785\n",
      "    values: 0.6554839516654969\n",
      "    values: 0.715467740561536\n",
      "    values: 0.9272387339048027\n",
      "    values: 0.8809228988588731\n",
      "    values: 0.3209759245024396\n",
      "    values: 0.7489873378593056\n",
      "    values: 0.4607960345528488\n",
      "    values: 0.6670646724993519\n",
      "    values: 0.12732327525854537\n",
      "    values: 0.2784630683186776\n",
      "    values: 0.2650300674199002\n",
      "    values: 0.9316565151603778\n",
      "    values: 0.7776604203405351\n",
      "    values: 0.6417155717399466\n",
      "    values: 0.8507369941708919\n",
      "    values: 0.642351335215158\n",
      "    values: 0.7474179245633135\n",
      "    values: 0.21804129569925934\n",
      "    values: 0.8728494471023033\n",
      "    values: 0.7622332174240901\n",
      "    values: 0.20074749368728573\n",
      "    values: 0.44151558244462397\n",
      "    values: 0.8425154869015485\n",
      "    values: 0.04741449995483615\n",
      "    values: 0.9440997983101715\n",
      "    values: 0.42411023515214175\n",
      "    values: 0.9425874100746151\n",
      "    values: 0.6609957150870998\n",
      "    values: 0.2224180745125267\n",
      "    values: 0.7491503887372211\n",
      "    values: 0.6698190347246741\n",
      "    values: 0.2658451164886243\n",
      "    values: 0.7371933052461332\n",
      "    values: 0.7521279131436975\n",
      "    values: 0.8023319183818733\n",
      "    values: 0.16903036776620772\n",
      "    values: 0.392255802986201\n",
      "    values: 0.6073635848658655\n",
      "    values: 0.9141628341684109\n",
      "    values: 0.0736069754501647\n",
      "    values: 0.09487816327712595\n",
      "    values: 0.8125345860584032\n",
      "    values: 0.41187848321379905\n",
      "    values: 0.804141683659572\n",
      "    values: 0.41162917221608486\n",
      "    values: 0.3802617751936269\n",
      "    values: 0.13495988261916525\n",
      "    values: 0.7327747677612214\n",
      "    values: 0.8216492706687886\n",
      "    values: 0.8925241634244648\n",
      "    values: 0.35974622794828015\n",
      "    values: 0.7374549734965586\n",
      "    values: 0.957184763684864\n",
      "    values: 0.5025483933122916\n",
      "    values: 0.4168941052762597\n",
      "    values: 0.8671548452471981\n",
      "    values: 0.5497357837775748\n",
      "    values: 0.9707837018118113\n",
      "    values: 0.20389618021218125\n",
      "    values: 0.18267465664805194\n",
      "    values: 0.6972493132301242\n",
      "    values: 0.3412328376963669\n",
      "    values: 0.7868062860405705\n",
      "    values: 0.08583808392294467\n",
      "    values: 0.8869677945525772\n",
      "    values: 0.8017836514251736\n",
      "    values: 0.6722662464576948\n",
      "    values: 0.8027907640349397\n",
      "    values: 0.14698363038584916\n",
      "    values: 0.20144664170260296\n",
      "    values: 0.44093119712362017\n",
      "    values: 0.07688415486930134\n",
      "    values: 0.3381736828524292\n",
      "    values: 0.46056831693531086\n",
      "    values: 0.08377123641100592\n",
      "    values: 0.1793226922489375\n",
      "    values: 0.5565266871134699\n",
      "    values: 0.169719790312172\n",
      "    values: 0.3637372451113221\n",
      "    values: 0.1713791963139838\n",
      "    values: 0.9004228125576195\n",
      "    values: 0.1764863049541222\n",
      "    values: 0.13739904155364524\n",
      "    values: 0.42235494018969544\n",
      "    values: 0.6399024573514367\n",
      "    values: 0.2898130150304964\n",
      "    values: 0.42714582228374365\n",
      "    values: 0.5168483558141002\n",
      "    values: 0.5001493340996458\n",
      "    values: 0.6318618395259056\n",
      "    values: 0.8265520476218083\n",
      "    values: 0.06561618673531144\n",
      "    values: 0.7104739935480615\n",
      "    values: 0.07347067061575019\n",
      "    values: 0.4912802558595146\n",
      "    values: 0.9248949708720032\n",
      "    values: 0.8504480033584457\n",
      "    values: 0.5411803439299754\n",
      "    values: 0.018632800284874018\n",
      "    values: 0.03933673312901642\n",
      "    values: 0.29302364566465167\n",
      "    values: 0.5996594822918598\n",
      "    values: 0.5345540817887916\n",
      "    values: 0.6249000782987766\n",
      "    values: 0.5352173322222971\n",
      "    values: 0.17135195819406646\n",
      "    values: 0.5615449233098269\n",
      "    values: 0.7705676479994685\n",
      "    values: 0.33911541430413006\n",
      "    values: 0.1105653586812112\n",
      "    values: 0.29896904779015254\n",
      "    values: 0.7417343051961591\n",
      "    values: 0.3285158724224191\n",
      "    values: 0.9655574031731488\n",
      "    values: 0.27502018326408806\n",
      "    values: 0.26430225088006176\n",
      "    values: 0.8392120803544353\n",
      "    values: 0.7221415123509165\n",
      "    values: 0.4331463601841652\n",
      "    values: 0.037257146215731285\n",
      "    values: 0.9919427451343519\n",
      "    values: 0.08522517507238958\n",
      "    values: 0.35715656244380034\n",
      "    values: 0.031254515932164106\n",
      "    values: 0.16219251523389222\n",
      "    values: 0.13700721277188987\n",
      "    values: 0.9723513533755028\n",
      "    values: 0.43754528184711605\n",
      "    values: 0.30796797096853346\n",
      "    values: 0.6836694638079737\n",
      "    values: 0.33824648736075613\n",
      "    values: 0.5545249604720329\n",
      "    values: 0.27832739111007954\n",
      "    values: 0.854928752974259\n",
      "    values: 0.4895133422280753\n",
      "    values: 0.6239893440542622\n",
      "    values: 0.8395599571965727\n",
      "    values: 0.6192761265064569\n",
      "    values: 0.805652305901602\n",
      "    values: 0.2585848720368159\n",
      "    values: 0.6553465280018973\n",
      "    values: 0.7557470955621153\n",
      "    values: 0.4310621702752131\n",
      "    values: 0.001796984804001367\n",
      "    values: 0.5929764185322879\n",
      "    values: 0.7335813973955654\n",
      "    values: 0.15568408380245569\n",
      "    values: 0.8858022795972084\n",
      "    values: 0.4578939539513034\n",
      "    values: 0.6354048565906055\n",
      "    values: 0.4298657332177055\n",
      "    values: 0.733177048083205\n",
      "    values: 0.8204744790373025\n",
      "    values: 0.14273507244910777\n",
      "    values: 0.8710725145822141\n",
      "    values: 0.5744191356158549\n",
      "    values: 0.7050745234490209\n",
      "    values: 0.8313123358381252\n",
      "    values: 0.5175944924859623\n",
      "    values: 0.47593568241038176\n",
      "    values: 0.9425045042025043\n",
      "    values: 0.8175184938414422\n",
      "    values: 0.8529928429012026\n",
      "    values: 0.21144288266472178\n",
      "    values: 0.48369060173026057\n",
      "    values: 0.9047078098433198\n",
      "    values: 0.034964507070033624\n",
      "    values: 0.06384928302687687\n",
      "    values: 0.35956652926301125\n",
      "    values: 0.5658614333124411\n",
      "    values: 0.17974187891556115\n",
      "    values: 0.5370210711968414\n",
      "    values: 0.8735438816941556\n",
      "    values: 0.3329178067875761\n",
      "    values: 0.5883810420097814\n",
      "    values: 0.387677244173279\n",
      "    values: 0.9895241163096693\n",
      "    values: 0.8614526026884918\n",
      "    values: 0.4193677200661229\n",
      "    values: 0.35016451015217476\n",
      "    values: 0.11745633765596242\n",
      "    values: 0.10388458884304208\n",
      "    values: 0.16619430256185508\n",
      "    values: 0.2815345602029653\n",
      "    values: 0.665518722631449\n",
      "    values: 0.03750859155262465\n",
      "    values: 0.3017559858259018\n",
      "    values: 0.8625535530204922\n",
      "    values: 0.5394994872636965\n",
      "    values: 0.8942073991475429\n",
      "    values: 0.42159577452808406\n",
      "    values: 0.2768596633242356\n",
      "    values: 0.728542630986381\n",
      "    values: 0.20589994935129086\n",
      "    values: 0.17247530561377933\n",
      "    values: 0.12995704160349808\n",
      "    values: 0.10055919741429908\n",
      "    values: 0.8136731197270176\n",
      "    values: 0.44528710882704925\n",
      "    values: 0.1730869587492455\n",
      "    values: 0.44578160380216514\n",
      "    values: 0.10520003737013539\n",
      "    values: 0.08605320003511752\n",
      "    values: 0.7756023510388784\n",
      "    values: 0.6118780824357198\n",
      "    values: 0.9275683583322913\n",
      "    values: 0.8010245500228894\n",
      "    values: 0.3016655002156452\n",
      "    values: 0.8565944678054914\n",
      "    values: 0.43741154899823054\n",
      "    values: 0.30767716252325095\n",
      "    values: 0.22932883013530592\n",
      "    values: 0.29601142236076694\n",
      "    values: 0.8086639829191657\n",
      "    values: 0.6641245045299766\n",
      "    values: 0.4360749750792461\n",
      "    values: 0.571004813744974\n",
      "    values: 0.23672638295825443\n",
      "    values: 0.4085487249436095\n",
      "    values: 0.01099898583077663\n",
      "    values: 0.0039279708840513905\n",
      "    values: 0.13064345109116626\n",
      "    values: 0.38382428546821623\n",
      "    values: 0.1811008929385186\n",
      "    values: 0.7806177055384026\n",
      "    values: 0.6991736182189345\n",
      "    values: 0.2403558830989776\n",
      "    values: 0.7187015164764012\n",
      "    values: 0.9833389835593405\n",
      "    values: 0.10229155931956857\n",
      "    values: 0.027545150026656917\n",
      "    values: 0.7106511285864645\n",
      "    values: 0.20305647245307257\n",
      "    values: 0.2816931684017724\n",
      "    values: 0.5059268285216719\n",
      "    values: 0.24056777765675152\n",
      "    values: 0.8043576967983141\n",
      "    values: 0.1292577683862557\n",
      "    values: 0.6503101066661767\n",
      "    values: 0.9622450994878421\n",
      "    values: 0.1807894798529902\n",
      "    values: 0.05977478779994572\n",
      "    values: 0.6257905334168709\n",
      "    values: 0.7837008061153413\n",
      "    values: 0.9621374573617909\n",
      "    values: 0.544643872143762\n",
      "    values: 0.059379518507757245\n",
      "    values: 0.18576539190280938\n",
      "    values: 0.5717012830702727\n",
      "    values: 0.5578725597387859\n",
      "    values: 0.8199060415576522\n",
      "    values: 0.36991706125921875\n",
      "    values: 0.3710921860625145\n",
      "    values: 0.10434450648724858\n",
      "    values: 0.9514397316323804\n",
      "    values: 0.7763267327939336\n",
      "    values: 0.9807699517422676\n",
      "    values: 0.8103811972265189\n",
      "    values: 0.11977620955951962\n",
      "    values: 0.17285019767254073\n",
      "    values: 0.695780348159029\n",
      "    values: 0.9840211678975557\n",
      "    values: 0.09885139066918547\n",
      "    values: 0.9073713296040896\n",
      "    values: 0.34786543093711075\n",
      "    values: 0.002887695948131208\n",
      "    values: 0.5066615550636354\n",
      "    values: 0.7858537119694285\n",
      "    values: 0.9292471259310581\n",
      "    values: 0.019674312550819084\n",
      "    values: 0.42776561099068\n",
      "    values: 0.01069690651139732\n",
      "    values: 0.9469625660520463\n",
      "    values: 0.527292624768451\n",
      "    values: 0.541988621103728\n",
      "    values: 0.4730299804855522\n",
      "    values: 0.3513435182017153\n",
      "    values: 0.45219155833940317\n",
      "    values: 0.39532689116729325\n",
      "    values: 0.9845599543207586\n",
      "    values: 0.3567656870706176\n",
      "    values: 0.24616532040426375\n",
      "    values: 0.8983462547968689\n",
      "    values: 0.2675972086837587\n",
      "    values: 0.20983702831642415\n",
      "    values: 0.8398808421024443\n",
      "    values: 0.484268379226129\n",
      "    values: 0.04284803333162612\n",
      "    values: 0.7302394002064223\n",
      "    values: 0.5738767266853044\n",
      "    values: 0.2666142396979124\n",
      "    values: 0.5915904814531608\n",
      "    values: 0.574735231708747\n",
      "    values: 0.8815006207070645\n",
      "    values: 0.04035859433375655\n",
      "    values: 0.4520177174201474\n",
      "    values: 0.5098781176757613\n",
      "    values: 0.3816255021681172\n",
      "    values: 0.15734553878209057\n",
      "    values: 0.7956101724023728\n",
      "    values: 0.9727887792853369\n",
      "    values: 0.46050860697235785\n",
      "    values: 0.8867925787142024\n",
      "    values: 0.9643621166678711\n",
      "    values: 0.2257532859375636\n",
      "    values: 0.874036754642226\n",
      "    values: 0.8558009364920456\n",
      "    values: 0.7516369683736298\n",
      "    values: 0.07531594142737508\n",
      "    values: 0.23636529347797297\n",
      "    values: 0.8046054632886251\n",
      "    values: 0.017480925925793867\n",
      "    values: 0.3031903022485112\n",
      "    values: 0.3543152704943371\n",
      "    values: 0.6930579831620294\n",
      "    values: 0.9602092707641767\n",
      "    values: 0.585064758787818\n",
      "    values: 0.5240480842845692\n",
      "    values: 0.4366598865308784\n",
      "    values: 0.18005714913376258\n",
      "    values: 0.9953868561243212\n",
      "    values: 0.6546077160952924\n",
      "    values: 0.26167298259678784\n",
      "    values: 0.24376922423041192\n",
      "    values: 0.296591608471621\n",
      "    values: 0.13964592592839586\n",
      "    values: 0.17492939931140172\n",
      "    values: 0.7128539455899825\n",
      "    values: 0.8386086458398756\n",
      "    values: 0.18175205823594087\n",
      "    values: 0.5499987931701247\n",
      "    values: 0.038239443958483554\n",
      "    values: 0.656652820808902\n",
      "    values: 0.8012364276999445\n",
      "    values: 0.6558980179075362\n",
      "    values: 0.7181995110708936\n",
      "    values: 0.3641305677019203\n",
      "    values: 0.7840834698032608\n",
      "    values: 0.9058391273676959\n",
      "    values: 0.4516001908093663\n",
      "    values: 0.5304158448862025\n",
      "    values: 0.20830121945193392\n",
      "    values: 0.9601768023873427\n",
      "    values: 0.06799421139787454\n",
      "    values: 0.11622663744394257\n",
      "    values: 0.9045586055596698\n",
      "    values: 0.10852056034001101\n",
      "    values: 0.8694195845472978\n",
      "    values: 0.9776416495667736\n",
      "    values: 0.8558229502725364\n",
      "    values: 0.5341681238390731\n",
      "    values: 0.09874696251551529\n",
      "    values: 0.6343782148705341\n",
      "    values: 0.335114755873694\n",
      "    values: 0.1305430885554436\n",
      "    values: 0.8492040652789808\n",
      "    values: 0.1701503896567088\n",
      "    values: 0.15680389176856535\n",
      "    values: 0.1991950989828961\n",
      "    values: 0.3747047187234781\n",
      "    values: 0.9816925569399363\n",
      "    values: 0.6096191756234463\n",
      "    values: 0.11380443111132399\n",
      "    values: 0.15176821682694364\n",
      "    values: 0.05644263909258129\n",
      "    values: 0.4765321217920664\n",
      "    values: 0.9133306281146076\n",
      "    values: 0.5632762255375104\n",
      "    values: 0.5800418175453365\n",
      "    values: 0.6049378275327681\n",
      "    values: 0.22825835509291437\n",
      "    values: 0.046531011273127754\n",
      "    values: 0.1736963573484428\n",
      "    values: 0.5897932884097313\n",
      "    values: 0.004467496088400247\n",
      "    values: 0.12584151771154506\n",
      "    values: 0.8147475329153084\n",
      "    values: 0.26967047106782416\n",
      "    values: 0.26643060370653393\n",
      "    values: 0.16037783352677115\n",
      "    values: 0.027294973200703287\n",
      "    values: 0.7828263814181949\n",
      "    values: 0.5041939958470232\n",
      "    values: 0.8703505242074319\n",
      "    values: 0.7314176757977626\n",
      "    values: 0.16862976179565936\n",
      "    values: 0.8584652256072811\n",
      "    values: 0.11059267581785792\n",
      "    values: 0.8289975840920387\n",
      "    values: 0.45418756679306505\n",
      "    values: 0.6132425511692112\n",
      "    values: 0.41485645864515\n",
      "    values: 0.38401907558516424\n",
      "    values: 0.5648781730726499\n",
      "    values: 0.7915849067954073\n",
      "    values: 0.5513714666149135\n",
      "    values: 0.047056010365690626\n",
      "    values: 0.39048702214423003\n",
      "    values: 0.6355526853008111\n",
      "    values: 0.4359561573387918\n",
      "    values: 0.7615873007039502\n",
      "    values: 0.7248711987328281\n",
      "    values: 0.783055822326521\n",
      "    values: 0.9957303712121968\n",
      "    values: 0.09281938962620495\n",
      "    values: 0.08805449529879206\n",
      "    values: 0.4557852959479257\n",
      "    values: 0.7247322312573724\n",
      "    values: 0.9532804137593837\n",
      "    values: 0.7692170560263132\n",
      "    values: 0.18941321440286318\n",
      "    values: 0.6940220423480187\n",
      "    values: 0.9781679976963153\n",
      "    values: 0.5870985758893236\n",
      "    values: 0.731642328655836\n",
      "    values: 0.5894719453835865\n",
      "    values: 0.08699917412636005\n",
      "    values: 0.581857817482733\n",
      "    values: 0.9147171429342716\n",
      "    values: 0.4797746031530664\n",
      "    values: 0.8689022071616811\n",
      "    values: 0.46825416333328773\n",
      "    values: 0.8289819019365491\n",
      "    values: 0.28968040852984533\n",
      "    values: 0.218568373387935\n",
      "    values: 0.6911493493949187\n",
      "    values: 0.9728376365495333\n",
      "    values: 0.22817043972553686\n",
      "    values: 0.3863885634895884\n",
      "    values: 0.7122114780842361\n",
      "    values: 0.611740796968631\n",
      "    values: 0.237275411844596\n",
      "    values: 0.9240485007898428\n",
      "    values: 0.6267171519370067\n",
      "    values: 0.635372272219464\n",
      "    values: 0.24803040450542257\n",
      "    values: 0.4353346387364768\n",
      "    values: 0.42188587958587764\n",
      "    values: 0.8351069592809962\n",
      "    values: 0.6049868292097409\n",
      "    values: 0.1470628754669573\n",
      "    values: 0.6702619133536928\n",
      "    values: 0.2096750058825414\n",
      "    values: 0.9576210434286268\n",
      "    values: 0.7481959284392122\n",
      "    values: 0.20900028010520932\n",
      "    values: 0.6503829908421809\n",
      "    values: 0.5462991701153971\n",
      "    values: 0.5761247763891115\n",
      "    values: 0.14434007240190783\n",
      "    values: 0.44075100643293197\n",
      "    values: 0.966138515199629\n",
      "    values: 0.3346745112894808\n",
      "    values: 0.6511868943710382\n",
      "    values: 0.05181583028048753\n",
      "    values: 0.10772455225645694\n",
      "    values: 0.27829391416625215\n",
      "    values: 0.0905815702218924\n",
      "    values: 0.8211860748671165\n",
      "    values: 0.9105095617738221\n",
      "    values: 0.21009885435870213\n",
      "    values: 0.856609307136091\n",
      "    values: 0.19507751048149247\n",
      "    values: 0.773615156926888\n",
      "    values: 0.4711651233628015\n",
      "    values: 0.9519461386503482\n",
      "    values: 0.5894471743539621\n",
      "    values: 0.022188787459025416\n",
      "    values: 0.40663918498607887\n",
      "    values: 0.004064045152280205\n",
      "    values: 0.22597789480785213\n",
      "    values: 0.8771582379802286\n",
      "    values: 0.9104715450781604\n",
      "    values: 0.013623226003872224\n",
      "    values: 0.14235022919479834\n",
      "    values: 0.8456081014755331\n",
      "    values: 0.8885153994941029\n",
      "    values: 0.5776509874660916\n",
      "    values: 0.2736613126050156\n",
      "    values: 0.3655433564536531\n",
      "    values: 0.4015398229543867\n",
      "    values: 0.2761572170028368\n",
      "    values: 0.7913861902884219\n",
      "    values: 0.5944774623051068\n",
      "    values: 0.03151391674427362\n",
      "    values: 0.005829068473039767\n",
      "    values: 0.8709565397527862\n",
      "    values: 0.7290744042604294\n",
      "    values: 0.6680957536200761\n",
      "    values: 0.5690784119075634\n",
      "    values: 0.2647762004237889\n",
      "    values: 0.7784223876553263\n",
      "    values: 0.08187248541622916\n",
      "    values: 0.797571575321594\n",
      "    values: 0.7536030478238546\n",
      "    values: 0.14529212998820362\n",
      "    values: 0.7931261988687779\n",
      "    values: 0.6178048664397441\n",
      "    values: 0.7833125181168248\n",
      "    values: 0.792690328425518\n",
      "    values: 0.5927000164194645\n",
      "    values: 0.24273062838067527\n",
      "    values: 0.1844499683680597\n",
      "    values: 0.03674306473100197\n",
      "    values: 0.03478164793206828\n",
      "    values: 0.7602237058764901\n",
      "    values: 0.9608047781837186\n",
      "    values: 0.30196256613724237\n",
      "    values: 0.28616163262023064\n",
      "    values: 0.29736033276015295\n",
      "    values: 0.6651712560297097\n",
      "    values: 0.25255609265084433\n",
      "    values: 0.20128561736538308\n",
      "    values: 0.9761488016358197\n",
      "    values: 0.6187401822398542\n",
      "    values: 0.46804866414650914\n",
      "    values: 0.8669243746307722\n",
      "    values: 0.7897931109651862\n",
      "    values: 0.6325538898644762\n",
      "    values: 0.677947075877616\n",
      "    values: 0.537883791457655\n",
      "    values: 0.5175986481625033\n",
      "    values: 0.8634039141044878\n",
      "    values: 0.38900830444895484\n",
      "    values: 0.8858396052150391\n",
      "    values: 0.716476265233863\n",
      "    values: 0.7276365206371389\n",
      "    values: 0.17069585163641776\n",
      "    values: 0.7739478359437112\n",
      "    values: 0.9207805513709922\n",
      "    values: 0.40771601128995294\n",
      "    values: 0.6777070241626794\n",
      "    values: 0.24149436465197816\n",
      "    values: 0.4228491139683288\n",
      "    values: 0.8540574771850953\n",
      "    values: 0.5045763490622074\n",
      "    values: 0.48168348517413473\n",
      "    values: 0.4340711724937265\n",
      "    values: 0.3129331018987971\n",
      "    values: 0.05709794161101733\n",
      "    values: 0.3576215391817783\n",
      "    values: 0.22150391682720172\n",
      "    values: 0.011434074497420177\n",
      "    values: 0.5450134275069075\n",
      "    values: 0.21994362224860442\n",
      "    values: 0.477362619863818\n",
      "    values: 0.83520653441227\n",
      "    values: 0.5358069580526581\n",
      "    values: 0.14233668291112955\n",
      "    values: 0.697630034974003\n",
      "    values: 0.6333265735603678\n",
      "    values: 0.736519338005322\n",
      "    values: 0.6846333575536285\n",
      "    values: 0.8323514235213699\n",
      "    values: 0.7737730800103659\n",
      "    values: 0.05203578123849295\n",
      "    values: 0.04306448732527868\n",
      "    values: 0.6662453970919266\n",
      "    values: 0.47830003093062545\n",
      "    values: 0.8006723623996856\n",
      "    values: 0.6464139547160375\n",
      "    values: 0.015011240865117514\n",
      "    values: 0.5533747462094801\n",
      "    values: 0.01361143103017326\n",
      "    values: 0.9135879691584722\n",
      "    values: 0.2854778600755622\n",
      "    values: 0.4179546126258271\n",
      "    values: 0.6430369708264954\n",
      "    values: 0.7877141301378312\n",
      "    values: 0.3112732703413481\n",
      "    values: 0.47999939022196825\n",
      "    values: 0.23703944869892002\n",
      "    values: 0.9335512291734372\n",
      "    values: 0.06730240262953258\n",
      "    values: 0.4478074241711638\n",
      "    values: 0.6112309913259187\n",
      "    values: 0.18416960642337765\n",
      "    values: 0.20400663493444193\n",
      "    values: 0.8919598232097541\n",
      "    values: 0.7372928758177953\n",
      "    values: 0.3789863247496752\n",
      "    values: 0.14973191963079335\n",
      "    values: 0.020161182784826925\n",
      "    values: 0.6011334046541121\n",
      "    values: 0.6478686999291475\n",
      "    values: 0.7850773420527063\n",
      "    values: 0.15299838530129184\n",
      "    values: 0.46343733442267865\n",
      "    values: 0.43216194459850543\n",
      "    values: 0.650827354949895\n",
      "    values: 0.010471925901262447\n",
      "    values: 0.00829619467866205\n",
      "    values: 0.9015497801997425\n",
      "    values: 0.15835237878370656\n",
      "    values: 0.282654386709041\n",
      "    values: 0.9797611350338223\n",
      "    values: 0.7256249629651907\n",
      "    values: 0.180205718483601\n",
      "    values: 0.21601950140370318\n",
      "    values: 0.47579336856298504\n",
      "    values: 0.22488191732237262\n",
      "    values: 0.16886400640607646\n",
      "    values: 0.7086179335501667\n",
      "    values: 0.6440967682166335\n",
      "    values: 0.3056690060411009\n",
      "    values: 0.7680391048044203\n",
      "    values: 0.5330680373195846\n",
      "    values: 0.8614866669820314\n",
      "    values: 0.035413114135230184\n",
      "    values: 0.545156014305518\n",
      "    values: 0.005900927947358414\n",
      "    values: 0.062481471741751005\n",
      "    values: 0.024373349802955135\n",
      "    values: 0.610037250185934\n",
      "    values: 0.5798040218238029\n",
      "    values: 0.3792770541186674\n",
      "    values: 0.9098386802396983\n",
      "    values: 0.9591664448440516\n",
      "    values: 0.03372366661064785\n",
      "    values: 0.04058385026918421\n",
      "    values: 0.04432878178973432\n",
      "    values: 0.45347304390447896\n",
      "    values: 0.5853319846309261\n",
      "    values: 0.17413605830954682\n",
      "    values: 0.5646085079395161\n",
      "    values: 0.6243237228009376\n",
      "    values: 0.2542843955334789\n",
      "    values: 0.49813588567004186\n",
      "    values: 0.9471977616272863\n",
      "    values: 0.2621008363384292\n",
      "    values: 0.45907144387925014\n",
      "    values: 0.2988413195292512\n",
      "    values: 0.77560430938308\n",
      "    values: 0.10330624997689619\n",
      "    values: 0.09949487582512861\n",
      "    values: 0.5119108093059547\n",
      "    values: 0.4712916480321412\n",
      "    values: 0.19522103483848263\n",
      "    values: 0.6956090233255406\n",
      "    values: 0.6889996537698064\n",
      "    values: 0.31977118993788745\n",
      "    values: 0.8988748145214507\n",
      "    values: 0.7036342050658023\n",
      "    values: 0.38176050829928865\n",
      "    values: 0.5491319770274395\n",
      "    values: 0.9282932317219543\n",
      "    values: 0.09611650010246109\n",
      "    values: 0.3768268175851639\n",
      "    values: 0.26775896058715853\n",
      "    values: 0.9280805716106704\n",
      "  }\n",
      "}\n",
      "\n",
      "Response:\n",
      "{'data': {'names': ['t:0', 't:1', 't:2', 't:3', 't:4', 't:5', 't:6', 't:7', 't:8', 't:9'], 'tensor': {'shape': [1, 10], 'values': [5.58847308e-25, 5.20967247e-30, 0.977413237, 0.00434717909, 4.43884675e-31, 0.018239649, 1.88422183e-15, 3.35216821e-21, 4.55872851e-09, 2.19507326e-26]}}, 'meta': {'requestPath': {'mnist-model': 'seldonio/tfserving-proxy:1.15.0-dev'}}}\n"
     ]
    }
   ],
   "source": [
    "r = sc.predict(gateway=\"istio\", transport=\"rest\", shape=(1, 784))\n",
    "print(r)\n",
    "assert r.success == True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### gRPC Request"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Success:True message:\n",
      "Request:\n",
      "{'meta': {}, 'data': {'tensor': {'shape': [1, 784], 'values': [0.9998375676089037, 0.6392211092205818, 0.7251277357504603, 0.7808896762945906, 0.9471029273755714, 0.43660688687977034, 0.17739519588149044, 0.2685829334433132, 0.5628708378672913, 0.8612175076530088, 0.2708916996211985, 0.0406158034626205, 0.09456029072837158, 0.2666814341367846, 0.7194599080791999, 0.9452695941148388, 0.5563611274094761, 0.3768816657869556, 0.20350304379689876, 0.8736840386635644, 0.17739603777375101, 0.5882026251989727, 0.1232152208633025, 0.7771061539607746, 0.34367025360840164, 0.2362496171749492, 0.3482959505637404, 0.47976274163712895, 0.6246524750977825, 0.9140435253156701, 0.7937910929365936, 0.38934636623427854, 0.26806514316035146, 0.3893990993199904, 0.49757057620468503, 0.012091057798297222, 0.4044151699262156, 0.7251585100654504, 0.4694628238213354, 0.8651811034421732, 0.5053952381547168, 0.85361227673482, 0.7366099133244872, 0.5180127600188984, 0.2179237298515888, 0.5433329794411055, 0.36930041031018446, 0.7788858414998587, 0.753340821805769, 0.17967511068886155, 0.6145448596582288, 0.16559553195077004, 0.2138529536286481, 0.8427983198788503, 0.8740981689914816, 0.3856242143771802, 0.18371104079163436, 0.4297583987445547, 0.10108105456864513, 0.8431707724572862, 0.6897498937252297, 0.660057146734252, 0.15669945259739615, 0.1285372377408216, 0.30683402720336717, 0.8645433035446762, 0.3695200366707969, 0.9278766778163701, 0.6990669274840023, 0.7738199643415615, 0.3726689316988163, 0.23131303425074046, 0.29464273540103847, 0.44851033406137186, 0.24958522328243227, 0.959718055349001, 0.6957063714598016, 0.47525806718409525, 0.12857671945875881, 0.3012517503948541, 0.07071461194365503, 0.05003219029386197, 0.6661959702260833, 0.8453636967639719, 0.8603493030476621, 0.6154781618016356, 0.17384973466974363, 0.3330026479577167, 0.5580056440950443, 0.6814454446446438, 0.8384207232138043, 0.5550674384177283, 0.1306261038178782, 0.014680840775067261, 0.9181509901035543, 0.7033332205450622, 0.6783174721908556, 0.8806353690204538, 0.46029385882380913, 0.07483606883369387, 0.5738492535113954, 0.09191873181615284, 0.1710354687450769, 0.6074442385836999, 0.6571894909573045, 0.6854921272320373, 0.6471512128723828, 0.6762134519420487, 0.5845542018630464, 0.3960811995818979, 0.7411669850023669, 0.14150103034111605, 0.32866166930243335, 0.2780107810210197, 0.7725567434416791, 0.07129138136851854, 0.45608604142725095, 0.31055509203263054, 0.7940703045116781, 0.8517122049017283, 0.342493184165072, 0.48289569137863053, 0.21321618799616993, 0.8858219313623279, 0.7550908808488739, 0.6943050355825677, 0.16483431391636327, 0.5005854167904734, 0.17238549464611208, 0.2086867478375054, 0.032389489319256004, 0.4347725717199533, 0.8579552070201638, 0.9138605227755245, 0.9758296202010837, 0.6949019761407101, 0.4191053853879734, 0.6659145992525622, 0.7677118645909855, 0.24958121704129033, 0.6508205846666189, 0.4588842664875551, 0.48570748901216254, 0.5937441698499041, 0.37694161792558833, 0.15365476652232424, 0.19167117069921025, 0.3299556262282495, 0.34042673478380925, 0.6577336000051281, 0.5096298336030443, 0.46270681873487773, 0.8632809483728608, 0.9738102075628994, 0.7131382043255359, 0.004079977617932995, 0.2592848651100146, 0.3856602560297604, 0.44087021298398654, 0.5029461657856321, 0.6932019040318891, 0.9675994276804958, 0.7713457068366164, 0.8435094958410188, 0.5289839384467337, 0.735685589615721, 0.7313252352223532, 0.03375290270087983, 0.13840557426864697, 0.4048120155605476, 0.5237633180807841, 0.4355969450907252, 0.14038250002033237, 0.7506403840369514, 0.1450190092792456, 0.4586781275651982, 0.3144058117495878, 0.9673728733287045, 0.22517307893487848, 0.4954757261933921, 0.6244490177357518, 0.2123639772826914, 0.13691828069809142, 0.05650911157497307, 0.2604179633596353, 0.9630038857348265, 0.6812390943133333, 0.34768997997096884, 0.45001420622795685, 0.35174595039916123, 0.9674828830932802, 0.6618377544206178, 0.8477760964345799, 0.9617900873523807, 0.32510636309116403, 0.7416648307261213, 0.9363631285339015, 0.17183020291607165, 0.8475589844039726, 0.224568557892719, 0.2457437399491681, 0.8664006256247765, 0.030028773891941474, 0.4186173023951326, 0.8287920058419038, 0.35973534988896105, 0.6511869173482786, 0.5294770563321308, 0.946412094954597, 0.812482519607099, 0.3672463596036494, 0.7344588610488945, 0.6031783940641463, 0.8656483415839102, 0.43986158402625497, 0.7524420613123411, 0.3965201441779188, 0.1905475008538451, 0.5484707264245137, 0.344203480816045, 0.009900629480145029, 0.6574972440758317, 0.8880546065754985, 0.5949771642765174, 0.7127709047357214, 0.5284806488131697, 0.5571052704197698, 0.19187958117242443, 0.4683002823027661, 0.6953404553051732, 0.15192594776818835, 0.7777480923836222, 0.42668291669449665, 0.8150150912279405, 0.8899240052085794, 0.5838594298538653, 0.4858467641689149, 0.9504536263296325, 0.7862540020148352, 0.642194058242723, 0.41352856151515105, 0.5718923120409027, 0.9927176688561632, 0.6713704439386358, 0.8693793588025164, 0.3134918523498196, 0.4238560309713939, 0.38162953165727087, 0.9358323934496319, 0.9737019888123128, 0.16779542061578134, 0.0463070982913788, 0.013981221400875543, 0.041562368647210635, 0.5591620147767499, 0.40043046533741156, 0.7755514656547978, 0.3737324456822547, 0.6825338279528202, 0.617150828370086, 0.1266471517416432, 0.9138195833364484, 0.3678759592244092, 0.8360064889897666, 0.2723009404655078, 0.16622266035376776, 0.43127338027664575, 0.7376211347031769, 0.1900736849268393, 0.43900843177376225, 0.9955014056647603, 0.89303635195075, 0.1534149135527212, 0.4964767951383465, 0.27829489574883537, 0.5743451500907372, 0.21762350501087535, 0.735034366330862, 0.7251166837090426, 0.664593388340304, 0.2977994030312553, 0.30144048229985254, 0.0845084634448161, 0.995635134851757, 0.9166142444823936, 0.9666806958938474, 0.2626902192330284, 0.6275168705386093, 0.891071449656104, 0.8415206090744147, 0.3818789160170304, 0.3282057734741426, 0.3142930695744207, 0.6971741142393718, 0.2682372295463441, 0.9568228986657156, 0.6186442422980434, 0.1718785257727482, 0.8727239529030081, 0.5478065584520785, 0.05652517300758886, 0.8904410672591415, 0.4384942279260724, 0.6442895976038356, 0.8280932269945421, 0.2834429511658507, 0.9158215603197403, 0.5093282160232573, 0.9080653112016185, 0.43085992170968335, 0.18785259830611656, 0.21612925850903497, 0.9729607119899871, 0.8442927183974572, 0.3132315717225326, 0.9889675720598395, 0.6184690004492538, 0.3234607696590871, 0.7970305933960604, 0.12163738705845795, 0.27102079436280324, 0.29043139927810013, 0.34154193556948464, 0.9806166693093233, 0.7463226794249775, 0.5762567819770816, 0.7799340916363083, 0.53976229047184, 0.5738236203113702, 0.9367811391146843, 0.8882973232082787, 0.9742516762714406, 0.9110159596495111, 0.8603623269075282, 0.3064912817990616, 0.23035242752436524, 0.8057051712575578, 0.40669937967368286, 0.1175110680990441, 0.895500747583005, 0.47009446133562816, 0.9054624459609492, 0.08568836430344529, 0.36626960973579004, 0.5602441915520207, 0.8345101081150171, 0.2228628702962202, 0.3460672112839138, 0.20484292661251136, 0.7467344266988611, 0.5011203578214714, 0.5810559001698333, 0.20042394435456556, 0.2550275459853818, 0.437495834078109, 0.3502670336901408, 0.041331175423582334, 0.7836090848737568, 0.4536330537680159, 0.8188232148429642, 0.3370303529455143, 0.7785899343260836, 0.19675607459979372, 0.5426950762010787, 0.2528998545361786, 0.16864662743535574, 0.07114743764279452, 0.6297021279907552, 0.4239616485533697, 0.30465152398668904, 0.11706584629217565, 0.5613236555392489, 0.561323683670498, 0.2326361313694839, 0.5177396320634163, 0.027835507647529623, 0.3051264320526832, 0.8133355480813557, 0.3580327461764722, 0.4782002780210751, 0.5531330439794514, 0.07914750923295255, 0.5797511130585181, 0.9780696759363786, 0.4863163290020599, 0.4123924698624992, 0.16341325654766192, 0.8917521176620512, 0.8481672612078209, 0.8510981110258017, 0.6564156911261676, 0.564831569959665, 0.8726089325964326, 0.6054518957772484, 0.33597831632445363, 0.35337976356322764, 0.688666794871606, 0.23229962095034895, 0.01484368424479543, 0.4039330793886836, 0.7620534654907409, 0.7169656247854092, 0.548496778834871, 0.3531376017083697, 0.8372522530315838, 0.16247854008417306, 0.40471070910417906, 0.4713888721819638, 0.50951687984169, 0.005333860172462623, 0.6139368993548, 0.5723880896453907, 0.8203885280808363, 0.9193782763206231, 0.772572731410142, 0.5350875924160577, 0.2169185306676037, 0.7818504303350879, 0.6619668563013515, 0.5010961170464322, 0.9657361044475519, 0.029756158670924404, 0.7118587804143762, 0.6302176548849213, 0.5144317764973361, 0.8770089050729013, 0.3357013501902979, 0.35367888873910247, 0.05685006908136636, 0.7420282208200184, 0.5134947040029337, 0.8088826121851742, 0.7639872128642832, 0.12390905073865277, 0.04477475955817767, 0.09364716546752361, 0.372505193919328, 0.7603293685077399, 0.08595211420401605, 0.4089167806327132, 0.2739296937941953, 0.3376259236656729, 0.7967616577087252, 0.23498374601487249, 0.46464819933710766, 0.025945258645318603, 0.284626148444878, 0.7226973370160598, 0.01078023305142306, 0.7666812711560178, 0.5349351960129669, 0.1366648316277128, 0.583999750890187, 0.5774993135897312, 0.7422669600417889, 0.061960152289858184, 0.16333076645775246, 0.0025607209562688027, 0.1951944756741445, 0.6270780823941542, 0.756598651149106, 0.9463960734870384, 0.1657086255376472, 0.8531205083602247, 0.044470829620854135, 0.44754220794858524, 0.7701071014661427, 0.0457811715296752, 0.42638562698722693, 0.27622638518246356, 0.09291471258383588, 0.17890955180023904, 0.4206643721887854, 0.524775511887901, 0.7741425753716078, 0.7814135275537942, 0.7630329593880127, 0.35488034355062537, 0.8194076372776392, 0.6261053193177466, 0.9322812731052698, 0.38371924165132265, 0.2721585234471142, 0.6202344965450154, 0.45398076243834473, 0.8972713225118831, 0.6226872324870506, 0.8705286133573261, 0.9215667189446192, 0.015975433864905186, 0.8714011142134623, 0.6019236078513434, 0.47305557927535624, 0.8481667281315558, 0.3627853794167184, 0.16738472338694899, 0.2814372246525456, 0.4248133011818911, 0.7209725379930139, 0.36049106201516623, 0.8197904407579293, 0.3086107736953896, 0.5891578909647137, 0.8995064496345951, 0.6169467403451937, 0.4044262165234429, 0.8529128104601208, 0.27216230697862387, 0.7263052076140986, 0.7403277918863618, 0.09990368551573803, 0.6511632187066719, 0.9316498214719467, 0.5844478532774479, 0.3240933121684608, 0.05030333741117288, 0.35350034837621014, 0.27227411803326007, 0.5943511364230658, 0.1908244644820457, 0.3052151163831046, 0.780693165747541, 0.45900511964755353, 0.11498890271082429, 0.41051625805273817, 0.385909450189989, 0.11612405915294033, 0.3073784471156583, 0.470253738736087, 0.8791433799603302, 0.27250041875429143, 0.5778628621584133, 0.05774796441444541, 0.19472818245878565, 0.9489467518256041, 0.3874301958766735, 0.5617160963972391, 0.6698918563056502, 0.5037553594619721, 0.8438871660586753, 0.911949614674525, 0.02709585834012629, 0.5051737243183662, 0.994165499192289, 0.20048488690273958, 0.9764821724826116, 0.26805133785379887, 0.8274579636299555, 0.11367551270044307, 0.7729264430047417, 0.6310395873267434, 0.9183742662793112, 0.509632685758568, 0.7921841076637581, 0.46311966693271167, 0.9218382204050641, 0.20080432990625308, 0.9937035382564232, 0.1031189097240055, 0.5058315208668248, 0.13340271825902128, 0.8865742263140193, 0.9889472584165465, 0.6804105676966486, 0.5137657820034426, 0.47123439290699654, 0.9345051895461288, 0.6515179662341368, 0.142084250144058, 0.9972266959163512, 0.0005949651844611159, 0.4179654887357557, 0.3473454066586362, 0.38787060170209986, 0.9366163800009827, 0.43007250933274155, 0.5079131347069591, 0.8189313546965682, 0.6417670249285709, 0.35004745446706753, 0.9420940548921395, 0.5796928186596902, 0.4917695139816023, 0.8554028051017076, 0.44641168508442153, 0.34520503188088203, 0.04504782803523821, 0.869074691034748, 0.46701231219883366, 0.18853528208362247, 0.9188101820250285, 0.13410264026802565, 0.34782581320456807, 0.9171882231045678, 0.5404900935897672, 0.40639607589630355, 0.37176125120164105, 0.8811032356997041, 0.48961219673353495, 0.5703825590798491, 0.657051398088682, 0.2675593162561105, 0.47652403250939546, 0.19139710470166793, 0.2605409612771883, 0.9074323211503212, 0.6887342277718735, 0.6940299152415016, 0.7829857174682875, 0.9064770218462095, 0.7210205246063583, 0.16216145157038797, 0.42486714990458374, 0.798330355432116, 0.9810676267528498, 0.20517514442598328, 0.2558821923676863, 0.8718143213018922, 0.7565757320937324, 0.10555612230287892, 0.5858090594039411, 0.935801600565141, 0.3417964696304636, 0.6226225195551526, 0.46370840313476636, 0.8937991787826053, 0.9780237638935213, 0.7598797634782815, 0.717244702133823, 0.30417432501366426, 0.9393724811606409, 0.6965584229458298, 0.041939794330914104, 0.25187692436921816, 0.416103774213179, 0.9047922009722897, 0.3217013185063887, 0.9901514968215416, 0.9597286917912559, 0.8867989649309458, 0.17699409686257805, 0.024799902490826864, 0.1210076275647205, 0.33817832485501254, 0.36152118155211377, 0.367181400164885, 0.793385079285185, 0.7069148551378989, 0.2904038047458095, 0.079943624013315, 0.16193746530304465, 0.5858733699793525, 0.21764965940738423, 0.9536071992169927, 0.8922159545100463, 0.9834246555811866, 0.018815471892799973, 0.5733699672377677, 0.8714546322697075, 0.6935395071777894, 0.7364544515942502, 0.33131631416218, 0.33145781065527335, 0.4684982014089213, 0.2919329141203564, 0.26101952469081136, 0.2171404518492559, 0.8348086023345749, 0.36270992745041253, 0.6005610261505202, 0.4917410057264371, 0.8148488472442061, 0.9001992401748655, 0.988137158626888, 0.6746429855154066, 0.7069863195180465, 0.0674843892016308, 0.15945105132366055, 0.29830145678205466, 0.6927594156540278, 0.8180672226024285, 0.17308553913232916, 0.9902223621206543, 0.6311861573513877, 0.809690453155234, 0.19192151150992587, 0.18313896190908896, 0.3491543220323795, 0.8254077258226903, 0.9657781337821518, 0.20615961082996803, 0.9899958248765538, 0.8284342100461449, 0.547190740464929, 0.2207811881322581, 0.11291703946465215, 0.29584039066680046, 0.5977371825958736, 0.26434466610073715, 0.15318730727751417, 0.8213628659685003, 0.14930861800812245, 0.39731154998719187, 0.826988152354807, 0.5512198216933386, 0.5440904782941621, 0.48272277164354893, 0.498103775855522, 0.8167376034713866, 0.8605540491540176, 0.9879344543042438, 0.37120859209886437, 0.46537008799759594, 0.9607482672066561, 0.28408748386684746, 0.01370358425114393, 0.9093291604893644, 0.9088707244220318, 0.15267555651701814, 0.5004009522039123, 0.16405911928878225, 0.935093374884947, 0.8094753690907022, 0.6474322516076126, 0.9573794226853352, 0.4026707645689923, 0.5736671866707472, 0.3227778643194148, 0.7789796552741124, 0.5370655611738934, 0.6823021904863539, 0.38692565116892097, 0.8295285173984724, 0.12687844028321082, 0.8129543065350571, 0.9952188380752344, 0.8003417296853862, 0.17019790697596127, 0.5426809411833465, 0.6305504923342433, 0.9723047789348951, 0.5407712933705637, 0.2576613132432922, 0.9236250744400989, 0.20481507765889329, 0.10346661640755417, 0.5156276104682014, 0.06103884933631909, 0.18907227940270432, 0.9136622226739916, 0.9932151006522282, 0.7266653088947709, 0.7300929317396078, 0.4668502288298021, 0.3102837784659479, 0.2659834785299512, 0.13026071841660603, 0.9254075293257326, 0.08529586460698046, 0.5525575307115453, 0.21343858683748318, 0.18060289706324217, 0.8683506637020498, 0.6848930695805433, 0.6755154127436478, 0.2061481738644, 0.4298575389580429, 0.34168578962829554, 0.8454335802901116, 0.9033778297545824, 0.3401244429147189, 0.08965274313171268, 0.1569139124055624, 0.646742011344807, 0.5871233685539055, 0.36616461039778603, 0.6383094108189803, 0.8555840857328585, 0.8645675155149569, 0.6607827923299152, 0.6761344905641318, 0.3017867901904404, 0.4517706542862252, 0.269490128193246]}}}\n",
      "Response:\n",
      "{'data': {'tftensor': {'dtype': 'DT_FLOAT', 'tensorShape': {'dim': [{'size': '1'}, {'size': '10'}]}, 'floatVal': [8.493433e-22, 2.851194e-35, 0.123584226, 0.066573136, 1.1826565e-28, 0.80983657, 4.1654608e-13, 1.4864153e-19, 6.0619104e-06, 2.4017428e-20]}}}\n"
     ]
    }
   ],
   "source": [
    "r = sc.predict(gateway=\"istio\", transport=\"grpc\", shape=(1, 784))\n",
    "print(r)\n",
    "assert r.success == True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And delete the model we deployed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io \"tfserving\" deleted\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl delete -f ./resources/mnist_rest.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Serve Tensorflow Model with Tensorflow protocol\n",
    "\n",
    "The config file below shows how you can deploy your Tensorflow model which\n",
    "exposes the Tensorflow protocol."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting ./resources/halfplustwo_rest.yaml\n"
     ]
    }
   ],
   "source": [
    "%%writefile ./resources/halfplustwo_rest.yaml\n",
    "apiVersion: machinelearning.seldon.io/v1alpha2\n",
    "kind: SeldonDeployment\n",
    "metadata:\n",
    "  name: hpt\n",
    "spec:\n",
    "  name: hpt\n",
    "  protocol: tensorflow\n",
    "  transport: rest\n",
    "  predictors:\n",
    "  - graph:\n",
    "      children: []\n",
    "      implementation: TENSORFLOW_SERVER\n",
    "      modelUri: gs://seldon-models/tfserving/half_plus_two\n",
    "      name:  halfplustwo\n",
    "      parameters:\n",
    "        - name: model_name\n",
    "          type: STRING\n",
    "          value: halfplustwo\n",
    "    name: default\n",
    "    replicas: 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io/hpt created\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl apply -f ./resources/halfplustwo_rest.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting for deployment \"hpt-default-0-halfplustwo\" rollout to finish: 0 of 1 updated replicas are available...\n",
      "deployment \"hpt-default-0-halfplustwo\" successfully rolled out\n"
     ]
    }
   ],
   "source": [
    "!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=hpt -o jsonpath='{.items[0].metadata.name}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'predictions': [2.5, 3.0, 4.5]}\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "X=!curl -s -d '{\"instances\": [1.0, 2.0, 5.0]}' \\\n",
    "   -X POST http://localhost:8003/seldon/seldon/hpt/v1/models/halfplustwo/:predict \\\n",
    "   -H \"Content-Type: application/json\"\n",
    "d=json.loads(\"\".join(X))\n",
    "print(d)\n",
    "assert(d[\"predictions\"][0] == 2.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'outputs': {'x': {'dtype': 'DT_FLOAT', 'tensorShape': {'dim': [{'size': '3'}]}, 'floatVal': [2.5, 3, 3.5]}}, 'modelSpec': {'name': 'halfplustwo', 'version': '123', 'signatureName': 'serving_default'}}\n"
     ]
    }
   ],
   "source": [
    "X=!cd ../executor/proto && grpcurl \\\n",
    "   -d '{\"model_spec\":{\"name\":\"halfplustwo\"},\"inputs\":{\"x\":{\"dtype\": 1, \"tensor_shape\": {\"dim\":[{\"size\": 3}]}, \"floatVal\" : [1.0, 2.0, 3.0]}}}' \\\n",
    "   -rpc-header seldon:hpt -rpc-header namespace:seldon \\\n",
    "   -plaintext -proto ./prediction_service.proto \\\n",
    "   0.0.0.0:8003 tensorflow.serving.PredictionService/Predict\n",
    "d=json.loads(\"\".join(X))\n",
    "print(d)\n",
    "assert(d[\"outputs\"][\"x\"][\"floatVal\"][0] == 2.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io \"hpt\" deleted\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl delete -f ./resources/halfplustwo_rest.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Serve MLFlow Elasticnet Wines Model\n",
    "We can deploy an MLFlow model uploaded to an object store by using the MLFlow\n",
    "model server implementation as the config below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing ./resources/elasticnet_wine.yaml\n"
     ]
    }
   ],
   "source": [
    "%%writefile ./resources/elasticnet_wine.yaml\n",
    "apiVersion: machinelearning.seldon.io/v1alpha2\n",
    "kind: SeldonDeployment\n",
    "metadata:\n",
    "  name: mlflow\n",
    "spec:\n",
    "  name: wines\n",
    "  predictors:\n",
    "  - componentSpecs:\n",
    "    - spec:\n",
    "        # We are setting high failureThreshold as installing conda dependencies\n",
    "        # can take long time and we want to avoid k8s killing the container prematurely\n",
    "        containers:\n",
    "        - name: classifier\n",
    "          livenessProbe:\n",
    "            initialDelaySeconds: 80\n",
    "            failureThreshold: 200\n",
    "            periodSeconds: 5\n",
    "            successThreshold: 1\n",
    "            httpGet:\n",
    "              path: /health/ping\n",
    "              port: http\n",
    "              scheme: HTTP\n",
    "          readinessProbe:\n",
    "            initialDelaySeconds: 80\n",
    "            failureThreshold: 200\n",
    "            periodSeconds: 5\n",
    "            successThreshold: 1\n",
    "            httpGet:\n",
    "              path: /health/ping\n",
    "              port: http\n",
    "              scheme: HTTP\n",
    "    graph:\n",
    "      children: []\n",
    "      implementation: MLFLOW_SERVER\n",
    "      modelUri: gs://seldon-models/v1.10.0-dev/mlflow/elasticnet_wine\n",
    "      name: classifier\n",
    "    name: default\n",
    "    replicas: 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io/mlflow created\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl apply -f ./resources/elasticnet_wine.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting for deployment \"mlflow-default-0-classifier\" rollout to finish: 0 of 1 updated replicas are available...\n",
      "deployment \"mlflow-default-0-classifier\" successfully rolled out\n"
     ]
    }
   ],
   "source": [
    "!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=mlflow -o jsonpath='{.items[0].metadata.name}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### REST requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'data': {'names': [], 'ndarray': [5.275558760255375]}, 'meta': {'requestPath': {'classifier': 'seldonio/mlflowserver:1.15.0-dev'}}}\n"
     ]
    }
   ],
   "source": [
    "X=!curl -s -d '{\"data\": {\"ndarray\":[[0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1.0,1.1]]}}' \\\n",
    "   -X POST http://localhost:8003/seldon/seldon/mlflow/api/v1.0/predictions \\\n",
    "   -H \"Content-Type: application/json\"\n",
    "d=json.loads(X[0])\n",
    "print(d)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "from seldon_core.seldon_client import SeldonClient\n",
    "\n",
    "sc = SeldonClient(deployment_name=\"mlflow\", namespace=\"seldon\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Success:True message:\n",
      "Request:\n",
      "meta {\n",
      "}\n",
      "data {\n",
      "  tensor {\n",
      "    shape: 1\n",
      "    shape: 11\n",
      "    values: 0.08557947016760259\n",
      "    values: 0.3904340887602685\n",
      "    values: 0.013650233499940545\n",
      "    values: 0.667873429847926\n",
      "    values: 0.8854441380301948\n",
      "    values: 0.2158356850044546\n",
      "    values: 0.39582510842260477\n",
      "    values: 0.6090407713228351\n",
      "    values: 0.7908824041187265\n",
      "    values: 0.6806393014966566\n",
      "    values: 0.5804883985108746\n",
      "  }\n",
      "}\n",
      "\n",
      "Response:\n",
      "{'data': {'names': [], 'tensor': {'shape': [1], 'values': [5.2237671948422015]}}, 'meta': {'requestPath': {'classifier': 'seldonio/mlflowserver:1.15.0-dev'}}}\n"
     ]
    }
   ],
   "source": [
    "r = sc.predict(gateway=\"istio\", transport=\"rest\", shape=(1, 11))\n",
    "print(r)\n",
    "assert r.success == True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### gRPC Requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'meta': {'requestPath': {'classifier': 'seldonio/mlflowserver:1.15.0-dev'}}, 'data': {'ndarray': [5.275558760255375]}}\n"
     ]
    }
   ],
   "source": [
    "X=!cd ../executor/proto && grpcurl -d '{\"data\":{\"ndarray\":[[0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1.0,1.1]]}}' \\\n",
    "         -rpc-header seldon:mlflow -rpc-header namespace:seldon \\\n",
    "         -plaintext \\\n",
    "         -proto ./prediction.proto  0.0.0.0:8003 seldon.protos.Seldon/Predict\n",
    "d=json.loads(\"\".join(X))\n",
    "print(d)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Success:True message:\n",
      "Request:\n",
      "{'meta': {}, 'data': {'tensor': {'shape': [1, 11], 'values': [0.1504501387663657, 0.09916272705865703, 0.48360619526082804, 0.10653978599045866, 0.28875817583971986, 0.2490316271245857, 0.7923655335917282, 0.2630601591638151, 0.33789481427553547, 0.30831566652953724, 0.6134317115838707]}}}\n",
      "Response:\n",
      "{'meta': {'requestPath': {'classifier': 'seldonio/mlflowserver:1.15.0-dev'}}, 'data': {'tensor': {'shape': [1], 'values': [5.225506841935588]}}}\n"
     ]
    }
   ],
   "source": [
    "r = sc.predict(gateway=\"istio\", transport=\"grpc\", shape=(1, 11))\n",
    "print(r)\n",
    "assert r.success == True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io \"mlflow\" deleted\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl delete -f ./resources/elasticnet_wine.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MLFlow V2 protocol"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting ./resources/elasticnet_wine_v2.yaml\n"
     ]
    }
   ],
   "source": [
    "%%writefile ./resources/elasticnet_wine_v2.yaml\n",
    "apiVersion: machinelearning.seldon.io/v1alpha2\n",
    "kind: SeldonDeployment\n",
    "metadata:\n",
    "  name: mlflow\n",
    "spec:\n",
    "  protocol: v2  # Activate v2 protocol\n",
    "  name: wines\n",
    "  predictors:\n",
    "    - graph:\n",
    "        children: []\n",
    "        implementation: MLFLOW_SERVER\n",
    "        modelUri: gs://seldon-models/v1.12.0-dev/mlflow/elasticnet_wine\n",
    "        name: classifier\n",
    "      name: default\n",
    "      replicas: 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io/mlflow created\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl apply -f ./resources/elasticnet_wine_v2.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting for deployment \"mlflow-default-0-classifier\" rollout to finish: 0 of 1 updated replicas are available...\n",
      "deployment \"mlflow-default-0-classifier\" successfully rolled out\n"
     ]
    }
   ],
   "source": [
    "!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=mlflow -o jsonpath='{.items[0].metadata.name}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## REST requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"model_name\": \"classifier\",\n",
      "  \"model_version\": \"v1\",\n",
      "  \"id\": \"a43e833e-0e20-46b9-b0b2-c8b6917febd7\",\n",
      "  \"parameters\": null,\n",
      "  \"outputs\": [\n",
      "    {\n",
      "      \"name\": \"predict\",\n",
      "      \"shape\": [\n",
      "        1\n",
      "      ],\n",
      "      \"datatype\": \"FP64\",\n",
      "      \"parameters\": null,\n",
      "      \"data\": [\n",
      "        6.016145744177844\n",
      "      ]\n",
      "    }\n",
      "  ]\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "\n",
    "import requests\n",
    "\n",
    "inference_request = {\n",
    "    \"parameters\": {\"content_type\": \"pd\"},\n",
    "    \"inputs\": [\n",
    "        {\n",
    "            \"name\": \"fixed acidity\",\n",
    "            \"shape\": [1],\n",
    "            \"datatype\": \"FP32\",\n",
    "            \"data\": [7.4],\n",
    "            \"parameters\": {\"content_type\": \"np\"},\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"volatile acidity\",\n",
    "            \"shape\": [1],\n",
    "            \"datatype\": \"FP32\",\n",
    "            \"data\": [0.7000],\n",
    "            \"parameters\": {\"content_type\": \"np\"},\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"citric acidity\",\n",
    "            \"shape\": [1],\n",
    "            \"datatype\": \"FP32\",\n",
    "            \"data\": [0],\n",
    "            \"parameters\": {\"content_type\": \"np\"},\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"residual sugar\",\n",
    "            \"shape\": [1],\n",
    "            \"datatype\": \"FP32\",\n",
    "            \"data\": [1.9],\n",
    "            \"parameters\": {\"content_type\": \"np\"},\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"chlorides\",\n",
    "            \"shape\": [1],\n",
    "            \"datatype\": \"FP32\",\n",
    "            \"data\": [0.076],\n",
    "            \"parameters\": {\"content_type\": \"np\"},\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"free sulfur dioxide\",\n",
    "            \"shape\": [1],\n",
    "            \"datatype\": \"FP32\",\n",
    "            \"data\": [11],\n",
    "            \"parameters\": {\"content_type\": \"np\"},\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"total sulfur dioxide\",\n",
    "            \"shape\": [1],\n",
    "            \"datatype\": \"FP32\",\n",
    "            \"data\": [34],\n",
    "            \"parameters\": {\"content_type\": \"np\"},\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"density\",\n",
    "            \"shape\": [1],\n",
    "            \"datatype\": \"FP32\",\n",
    "            \"data\": [0.9978],\n",
    "            \"parameters\": {\"content_type\": \"np\"},\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"pH\",\n",
    "            \"shape\": [1],\n",
    "            \"datatype\": \"FP32\",\n",
    "            \"data\": [3.51],\n",
    "            \"parameters\": {\"content_type\": \"np\"},\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"sulphates\",\n",
    "            \"shape\": [1],\n",
    "            \"datatype\": \"FP32\",\n",
    "            \"data\": [0.56],\n",
    "            \"parameters\": {\"content_type\": \"np\"},\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"alcohol\",\n",
    "            \"shape\": [1],\n",
    "            \"datatype\": \"FP32\",\n",
    "            \"data\": [9.4],\n",
    "            \"parameters\": {\"content_type\": \"np\"},\n",
    "        },\n",
    "    ],\n",
    "}\n",
    "\n",
    "endpoint = \"http://localhost:8003/seldon/seldon/mlflow/v2/models/infer\"\n",
    "response = requests.post(endpoint, json=inference_request)\n",
    "\n",
    "print(json.dumps(response.json(), indent=2))\n",
    "assert response.ok"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io \"mlflow\" deleted\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl delete -f ./resources/elasticnet_wine_v2.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.12 ('base')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  },
  "vscode": {
   "interpreter": {
    "hash": "3c5b05c738269f3db37ab2b93400d252a3eceb4ebe547b00838e0efaaa2ac1dc"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}