Add helpers to build custom MLServer images (#371)

Author: Adrian Gonzalez-Martin

.gitignore

@@ -18,6 +18,7 @@ bin
 *.swp
 *.swo
 *~
+.env
 
 # Python dev
 .mypy_cache

Dockerfile

@@ -2,11 +2,12 @@ FROM python:3.8-slim AS wheel-builder
 SHELL ["/bin/bash", "-c"]
 
 COPY ./hack/build-wheels.sh ./hack/build-wheels.sh
-
-COPY setup.py .
-COPY README.md .
-COPY ./mlserver/ ./mlserver/
-COPY ./runtimes/ ./runtimes/
+COPY ./mlserver ./mlserver
+COPY ./runtimes ./runtimes
+COPY \
+    setup.py \
+    README.md \
+    .
 
 # This will build the wheels and place will place them in the
 # /opt/mlserver/dist folder
@@ -53,11 +54,15 @@ COPY requirements/docker.txt requirements/docker.txt
 RUN pip install -r requirements/docker.txt
 
 COPY ./licenses/license.txt .
-
-COPY ./hack/activate-env.sh ./hack/activate-env.sh
+COPY \
+    ./hack/build-env.sh \
+    ./hack/generate_dotenv.py \
+    ./hack/activate-env.sh \
+    ./hack/
 
 USER 1000
 
-# Need to source `activate-env.sh` so that env changes get persisted
-CMD . ./hack/activate-env.sh $MLSERVER_ENV_TARBALL \
-    && mlserver start $MLSERVER_MODELS_DIR
+# We need to build and activate the "hot-loaded" environment before MLServer
+# starts
+CMD source ./hack/activate-env.sh $MLSERVER_ENV_TARBALL . && \
+    mlserver start $MLSERVER_MODELS_DIR

README.md

@@ -64,6 +64,8 @@ Out of the box, MLServer comes with a set of pre-packaged runtimes which let
 you interact with a subset of common frameworks.
 This allows you to start serving models saved in these frameworks straight
 away.
+However, it's also possible to **write [custom
+runtimes](./docs/runtimes/custom)**.
 
 Out of the box, MLServer provides support for:
 

docs/conf.py

@@ -20,11 +20,11 @@
 
 project = "MLServer"
 copyright = "2021, Seldon Technologies"
-html_title = "MLServer"
+html_title = "MLServer Documentation"
 author = "Seldon Technologies"
 
 # The full version, including alpha/beta/rc tags
-release = "0.6.0.dev2"
+release = "0.6.0.dev3"
 
 
 # -- General configuration ---------------------------------------------------
@@ -37,6 +37,7 @@
     "sphinx_search.extension",
     "sphinxcontrib.bibtex",
     "sphinxcontrib.autodoc_pydantic",
+    "sphinx_click",
 ]
 
 # Add any paths that contain templates here, relative to this directory.
@@ -122,6 +123,7 @@
     html_theme_path = sphinx_material.html_theme_path()
     html_context = sphinx_material.get_html_context()
 
+html_favicon = "favicon.ico"
 html_sidebars = {
     "**": ["logo-text.html", "globaltoc.html", "localtoc.html", "searchbox.html"]
 }

docs/examples/custom/README.ipynb

@@ -200,8 +200,7 @@ The next step will be to create 2 configuration files:
     "name": "numpyro-divorce",
     "implementation": "models.NumpyroModel",
     "parameters": {
-        "uri": "./numpyro-divorce.json",
-        "version": "v0.1.0",
+        "uri": "./numpyro-divorce.json"
     }
 }
 ```
@@ -240,13 +239,104 @@ inference_request = {
     ]
 }
 
-endpoint = "http://localhost:8080/v2/models/numpyro-divorce/versions/v0.1.0/infer"
+endpoint = "http://localhost:8080/v2/models/numpyro-divorce/infer"
 response = requests.post(endpoint, json=inference_request)
 
 response.json()
 ```
 
+## Deployment
+
+Now that we have written and tested our custom model, the next step is to deploy it.
+With that goal in mind, the rough outline of steps will be to first build a custom image containing our code, and then deploy it.
+
+
+### Building a custom image
+
+```{note}
+This section expects that Docker is available and running in the background. 
+```
+
+MLServer offers helpers to build a custom Docker image containing your code.
+In this example, we will use the `mlserver build` subcommand to create an image, which we'll be able to deploy later.
+
+
+Note that this section expects that Docker is available and running in the background, as well as a functional cluster with Seldon Core installed and some familiarity with `kubectl`. 
+
+
+```bash
+%%bash
+mlserver build . -t 'my-custom-numpyro-server'
+```
+
+To ensure that the image is fully functional, we can spin up a container and then send a test request. To start the container, you can run something along the following lines in a separate terminal:
+
+```bash
+docker run -it --rm -p 8080:8080 my-custom-numpyro-server
+```
+
 
 ```python
+import requests
 
+x_0 = [28.0]
+inference_request = {
+    "inputs": [
+        {
+          "name": "marriage",
+          "shape": [1],
+          "datatype": "FP32",
+          "data": x_0
+        }
+    ]
+}
+
+endpoint = "http://localhost:8080/v2/models/numpyro-divorce/infer"
+response = requests.post(endpoint, json=inference_request)
+
+response.json()
+```
+
+As we should be able to see, the server running within our Docker image responds as expected.
+
+### Deploying our custom image
+
+```{note}
+This section expects access to a functional Kubernetes cluster with Seldon Core installed and some familiarity with `kubectl`. 
+```
+
+Now that we've built a custom image and verified that it works as expected, we can move to the next step and deploy it.
+There is a large number of tools out there to deploy images.
+However, for our example, we will focus on deploying it to a cluster running [Seldon Core](https://docs.seldon.io/projects/seldon-core/en/latest/).
+
+For that, we will need to create a `SeldonDeployment` resource which instructs Seldon Core to deploy a model embedded within our custom image and compliant with the [V2 Inference Protocol](https://github.com/kserve/kserve/tree/master/docs/predict-api/v2).
+This can be achieved by _applying_ (i.e. `kubectl apply`) a `SeldonDeployment` manifest to the cluster, similar to the one below:
+
+
+```python
+%%writefile seldondeployment.yaml
+apiVersion: machinelearning.seldon.io/v1
+kind: SeldonDeployment
+metadata:
+  name: numpyro-model
+spec:
+  predictors:
+    - name: default
+      annotations:
+        seldon.io/no-engine: "true"
+      graph:
+        name: mlflow-model
+        type: MODEL
+      componentSpecs:
+        - spec:
+            containers:
+              - name: mlflow-model
+                image: my-custom-numpyro-server
+                ports:
+                  - containerPort: 8080
+                    name: http
+                    protocol: TCP
+                  - containerPort: 8081
+                    name: grpc
+                    protocol: TCP
 ```

docs/examples/custom/README.md

@@ -2,7 +2,6 @@
     "name": "numpyro-divorce",
     "implementation": "models.NumpyroModel",
     "parameters": {
-        "uri": "./numpyro-divorce.json",
-        "version": "v0.1.0",
+        "uri": "./numpyro-divorce.json"
     }
 }

docs/examples/custom/model-settings.json

@@ -1 +1,3 @@
 numpy==1.18.5
+numpyro==0.8.0
+jax==0.2.24

docs/examples/custom/requirements.txt

@@ -0,0 +1,24 @@
+apiVersion: machinelearning.seldon.io/v1
+kind: SeldonDeployment
+metadata:
+  name: numpyro-model
+spec:
+  predictors:
+    - name: default
+      annotations:
+        seldon.io/no-engine: "true"
+      graph:
+        name: mlflow-model
+        type: MODEL
+      componentSpecs:
+        - spec:
+            containers:
+              - name: mlflow-model
+                image: my-custom-numpyro-server
+                ports:
+                  - containerPort: 8080
+                    name: http
+                    protocol: TCP
+                  - containerPort: 8081
+                    name: grpc
+                    protocol: TCP