MemVerge MMCloud Agent (#1128)

* Add MMCloud plugin to Integrations Signed-off-by: Edwin Yu <[email protected]> * Remove default image from README and example Signed-off-by: Edwin Yu <[email protected]> * Fix end of requirements.in Signed-off-by: Edwin Yu <[email protected]> * update requirement.txt for mmcloud_plugin Signed-off-by: helenzhangyc <[email protected]> --------- Signed-off-by: Edwin Yu <[email protected]> Signed-off-by: helenzhangyc <[email protected]> Co-authored-by: helenzhangyc <[email protected]>
flyteorg · Oct 2, 2023 · fe4968d · fe4968d
1 parent 0ca1523
commit fe4968d
Show file tree

Hide file tree

Showing 8 changed files with 510 additions and 0 deletions.
diff --git a/docs/index.md b/docs/index.md
@@ -448,6 +448,7 @@ auto_examples/greatexpectations_plugin/index
 auto_examples/hive_plugin/index
 auto_examples/k8s_pod_plugin/index
 auto_examples/mlflow_plugin/index
+auto_examples/mmcloud_plugin/index
 auto_examples/modin_plugin/index
 auto_examples/kfmpi_plugin/index
 auto_examples/onnx_plugin/index

diff --git a/docs/integrations.md b/docs/integrations.md
@@ -104,6 +104,8 @@ the Flyte task that use the respective plugin.
   - Running tasks and workflows on AWS batch service
 * - {doc}`Hive <auto_examples/hive_plugin/index>`
   - Run Hive jobs in your workflows.
+* - {doc}`MMCloud <auto_examples/mmcloud_plugin/index>`
+  - Execute tasks using MemVerge Memory Machine Cloud
 * - {doc}`Snowflake <auto_examples/snowflake_plugin/index>`
   - Run Snowflake jobs in your workflows.
 * - {doc}`Databricks <auto_examples/databricks_plugin/index>`

diff --git a/examples/mmcloud_plugin/Dockerfile b/examples/mmcloud_plugin/Dockerfile
@@ -0,0 +1,27 @@
+FROM python:3.11-slim-bookworm
+LABEL org.opencontainers.image.source https://github.com/flyteorg/flytesnacks
+
+WORKDIR /root
+ENV VENV /opt/venv
+ENV LANG C.UTF-8
+ENV LC_ALL C.UTF-8
+ENV PYTHONPATH /root
+
+WORKDIR /root
+
+ENV VENV /opt/venv
+# Virtual environment
+RUN python3 -m venv ${VENV}
+ENV PATH="${VENV}/bin:$PATH"
+
+# Install Python dependencies
+COPY requirements.txt /root
+RUN pip install -r /root/requirements.txt
+
+# Copy the actual code
+COPY . /root
+
+# This tag is supplied by the build script and will be used to determine the version
+# when registering tasks, workflows, and launch plans
+ARG tag
+ENV FLYTE_INTERNAL_IMAGE $tag
diff --git a/examples/mmcloud_plugin/README.md b/examples/mmcloud_plugin/README.md
@@ -0,0 +1,80 @@
+# Memory Machine Cloud
+
+```{eval-rst}
+.. tags:: AWS, GCP, AliCloud, Integration, Advanced
+```
+
+[MemVerge](https://memverge.com/) [Memory Machine Cloud](https://www.mmcloud.io/) (MMCloud)—available on AWS, GCP, and AliCloud—empowers users to continuously optimize cloud resources during runtime, safely execute stateful tasks on spot instances, and monitor resource usage in real time. These capabilities make it an excellent fit for long-running batch workloads.
+
+Flyte can be integrated with MMCloud, allowing you to execute Flyte tasks using MMCloud.
+
+## Installation
+
+To install the plugin, run the following command:
+
+```{eval-rst}
+.. prompt:: bash
+
+    pip install flytekitplugins-mmcloud
+```
+
+To get started with MMCloud, refer to the [MMCloud User Guide](https://docs.memverge.com/mmce/current/userguide/olh/index.html).
+
+## Configuring the backend to get MMCloud working
+
+The MMCloud plugin is [enabled in FlytePropeller's configuration](https://docs.flyte.org/en/latest/deployment/plugins/memverge/mmcloud.html).
+
+## Getting Started
+
+This plugin allows executing `PythonFunctionTask` using MMCloud without changing any function code.
+
+```{eval-rst}
+.. testcode:: awsbatch-quickstart
+    from flytekitplugins.mmcloud import MMCloudConfig
+
+    @task(task_config=MMCloudConfig())
+    def to_str(i: int) -> str:
+        return str(i)
+```
+
+[Resource](https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/productionizing/customizing_resources.html) (cpu and mem) requests and limits, [container](https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/customizing_dependencies/multi_images.html) images, and [environment](https://docs.flyte.org/projects/flytekit/en/latest/generated/flytekit.task.html) variable specifications are supported.
+
+[ImageSpec](https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/customizing_dependencies/image_spec.html) may be used to define images to run tasks.
+
+### Credentials
+
+The following [secrets](https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/productionizing/use_secrets.html) are required to be defined for the agent server:
+* `mmc_address`: MMCloud OpCenter address
+* `mmc_username`: MMCloud OpCenter username
+* `mmc_password`: MMCloud OpCenter password
+
+### Defaults
+
+Compute resources:
+* If only requests are specified, there are no limits.
+* If only limits are specified, the requests are equal to the limits.
+* If neither resource requests nor limits are specified, the default requests used for job submission are `cpu="1"` and `mem="1Gi"`, and there are no limits.
+
+### Agent Image
+
+Install `flytekitplugins-mmcloud` in the agent image.
+
+A `float` binary (obtainable via the OpCenter) is required. Copy it to the agent image `PATH`.
+
+Sample `Dockerfile` for building an agent image:
+```dockerfile
+FROM python:3.11-slim-bookworm
+
+WORKDIR /root
+ENV PYTHONPATH /root
+
+# flytekit will autoload the agent if package is installed.
+RUN pip install flytekitplugins-mmcloud
+COPY float /usr/local/bin/float
+
+CMD pyflyte serve --port 8000
+```
+
+```{auto-examples-toc}
+example
+```
diff --git a/examples/mmcloud_plugin/mmcloud_plugin/__init__.py b/examples/mmcloud_plugin/mmcloud_plugin/__init__.py
diff --git a/examples/mmcloud_plugin/mmcloud_plugin/example.py b/examples/mmcloud_plugin/mmcloud_plugin/example.py
@@ -0,0 +1,42 @@
+# %% [markdown]
+# # Memory Machine Cloud
+#
+# This example shows how to use the MMCloud plugin to execute tasks on MemVerge Memory Machine Cloud.
+
+# %%
+from flytekit import Resources, task, workflow
+from flytekitplugins.mmcloud import MMCloudConfig
+
+# %% [markdown]
+# `MMCloudConfig` configures `MMCloudTask`. Tasks specified with `MMCloudConfig` will be executed using MMCloud. Tasks will be executed with requests `cpu="1"` and `mem="1Gi"` by default.
+
+# %%
+@task(task_config=MMCloudConfig())
+def to_str(i: int) -> str:
+    return str(i)
+
+
+@task(task_config=MMCloudConfig())
+def to_int(s: str) -> int:
+    return int(s)
+
+
+# %% [markdown]
+# [Resource](https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/productionizing/customizing_resources.html) (cpu and mem) requests and limits, [container](https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/customizing_dependencies/multi_images.html) images, and [environment](https://docs.flyte.org/projects/flytekit/en/latest/generated/flytekit.task.html) variable specifications are supported.
+
+# %%
+@task(
+    task_config=MMCloudConfig(submit_extra="--migratePolicy [enable=true]"),
+    requests=Resources(cpu="1", mem="1Gi"),
+    limits=Resources(cpu="2", mem="4Gi"),
+    environment={"KEY": "value"},
+)
+def concatenate_str(s1: str, s2: str) -> str:
+    return s1 + s2
+
+
+@workflow
+def concatenate_int_wf(i1: int, i2: int) -> int:
+    i1_str = to_str(i=i1)
+    i2_str = to_str(i=i2)
+    return to_int(s=concatenate_str(s1=i1_str, s2=i2_str))
diff --git a/examples/mmcloud_plugin/requirements.in b/examples/mmcloud_plugin/requirements.in
@@ -0,0 +1,2 @@
+flytekitplugins-envd
+flytekitplugins-mmcloud