feat(sdk): support volume mount in tune API

[truc0]

What this PR does / why we need it:

As discussed on #2247 , providing a clean and simple way for specifying volume mount in tune API of katib Python SDK will enhance develop experience.

This PR adds storage_per_trial argument to KatibClient.tune() method.

Design

The storage_per_trial argument is designed to have the following type:

List[TypedDict({
    "volume": client.V1Volume,
    "mount_path": client.V1VolumeMount,
})]

To simplify the usage, there are some enhancements:

1. User can omit the outer list wrapper if there is only one storage to be mount. This optimization change the type of storage_per_trial to: Union[TuneStoragePerTrial, List[TuneStoragePerTrial]]
2. User can specify a dict instead of a client.V1Volume object when the storage config is simple:

   {
        "name": "volume-name",  # Required: Name of the volume
        "type": "pvc|secret|config_map|empty_dir",  # Required: Volume type
        # Optional fields based on type:
        # For PVC:
        "claim_name": "pvc-name",
        "read_only": False,
        # For Secret:
        "secret_name": "secret-name", 
        "items": [...],
        "default_mode": 0644,
        "optional": False,
        # For ConfigMap:
        "config_map_name": "config-name",
        "items": [...],
        "default_mode": 0644,
        "optional": False,
        # For EmptyDir:
        "medium": None,
        "size_limit": None
    }

3. User can specify mount_path with a str instead of client.V1VolumeMount

Example Usage

storage_per_trial = [
    {
        "volume": {
            "name": "data",
            "type": "pvc",
            "claim_name": "my-data-pvc"
        },
        "mount_path": "/data"
    },
    {
        "volume": {
            "name": "config",
            "type": "config_map",
            "config_map_name": "model-config"
        },
        "mount_path": "/etc/config"
    }
]

Full Example

import kubeflow.katib as katib
from kubernetes import client
import json
import base64

VOLUME_NAME = "katib-secret-volume"
SECRET_NAME = "katib-secret-test"

# create the secret (for testing purposes, optional)
secret = client.V1Secret(
    metadata=client.V1ObjectMeta(name=SECRET_NAME),
    data={
        "credentials.json": base64.b64encode(
            json.dumps({"username": "admin", "password": "password"}).encode()
        ).decode()
    },
)
# client.CoreV1Api().create_namespaced_secret(namespace="kubeflow", body=secret)


def objective(parameters):
    import os

    result = 4 * int(parameters["x"]) - float(parameters["y"]) ** 2
    # The result will be negative if the secret is mounted successfully
    if os.path.exists("/secret/credentials.json"):
        result *= -1
    print(f"result={result}")


parameters = {
    "x": katib.search.int(min=10, max=20),
    "y": katib.search.double(min=0.1, max=0.2),
}


katib_client = katib.KatibClient(namespace="kubeflow")

storage_per_trial = {
    "volume": {
        "name": VOLUME_NAME,
        "type": "secret",
        "secret_name": SECRET_NAME,
    },
    "mount_path": "/secret",
}

name = "katib-experiment-secret"
katib_client.tune(
    name=name,
    parameters=parameters,
    objective=objective,
    objective_metric_name="result",
    max_trial_count=12,
    resources_per_trial={"cpu": "2"},
    storage_per_trial=storage_per_trial,
)

katib_client.wait_for_experiment_condition(name=name)
print(katib_client.get_optimal_hyperparameters(name=name))

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #2247

Checklist:

• Docs included if any changes are user facing
• Support mounting volume
• Documenting usage of storage_per_trial

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign johnugeorge for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Electronic-Waste]

@truc0 Amazing! Thanks for doing this. And I'm sorry for the late reply. I'll review this PR in this week.

cc @kubeflow/wg-automl-leads @helenxie-bit @mahdikhashan

[mahdikhashan]

@truc0 thank you - would you please add unit tests and e2e for your changes -

unit tests in this path: sdk/python/v1beta1/kubeflow/katib/api/katib_client_test.py
e2e here: test/e2e/v1beta1/scripts/gh-actions/run-e2e-tune-api.py

I'll review the code and functionality after it. thanks for your time.

[truc0]

Sure, I will add it soon

[Electronic-Waste]

@truc0 I'm so sorry for the late response. Here are my initial comments for you.

Please also take a look at this PR if you are available @kubeflow/wg-automl-leads @helenxie-bit @mahdikhashan @Doris-xm

[Electronic-Waste]

@truc0 That's an amazing optimization from the perspective of Data Scientists. Could you also help with the review of kubeflow/trainer#2449 (comment), which might be simiar to this scenario?👀

[Electronic-Waste]

I'm thinking about the name convention here. We already have storage_config parameters here and have similar functionality. Do we have any ideas dealing with them? Or combine them together?

[Electronic-Waste]

Suggested change

	- type: One of "pvc", "secret", "config_map", or "empty_dir"
	- type: One of "pvc", "secret", "configmap", or "empty_dir"

Usually, we'll name it as configmap instead of config_map:)

[Electronic-Waste]

Same naming convention issue.

[Electronic-Waste]

Same naming convention issue.

[Electronic-Waste]

/ok-to-test
/rerun-all

[Electronic-Waste]

Could you please fix the pre-commit error? Now, you can rerun the CI test with /rerun-all on yourself.

[mahdikhashan]

@truc0 I'm so sorry for the late response. Here are my initial comments for you.

Please also take a look at this PR if you are available @kubeflow/wg-automl-leads @helenxie-bit @mahdikhashan @Doris-xm

thanks @Electronic-Waste , i already have requested some changes here: #2508 (comment)

cc: @truc0