feat(cli): add cleanup subcommand to remove run leftovers from the namespace

llmdbenchmark run tears down the harness launcher pod when a run
finishes, but leaves the data-access pod, the harness service, the
benchmark ConfigMaps and the workload PVC behind. The PVC keeps billing
against its storage backend, a stale claim wedges retries after a
wrong-StorageClass failure, and the only way out was a hand-written
kubectl delete naming CLI-internal resource names.

Add 'llmdbenchmark --spec <specification> cleanup [--keep-pvc]':

- takes --spec like every other subcommand and renders the scenario, so
  resource names come from the plan config rather than from constants:
  harness.podLabel, harness.name (the '*-profiles' ConfigMap),
  labels.app (the harness service) and storage.workloadPvc.name
- resolution follows the precedence the run steps use (executor/step.py):
  CLI flag > scenario > default
- cleans both namespaces: launcher pods and their ConfigMaps render into
  'harness.namespace | default(namespace.name)' while the PVC, service
  and data-access pod render into 'namespace.name', so a single
  namespace cannot reach everything
- also deletes the resources with no scenario key of their own (the
  data-access pod label and the llm-d-benchmark-* / harness-scripts
  ConfigMaps), which stay matched by name
- pods are deleted before the PVC so the pvc-protection finalizer can
  clear
- idempotent: missing resources are skipped and a namespace with no
  benchmark resources exits 0
- --keep-pvc preserves the workload PVC so workload data survives
  between runs; --namespace and --pvc-name override the scenario

Fixes: #1789
Signed-off-by: Himanshu Prajapati <himanshuprajapati15072003@gmail.com>

Author: himanshu1573

README.md

@@ -648,6 +648,7 @@ llmdbenchmark/                Python package
         plan.py               Plan subcommand
         standup.py            Standup subcommand
         teardown.py           Teardown subcommand
+        cleanup.py            Cleanup subcommand (remove run leftovers)
         run.py                Run subcommand
         experiment.py         Experiment subcommand (DoE orchestration)
 

docs/lifecycle.md

@@ -120,3 +120,13 @@ llmdbenchmark teardown
 
 > [!NOTE]
 > The scenario can also be indicated as part of the command line options for `llmdbenchmark teardown` (e.g., `llmdbenchmark teardown --spec kubernetes_H200_modelservice_llama-8b`)
+
+`teardown` removes the deployed llm-d stack, but the harness namespace still holds what benchmark runs created there: the data-access pod, the harness service, the benchmark ConfigMaps and the workload PVC (which keeps billing against its storage backend until deleted). Remove all of it with
+
+```
+llmdbenchmark --spec <specification> cleanup
+```
+
+`cleanup` takes the same `--spec` as every other subcommand and reads the resource names from the scenario, so a scenario that overrides `harness.podLabel`, `harness.name`, `labels.app` or `storage.workloadPvc.name` is cleaned correctly. It also reaches both namespaces when the scenario sets `harness.namespace` separately from `namespace.name`.
+
+The command is idempotent -- resources that no longer exist are skipped, and a namespace with no benchmark resources exits 0. Pass `--keep-pvc` to preserve the workload PVC so workload data survives for the next run. `--namespace` overrides the namespaces from the scenario (comma-separated `model,harness`), and `--pvc-name` overrides `storage.workloadPvc.name`.

llmdbenchmark/cli.py

@@ -34,6 +34,7 @@
 from llmdbenchmark.interface import smoketest as smoketest_interface
 from llmdbenchmark.interface import experiment as experiment_interface
 from llmdbenchmark.interface import results
+from llmdbenchmark.interface import cleanup as cleanup_interface
 from llmdbenchmark.parser.cli_overrides import (
     GLOBAL_SELECTOR,
     REDACTED,
@@ -93,6 +94,7 @@ def dispatch_cli(args: argparse.Namespace, logger: logging.Logger) -> None:
         Command.SMOKETEST.value,
         Command.TEARDOWN.value,
         Command.RUN.value,
+        Command.CLEANUP.value,
     ):
         # Resolve templates, scenarios, and values into the workspace
         try:
@@ -164,7 +166,10 @@ def dispatch_cli(args: argparse.Namespace, logger: logging.Logger) -> None:
         # This enables kustomize overlays and full manifest inspection.
         # Runs even in dry-run mode - helmfile template is purely local
         # and does not touch the cluster.
-        _render_helm_manifests(config.plan_dir, logger)
+        # Cleanup deletes by name and label and never reads the manifests,
+        # so it skips the helmfile shell-outs.
+        if args.command != Command.CLEANUP.value:
+            _render_helm_manifests(config.plan_dir, logger)
 
     if args.command == Command.STANDUP.value:
         _execute_standup(args, logger, render_plan_errors)
@@ -178,6 +183,9 @@ def dispatch_cli(args: argparse.Namespace, logger: logging.Logger) -> None:
     if args.command == Command.RUN.value:
         _execute_run(args, logger, render_plan_errors)
 
+    if args.command == Command.CLEANUP.value:
+        cleanup_interface.execute(args, logger)
+
 
 def _render_helm_manifests(plan_dir: Path, logger) -> None:
     """Pre-render modelservice Helm chart manifests into each stack's plan directory.
@@ -2039,6 +2047,7 @@ def cli() -> None:
     run.add_subcommands(subparsers, parents=[benchmark_parser])
     experiment_interface.add_subcommands(subparsers, parents=[benchmark_parser])
     results.add_subcommands(subparsers, parents=[])
+    cleanup_interface.add_subcommands(subparsers, parents=[benchmark_parser])
     args = parser.parse_args()
 
     # Merge env vars for boolean flags (store_true can't use default=)
@@ -2067,7 +2076,7 @@ def cli() -> None:
             "the following arguments are required: --specification_file/--spec"
         )
 
-    # Results command is handled separately
+    # Results reads the local store only -- no specification, no plans
     if args.command == Command.RESULTS.value:
         temp_dir = Path(tempfile.gettempdir()) / "llmdbenchmark" / "logs"
         temp_dir.mkdir(parents=True, exist_ok=True)