Skip to content

Pin FMA+KEDA workloads to a target node for benchmarking - #1883

Draft
dumb0002 wants to merge 1 commit into
mainfrom
fma-keda-beanchmarking
Draft

Pin FMA+KEDA workloads to a target node for benchmarking#1883
dumb0002 wants to merge 1 commit into
mainfrom
fma-keda-beanchmarking

Conversation

@dumb0002

@dumb0002 dumb0002 commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Confines an FMA+KEDA benchmark to a single node — including a tainted, admin-reserved one — so load generation and other tenants can't perturb the measurements.

Node name, label, and tolerations are set in one place: the new top-level nodePinning block in config/templates/values/defaults.yaml. Filling it in places the FMA launchers, the FMA requester, and EPP on that node; no per-scenario edits. The block is inert while enabled: false.

nodePinning:
  enabled: true
  nodeName: <node>
  nodeLabel: <key>
  nodeLabelValue: <value>
  tolerations:
    - key: <taint-key>
      operator: Equal
      value: <taint-value>
      effect: NoSchedule

FMA launchers and requester

  • nodeName pins to a named node instead of auto-selecting. Required for a tainted node: the auto-scan ranks on free GPU/CPU and does not consider taints, so it can pick a node whose pods never schedule. A named node is validated (exists, carries the selection label, has free GPUs) but never labeled or otherwise mutated — it is assumed prepared by whoever owns the cluster. A node with taints but no fma.tolerations warns rather than failing silently.
  • nodeLabelValue makes the selection label value configurable; previously hardcoded to 'true'.
  • fma.tolerations is applied to both the launcher podTemplate (launcher pods are created by the FMA controller from it) and the requester Deployment.
  • Teardown no longer strips the selection label when nodeName is set, which would otherwise break subsequent runs against an admin-prepared node.

EPP

_apply_epp_node_pinning() in render_plans.py injects EPP's placement from the same block. The router chart exposes affinity and tolerations for EPP but no nodeSelector, so node selection becomes a nodeAffinity term there while the FMA requester uses nodeSelector — different syntax, same effect. Adding router.epp.nodeSelector would be silently ignored, which is why this is done in the renderer rather than left to scenario authors.

@dumb0002 dumb0002 added the do-not-merge Indicates that a PR should not merge label Sep 3, 2026
Adds explicit node targeting for the FMA launcher/requester workloads so a
benchmark can be confined to one node, including a tainted one:

Signed-off-by: dumb0002 <Braulio.Dumba@ibm.com>
@dumb0002
dumb0002 force-pushed the fma-keda-beanchmarking branch from 4a6d822 to 3b59c45 Compare September 3, 2026 18:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

do-not-merge Indicates that a PR should not merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant