[Feature] Multi-components workload scheduling

[RainbowMango]

What is multi-components workload
Kubernetes workloads like Deployment, StatefulSet, and Pod all consist of one component(one pod template) with one or more replicas.
AI training and Big-data workloads, usually consist of more than one component, each component might have multiple replicas.

What would you like to be added:
Provide fine-grained support for workloads which usually consist of two or more components, like:

• Kubeflow Training/Big-data workloads
  • The following APIs are being deprecated since July 2025. See the announcement here.
    • TFJob(Tensorflow Job), which may consist of multiple components like PS, Worker, Chief, Master, and Evaluator.
    • PyTorchJob which may consist of two components, like Master and Worker.
    • MXJob, which may consist of multiple components, like Scheduler, Server, Worker, TunerTracker, TunerServer, and Tuner.
    • XGBoostJob, which may consist of two components, like Master and Worker.
    • MPIJob, which may consist of two components, like Launcher and Worker.
    • PaddleJob, which may consist of two components, like Master and Worker.
  • SparkApplication, which may consist of two components, like driver and executor.
  • TrainJob, which may consist of two components, like RuntimeRef(refer to JobSet), and Trainer.
• FlinkDeployment, which may consist of two components, like jobManager and taskManager.
• Volcano Job, which may consist of multiple tasks.
• RayJob which contains HeadGroup and WorkerGroup.
• etc

User story
As a user, I want to deploy PyTorchJob Job to Karmada, and I hope Karmada schedules it to exactly one member cluster based on the available resources on the member cluster. (See #4049 )

Working Progress

• Proposal part 1: Proposal for multiple pod template support #5085
• Proposal part 2: Refine Multiple Pod Templates Scheduling proposal #6535
• Iteration in release-1.15: [Umbrella] Multi-components workload scheduling - phase I #6641
• Iteration in release-1.16: [Umbrella] Multi-components workload scheduling - phase ​​II​​ #6734

Why is this needed:

1. For the FederatedResourceQuota feature, it requires accurate resource usage. (see [Feature] FederatedResourceQuota Enhancement - Phase II #6486 (comment))

References

• CRDs(might be needed when perform the tests)
  • Volcano Job

[Monokaix]

Replica division is a key issue for multi pod template workload, is there any resolution here?

[KunWuLuan]

Some other job types:

RayJob

TrainJob

[RainbowMango]

Thanks for the report. I will look at them and get back to you.

[RainbowMango]

For the RayJob, I can see the workers are declared with the WokerGroupSpecs, is there a master? (I don't know if you usually say master.)

[RainbowMango]

And For the TrainJob, I can see only one Trainner for each TrainJob, so it seems not a multi-compoments workload. Please correct me if I'm wrong. @KunWuLuan

[KunWuLuan]

For the RayJob, I can see the workers are declared with the WokerGroupSpecs, is there a master? (I don't know if you usually say master.)

For a RayJob, there will be some head pods and some worker groups. Worker pods communicate with head pods to run tasks.

[KunWuLuan]

And For the TrainJob, I can see only one Trainner for each TrainJob, so it seems not a multi-compoments workload. Please correct me if I'm wrong. @KunWuLuan

TrainJob will refer to a TrainingRuntime, which will refer to a JobSet.

[RainbowMango]

For a RayJob, there will be some head pods and some worker groups. Worker pods communicate with head pods to run tasks.

Got it, thanks. Given worker group could have a different pod template, so each worker group can be treated as a component. Right?

[KunWuLuan]

For a RayJob, there will be some head pods and some worker groups. Worker pods communicate with head pods to run tasks.

Got it, thanks. Given worker group could have a different pod template, so each worker group can be treated as a component. Right?

Yes, each worker group can be treated as a component.

[RainbowMango]

OK. Just added both of them to the issue description.