From 549c6b32dbaea4af0fbc3f3d0a4e00f1d80215fe Mon Sep 17 00:00:00 2001 From: Kevin Hannon Date: Fri, 17 Jan 2025 10:51:41 -0500 Subject: [PATCH 1/6] annual report for wg-batch 2024 --- wg-batch/annual-report-2024.md | 129 ++++++++++++++++++++++++++++++--- 1 file changed, 117 insertions(+), 12 deletions(-) diff --git a/wg-batch/annual-report-2024.md b/wg-batch/annual-report-2024.md index 662c87b33ce..4624354e346 100644 --- a/wg-batch/annual-report-2024.md +++ b/wg-batch/annual-report-2024.md @@ -2,27 +2,132 @@ ## Current initiatives and Project Health - 1. What work did the WG do this year that should be highlighted? - +See [2024 Highlights](#2024-highlights). 2. Are there any areas and/or subprojects that your group needs help with (e.g. fewer than 2 active OWNERS)? + None. + +### 2024 Highlights + +We will breakdown our highlights into Sub Projects, KEPs, talks, community adoption. + +#### Sub Projects + +##### Kueue + +Kueue has had 5 releases in 2024. + +- [Release 0.6](https://github.com/kubernetes-sigs/kueue/releases/tag/v0.6.0) + +- [Release 0.7](https://github.com/kubernetes-sigs/kueue/releases/tag/v0.7.0) + +- [Release 0.8](https://github.com/kubernetes-sigs/kueue/releases/tag/v0.8.0) + +- [Release 0.9](https://github.com/kubernetes-sigs/kueue/releases/tag/v0.9.0) + +- [Release 0.10](https://github.com/kubernetes-sigs/kueue/releases/tag/v0.10.0) + +In 2024, the kueue community would like to highlight are Topology aware scheduling, MultiKueue, Kueue Dashboard, KueueCtrl, Deployment/Statefulset integration for serving and Fair sharing. + +Topology aware scheduling facilitates scheduling of workloads that take in account data center topology. Workloads benefit from using interconnects that are physically close together. + +MultiKueue provides a way of dispatching batch workloads to worker clusters. Kueue provides multicluster dispatching for popular batch workloads such as Ray, Job, Kubeflow and JobSet. This feature went beta in 0.9. + +Kueue Dashboards has been a popular ask for Kueue. Users would like to have a visualization representation of queueing and we are happy to announce that a dashboard has been created for Kueue. This went into kueue in late 2024 and a big focus of 2025 will be to harden this for production. + +KueueCtrl provides a cli for creating kueue objects. The plugin is hosted in krew and is easily installed as a kueue plugin. + +Deployment/StatefulSet integration provides an avenue for the usage of Kueue for serving workloads. Serving leads to a need for sharing/preemption of model servers that may leverage accelerators. Kueue provides an integration with popular methods of deploying services (Deployment/StatefulSet). + +##### JobSet + +Jobset has had 4 release in 2024. + +- [Release 0.4](https://github.com/kubernetes-sigs/jobset/releases/tag/v0.4.0) + +- [Release 0.5](https://github.com/kubernetes-sigs/jobset/releases/tag/v0.5.0) + +- [Release 0.6](https://github.com/kubernetes-sigs/jobset/releases/tag/v0.6.0) + +- [Release 0.7](https://github.com/kubernetes-sigs/jobset/releases/tag/v0.7.0) + +A major achievement of JobSet has been the adoption of JobSet as a component for Kubeflow Training Operator V2. +There has been a collaborative effort with the Kubeflow community and the batch community to implement the features needed for this integration. + +[Metaflow](https://github.com/Netflix/metaflow/pull/1804) has adopted the use of JobSet for distributed ML training. + +##### KJob + +[KJob](https://github.com/kubernetes-sigs/kjob?tab=readme-ov-file#kjob) has been started to provide a CLI friendly way for users to submit batch jobs. +The HPC/ML community tend to prefer CLI over YAML so the focus was to provide a templated solution for submitting batch jobs. +Another focus of this project is to provide a smooth transition for Slurm users. + +#### KEPs + +WG-Batch provided a series of kubernetes enhancements that improved the experience of batch workloads on Kubernetes. In 2024, this group proposed/implemented the following KEPs. + +- [Job Managed By](https://github.com/kubernetes/enhancements/issues/4368) + - Promoted to beta in 2024 + +- [Job Success Policy](https://github.com/kubernetes/enhancements/issues/3998) + - Promoted to beta. + +- [Elastic Index Jobs](https://github.com/kubernetes/enhancements/issues/3715) + - Promoted to stable. + +- [Pod Failure Policy](https://github.com/kubernetes/enhancements/issues/3329) + - Promoted to stable. + +- [Pod Index Label](https://github.com/kubernetes/enhancements/issues/4017) + - Promoted to stable. + +### Talks + +- WG-Batch Update at Kubecon NA 2024 + - Authors: Kevin Hannon and Marcin Wielgus + +- Keynote: MultiCluster Batch Jobs Dispatching with Kueue at CERN + - Authors: Ricardo Rocha and Marcin Wielgus + - Kubecon NA 2024 + +- Multitenancy and Fairness at Scale with Kueue: A Case Study + - Authors: Aldo Culquicondor & Rajat Phull + - Kubecon NA 2024 + +- Advanced Resource Management for Running AI/ML Workloads with Kueue + - Authors: Michał Woźniak & Yuki Iwai + - Kubecon EU 2024 + +- Scale Your Batch / Big Data / AI Workloads Beyond the Kubernetes Scheduler + - Authors: Antonin Stefanutti & Anish Asthana + - KubeCon EU, March, Paris + +- WG-Batch Update at Kubecon EU 2024 + - Authors: Martin Wielgus + +- How the Kubernetes Community is Improving Kubernetes for HPC/AI/ML Workloads + - Authors: Kevin Hannon + - FOSDEM 2024 + +### Community adoption + +- [Kubeflow Training Operator v2](https://github.com/kubeflow/training-operator/blob/0c30f5cd306611f061b6dd529d3c7b7981a7d27c/docs/proposals/2170-kubeflow-training-v2/README.md#kep-2170-kubeflow-training-v2-api) will be using JobSet as a critical component for training and finetuning. + +- [Metaflow supports JobSet](https://github.com/Netflix/metaflow/pull/1804) for distributed training. + +- Airflow has built an [integration](https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/_api/airflow/providers/cncf/kubernetes/operators/kueue/index.html) with Kueue. + ## Operational Operational tasks in [wg-governance.md]: -- [ ] [README.md] reviewed for accuracy and updated if needed -- [ ] WG leaders in [sigs.yaml] are accurate and active, and updated if needed -- [ ] Meeting notes and recordings for 2024 are linked from [README.md] and updated/uploaded if needed -- [ ] Updates provided to sponsoring SIGs in 2024 +- [x] [README.md] reviewed for accuracy and updated if needed +- [x] WG leaders in [sigs.yaml] are accurate and active, and updated if needed +- [x] Meeting notes and recordings for 2024 are linked from [README.md] and updated/uploaded if needed +- [] Updates provided to sponsoring SIGs in 2024 - [$sig-name](https://git.k8s.io/community/$sig-id/) - links to email, meeting notes, slides, or recordings, etc - [$sig-name](https://git.k8s.io/community/$sig-id/) From 8350db306636ac215f7a773c85e423a8a3af111d Mon Sep 17 00:00:00 2001 From: Kevin Hannon Date: Fri, 17 Jan 2025 13:47:09 -0500 Subject: [PATCH 2/6] address feedback from mimowo --- wg-batch/annual-report-2024.md | 35 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 18 deletions(-) diff --git a/wg-batch/annual-report-2024.md b/wg-batch/annual-report-2024.md index 4624354e346..e42b99ea70f 100644 --- a/wg-batch/annual-report-2024.md +++ b/wg-batch/annual-report-2024.md @@ -70,7 +70,7 @@ Another focus of this project is to provide a smooth transition for Slurm users. WG-Batch provided a series of kubernetes enhancements that improved the experience of batch workloads on Kubernetes. In 2024, this group proposed/implemented the following KEPs. - [Job Managed By](https://github.com/kubernetes/enhancements/issues/4368) - - Promoted to beta in 2024 + - Promoted to beta. - [Job Success Policy](https://github.com/kubernetes/enhancements/issues/3998) - Promoted to beta. @@ -86,30 +86,32 @@ WG-Batch provided a series of kubernetes enhancements that improved the experien ### Talks -- WG-Batch Update at Kubecon NA 2024 +- WG-Batch Update at Kubecon - Authors: Kevin Hannon and Marcin Wielgus + - Kubecon NA, Salt Lake City - Keynote: MultiCluster Batch Jobs Dispatching with Kueue at CERN - Authors: Ricardo Rocha and Marcin Wielgus - - Kubecon NA 2024 + - Kubecon NA, Salt Lake City - Multitenancy and Fairness at Scale with Kueue: A Case Study - - Authors: Aldo Culquicondor & Rajat Phull - - Kubecon NA 2024 + - Authors: Aldo Culquicondor and Rajat Phull + - Kubecon NA, Salt Lake City - Advanced Resource Management for Running AI/ML Workloads with Kueue - - Authors: Michał Woźniak & Yuki Iwai - - Kubecon EU 2024 + - Authors: Michał Woźniak and Yuki Iwai + - Kubecon EU, Paris - Scale Your Batch / Big Data / AI Workloads Beyond the Kubernetes Scheduler - - Authors: Antonin Stefanutti & Anish Asthana - - KubeCon EU, March, Paris + - Authors: Antonin Stefanutti and Anish Asthana + - KubeCon EU, Paris -- WG-Batch Update at Kubecon EU 2024 - - Authors: Martin Wielgus +- WG-Batch Update + - Author: Marcin Wielgus + - KubeCon EU, Paris - How the Kubernetes Community is Improving Kubernetes for HPC/AI/ML Workloads - - Authors: Kevin Hannon + - Author: Kevin Hannon - FOSDEM 2024 ### Community adoption @@ -127,12 +129,9 @@ Operational tasks in [wg-governance.md]: - [x] [README.md] reviewed for accuracy and updated if needed - [x] WG leaders in [sigs.yaml] are accurate and active, and updated if needed - [x] Meeting notes and recordings for 2024 are linked from [README.md] and updated/uploaded if needed -- [] Updates provided to sponsoring SIGs in 2024 - - [$sig-name](https://git.k8s.io/community/$sig-id/) - - links to email, meeting notes, slides, or recordings, etc - - [$sig-name](https://git.k8s.io/community/$sig-id/) - - links to email, meeting notes, slides, or recordings, etc - - +- [x] Updates provided to sponsoring SIGs in 2024 + - WG-Batch Updates at Kubecon EU 2024 + - WG-Batch Updates at Kubecon NA 2024 [wg-governance.md]: https://git.k8s.io/community/committee-steering/governance/wg-governance.md [README.md]: https://git.k8s.io/community/wg-batch/README.md From e478776ec63a08134462d8af44b58cba983c2e52 Mon Sep 17 00:00:00 2001 From: Kevin Hannon Date: Fri, 17 Jan 2025 16:39:42 -0500 Subject: [PATCH 3/6] jobset risk --- wg-batch/annual-report-2024.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/wg-batch/annual-report-2024.md b/wg-batch/annual-report-2024.md index e42b99ea70f..b165220f648 100644 --- a/wg-batch/annual-report-2024.md +++ b/wg-batch/annual-report-2024.md @@ -8,7 +8,7 @@ See [2024 Highlights](#2024-highlights). 2. Are there any areas and/or subprojects that your group needs help with (e.g. fewer than 2 active OWNERS)? - None. + Yes, JobSet has 1 active owner at the moment. ### 2024 Highlights From cfd3cd82845b182b8b140277b8bd0c612ecbc0cc Mon Sep 17 00:00:00 2001 From: Kevin Hannon Date: Mon, 20 Jan 2025 10:10:53 -0500 Subject: [PATCH 4/6] address feedback from review --- wg-batch/annual-report-2024.md | 48 +++++++++++++++++++++------------- 1 file changed, 30 insertions(+), 18 deletions(-) diff --git a/wg-batch/annual-report-2024.md b/wg-batch/annual-report-2024.md index b165220f648..94625759126 100644 --- a/wg-batch/annual-report-2024.md +++ b/wg-batch/annual-report-2024.md @@ -18,7 +18,7 @@ We will breakdown our highlights into Sub Projects, KEPs, talks, community adopt ##### Kueue -Kueue has had 5 releases in 2024. +Kueue has had 5 minor releases in 2024. - [Release 0.6](https://github.com/kubernetes-sigs/kueue/releases/tag/v0.6.0) @@ -32,19 +32,25 @@ Kueue has had 5 releases in 2024. In 2024, the kueue community would like to highlight are Topology aware scheduling, MultiKueue, Kueue Dashboard, KueueCtrl, Deployment/Statefulset integration for serving and Fair sharing. -Topology aware scheduling facilitates scheduling of workloads that take in account data center topology. Workloads benefit from using interconnects that are physically close together. +[Topology aware scheduling](https://kueue.sigs.k8s.io/docs/concepts/topology_aware_scheduling/) facilitates scheduling of workloads that take in account data center topology. +Workloads benefit from using interconnects that are physically close together. -MultiKueue provides a way of dispatching batch workloads to worker clusters. Kueue provides multicluster dispatching for popular batch workloads such as Ray, Job, Kubeflow and JobSet. This feature went beta in 0.9. +[MultiKueue](https://kueue.sigs.k8s.io/docs/concepts/multikueue/) provides a way of dispatching batch workloads to worker clusters. +Kueue provides multicluster dispatching for popular batch workloads such as Ray, Job, Kubeflow and JobSet. +This feature went beta in 0.9. -Kueue Dashboards has been a popular ask for Kueue. Users would like to have a visualization representation of queueing and we are happy to announce that a dashboard has been created for Kueue. This went into kueue in late 2024 and a big focus of 2025 will be to harden this for production. +[Kueue Dashboards](https://github.com/kubernetes-sigs/kueue/tree/release-0.10/cmd/experimental/kueue-viz) has been a popular ask for Kueue. +Users would like to have a visualization representation of queueing and we are happy to announce that a dashboard has been created for Kueue. +This went into kueue in late 2024 and a big focus of 2025 will be to harden this for production. -KueueCtrl provides a cli for creating kueue objects. The plugin is hosted in krew and is easily installed as a kueue plugin. +[KueueCtl](https://kueue.sigs.k8s.io/docs/reference/kubectl-kueue/) provides a cli for creating kueue objects. +The plugin is hosted in krew and is easily installed as a kueue plugin. -Deployment/StatefulSet integration provides an avenue for the usage of Kueue for serving workloads. Serving leads to a need for sharing/preemption of model servers that may leverage accelerators. Kueue provides an integration with popular methods of deploying services (Deployment/StatefulSet). +[Deployment](https://kueue.sigs.k8s.io/docs/tasks/run/deployment/) and [StatefulSet](https://kueue.sigs.k8s.io/docs/tasks/run/statefulset/) integration provides an avenue for the usage of Kueue for serving workloads. Serving leads to a need for sharing/preemption of model servers that may leverage accelerators. Kueue provides an integration with popular methods of deploying services (Deployment/StatefulSet). ##### JobSet -Jobset has had 4 release in 2024. +Jobset has had 4 minor releases in 2024. - [Release 0.4](https://github.com/kubernetes-sigs/jobset/releases/tag/v0.4.0) @@ -54,14 +60,14 @@ Jobset has had 4 release in 2024. - [Release 0.7](https://github.com/kubernetes-sigs/jobset/releases/tag/v0.7.0) -A major achievement of JobSet has been the adoption of JobSet as a component for Kubeflow Training Operator V2. +A major achievement of JobSet has been the adoption of JobSet as a component for [Kubeflow Training Operator](https://github.com/kubeflow/training-operator) V2. There has been a collaborative effort with the Kubeflow community and the batch community to implement the features needed for this integration. [Metaflow](https://github.com/Netflix/metaflow/pull/1804) has adopted the use of JobSet for distributed ML training. ##### KJob -[KJob](https://github.com/kubernetes-sigs/kjob?tab=readme-ov-file#kjob) has been started to provide a CLI friendly way for users to submit batch jobs. +[KJob](https://github.com/kubernetes-sigs/kjob) has been started to provide a CLI friendly way for users to submit batch jobs. The HPC/ML community tend to prefer CLI over YAML so the focus was to provide a templated solution for submitting batch jobs. Another focus of this project is to provide a smooth transition for Slurm users. @@ -87,32 +93,39 @@ WG-Batch provided a series of kubernetes enhancements that improved the experien ### Talks - WG-Batch Update at Kubecon - - Authors: Kevin Hannon and Marcin Wielgus + - Speakers: Kevin Hannon and Marcin Wielgus - Kubecon NA, Salt Lake City + - [Recording](https://www.youtube.com/watch?v=C2ABOEzZTWg&list=PLj6h78yzYM2Pw4mRw4S-1p_xLARMqPkA7&index=283&pp=iAQB) - Keynote: MultiCluster Batch Jobs Dispatching with Kueue at CERN - - Authors: Ricardo Rocha and Marcin Wielgus + - Speakers: Ricardo Rocha and Marcin Wielgus - Kubecon NA, Salt Lake City + - [Recording](https://www.youtube.com/watch?v=xMmskWIlktA&list=PLj6h78yzYM2Pw4mRw4S-1p_xLARMqPkA7&index=193&pp=iAQB) - Multitenancy and Fairness at Scale with Kueue: A Case Study - - Authors: Aldo Culquicondor and Rajat Phull + - Speakers: Aldo Culquicondor and Rajat Phull - Kubecon NA, Salt Lake City + - [Recording](https://www.youtube.com/watch?v=GYiuTQCvTx8&list=PLj6h78yzYM2Mvqk_mNejD7kbe3tldxxsr&index=5&pp=iAQB) - Advanced Resource Management for Running AI/ML Workloads with Kueue - - Authors: Michał Woźniak and Yuki Iwai + - Speakers: Michał Woźniak and Yuki Iwai - Kubecon EU, Paris + - [Recording](https://www.youtube.com/watch?v=6k_8Go3u8Qk) - Scale Your Batch / Big Data / AI Workloads Beyond the Kubernetes Scheduler - - Authors: Antonin Stefanutti and Anish Asthana + - Speaker: Antonin Stefanutti and Anish Asthana - KubeCon EU, Paris + - [Recording](https://www.youtube.com/watch?v=Ij5EAnuF-jk&list=PLj6h78yzYM2PWGv34W6w5ssq1b1meRmY7&index=15&pp=iAQB) - WG-Batch Update - - Author: Marcin Wielgus + - Speaker: Michał Woźniak and Yuki Iwai - KubeCon EU, Paris + - [Recording](https://www.youtube.com/watch?v=2D2QSzUnS0M&list=PLj6h78yzYM2N8nw1YcqqKveySH6_0VnI0&index=84&pp=iAQB) - How the Kubernetes Community is Improving Kubernetes for HPC/AI/ML Workloads - Author: Kevin Hannon - FOSDEM 2024 + - [Recording](https://live.fosdem.org/watch/ua2118) ### Community adoption @@ -130,9 +143,8 @@ Operational tasks in [wg-governance.md]: - [x] WG leaders in [sigs.yaml] are accurate and active, and updated if needed - [x] Meeting notes and recordings for 2024 are linked from [README.md] and updated/uploaded if needed - [x] Updates provided to sponsoring SIGs in 2024 - - WG-Batch Updates at Kubecon EU 2024 - - WG-Batch Updates at Kubecon NA 2024 - + - [WG-Batch Updates at Kubecon EU 2024](https://www.youtube.com/watch?v=2D2QSzUnS0M&list=PLj6h78yzYM2N8nw1YcqqKveySH6_0VnI0&index=84&pp=iAQB) + - [WG-Batch Updates at Kubecon NA 2024](https://www.youtube.com/watch?v=C2ABOEzZTWg&list=PLj6h78yzYM2Pw4mRw4S-1p_xLARMqPkA7&index=283&pp=iAQB) [wg-governance.md]: https://git.k8s.io/community/committee-steering/governance/wg-governance.md [README.md]: https://git.k8s.io/community/wg-batch/README.md [sigs.yaml]: https://git.k8s.io/community/sigs.yaml From db02c0648d60a3d67995adca4646d47d39ccf012 Mon Sep 17 00:00:00 2001 From: Kevin Hannon Date: Mon, 3 Feb 2025 11:05:39 -0500 Subject: [PATCH 5/6] address nits --- wg-batch/annual-report-2024.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/wg-batch/annual-report-2024.md b/wg-batch/annual-report-2024.md index 94625759126..c31c73413bd 100644 --- a/wg-batch/annual-report-2024.md +++ b/wg-batch/annual-report-2024.md @@ -30,9 +30,9 @@ Kueue has had 5 minor releases in 2024. - [Release 0.10](https://github.com/kubernetes-sigs/kueue/releases/tag/v0.10.0) -In 2024, the kueue community would like to highlight are Topology aware scheduling, MultiKueue, Kueue Dashboard, KueueCtrl, Deployment/Statefulset integration for serving and Fair sharing. +In 2024, the kueue community would like to highlight Topology aware scheduling, MultiKueue, Kueue Dashboard, KueueCtrl, Deployment/Statefulset integration for serving and Fair sharing. -[Topology aware scheduling](https://kueue.sigs.k8s.io/docs/concepts/topology_aware_scheduling/) facilitates scheduling of workloads that take in account data center topology. +[Topology aware scheduling](https://kueue.sigs.k8s.io/docs/concepts/topology_aware_scheduling/) facilitates scheduling of workloads that take into account data center topology. Workloads benefit from using interconnects that are physically close together. [MultiKueue](https://kueue.sigs.k8s.io/docs/concepts/multikueue/) provides a way of dispatching batch workloads to worker clusters. From d2777b03f605b79eab69641e0e317d471859064a Mon Sep 17 00:00:00 2001 From: Kevin Hannon Date: Tue, 11 Feb 2025 16:21:44 -0500 Subject: [PATCH 6/6] address feedback from andrey --- wg-batch/annual-report-2024.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/wg-batch/annual-report-2024.md b/wg-batch/annual-report-2024.md index c31c73413bd..0ec0aa9d8b1 100644 --- a/wg-batch/annual-report-2024.md +++ b/wg-batch/annual-report-2024.md @@ -60,8 +60,8 @@ Jobset has had 4 minor releases in 2024. - [Release 0.7](https://github.com/kubernetes-sigs/jobset/releases/tag/v0.7.0) -A major achievement of JobSet has been the adoption of JobSet as a component for [Kubeflow Training Operator](https://github.com/kubeflow/training-operator) V2. -There has been a collaborative effort with the Kubeflow community and the batch community to implement the features needed for this integration. +A major achievement of JobSet has been the adoption of JobSet +as a component for [Kubeflow Trainer](https://github.com/kubeflow/trainer) V2, the next generation of the Kubeflow Training Operator project. [Metaflow](https://github.com/Netflix/metaflow/pull/1804) has adopted the use of JobSet for distributed ML training. @@ -92,6 +92,11 @@ WG-Batch provided a series of kubernetes enhancements that improved the experien ### Talks +- Democratizing AI Model Training on Kubernetes with Kubeflow TrainJob and JobSet + - Speakers: Andrey Velichkevich and Yuki Iwai + - Kubecon NA, Salt Lake City + - [Recording](https://www.youtube.com/watch?v=Lgy4ir1AhYw) + - WG-Batch Update at Kubecon - Speakers: Kevin Hannon and Marcin Wielgus - Kubecon NA, Salt Lake City @@ -129,8 +134,7 @@ WG-Batch provided a series of kubernetes enhancements that improved the experien ### Community adoption -- [Kubeflow Training Operator v2](https://github.com/kubeflow/training-operator/blob/0c30f5cd306611f061b6dd529d3c7b7981a7d27c/docs/proposals/2170-kubeflow-training-v2/README.md#kep-2170-kubeflow-training-v2-api) will be using JobSet as a critical component for training and finetuning. - +- [Kubeflow Trainer v2](https://github.com/kubeflow/trainer/tree/62e958fa8c07ae73be0b10a30e1fb5e4c3d0e8f3/docs/proposals/2170-kubeflow-training-v2) will be using JobSet as a critical component for distributed training and LLMs fine-tuning. - [Metaflow supports JobSet](https://github.com/Netflix/metaflow/pull/1804) for distributed training. - Airflow has built an [integration](https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/_api/airflow/providers/cncf/kubernetes/operators/kueue/index.html) with Kueue.