Add support for multiple Celery worker groups with queue-specific configurations

[HsiuChuanHsu]

Summary

This PR adds support for multiple Celery worker groups. Each group can now be independently configured to target specific queues and use distinct resource settings

The goal is to achieve fine-grained control over task distribution by assigning queues to specific worker nodes.

Changes

New Configuration: workers.celeryQueueGroups
Added support for defining multiple worker deployments through the workers.celeryQueueGroups configuration. Each group can specify:

• name: (required) Unique name for the worker group
• queues: (required) Comma-separated list of Celery queues
• replicas: (required) Number of worker replicas
• nodeSelector: (optional)
• affinity: (optional)
• tolerations: (optional)
• topologySpreadConstraints: (optional)
• resources: (optional)
• labels: (optional)
• podAnnotations: (optional)
• priorityClassName: (optional)
• env: (optional)

worker-deployment.yaml Logic

• Modified the worker deployment template to loop through all configured worker groups
• For default behavior (no celeryQueueGroups defined), the template maintains original setup using existing workers.* configuration
• Each worker group creates a separate Deployment/StatefulSet with:
  • Unique name suffix based on group name (e.g., airflow-worker-high-priority)
  • worker-group label for identification
  • Queue-specific command arguments: exec airflow celery worker --queues <group-queues>
  • Custom resource configurations override default settings per group

worker-service.yaml Logic

• Extended service template to create separate services for each worker group
• Each service is scoped to its corresponding worker group using the worker-group label selector
• Service naming follows the pattern: {{ airflow.fullname }}-worker-<group-name>

closes: #56591

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[HsiuChuanHsu]

Example Configuration

workers:
  celeryQueueGroups:
    - name: default-workers
      queues: "default,email,notifications"
      replicas: 1
    - name: heavy-workers
      queues: "reports,batch_processing"
      replicas: 2

[rcampos87]

We have this exact use case where we want to define different queues for different celery workers, potentially with different labels and tolerations. Thanks for the PR!

[Miretpl]

Hi @HsiuChuanHsu, thanks for this PR! I personally planned to work on this feature somewhere in the future. I have one question: some time ago, I merged the split of the workers section to workers.kubernetes and workers.celery (for now, only for the service account conf), as with the Hybrid Executor setup, having workers section used by these two different executors becomes a little problematic. What do you think about placing your works under the workers.celery section, as it is only applicable to the celery workers?

[HsiuChuanHsu]

Hi @Miretpl,
Thanks for pointing out the newly added workers.celery & workers. kubernetes sections within the configuration.
I hadn't noticed those sections. You're right; the celeryQueueGroups setting definitely belongs under workers.celery. I'll make that change.