Merge pull request #7753 from grondo/vqueues-support

add core support for RFC 33 virtual queues

Author: mergify[bot]

doc/guide/admin_config.rst

@@ -575,6 +575,17 @@ define a default queue.
 Finally, queues can override the ``[policy]`` table on a per queue basis.
 This is useful for setting queue-specific limits.
 
+A queue may also be configured as a *virtual queue* by setting ``parent``
+to the name of another queue. A virtual queue is an alternate submission
+name for the parent queue's resources with different policy: it inherits the
+parent's policy and overrides only the keys it sets. For example, a virtual
+queue that raises the parent's limits can allow a subset of users to submit
+jobs that bypass those limits. Note that Flux does not yet enforce per-queue
+access control, so restricting who may use a virtual queue currently requires
+an external mechanism such as the flux-accounting multi-factor priority
+plugin. A virtual queue may also be started, stopped, enabled, or disabled
+independently of its parent. See :man5:`flux-config-queues` for details.
+
 Here is an example that puts these concepts together:
 
 .. code-block:: toml
@@ -630,8 +641,12 @@ is rejected.
   ``nnodes`` *and* ``ncores`` limits when configuring job size policy limits.
 
 Limits are global when set in the top level ``[policy]`` table.  Global limits
-may be overridden by a ``policy`` table within a ``[queues]`` entry.  Here is
-an example which implements duration and job size limits for two queues:
+may be overridden by a ``policy`` table within a ``[queues]`` entry. A virtual
+queue's ``policy`` table overrides its parent queue's limits on a per-key
+basis, so a virtual queue that overrides only the duration limit still
+inherits the parent's job size limits. Here is an example which implements
+duration and job size limits for two queues, and adds a virtual queue for
+smaller jobs with a longer duration limit in the batch queue.
 
 .. code-block:: toml
 
@@ -652,6 +667,12 @@ an example which implements duration and job size limits for two queues:
  policy.limits.job-size.max.nnodes = 16
  policy.limits.job-size.max.ncores = 128
 
+ [queues.small]
+ parent = "batch"
+ policy.limits.duration = "24h"
+ policy.limits.job-size.max.nnodes = 2
+ policy.limits.job-size.max.ncores = 16
+
 See also: :man5:`flux-config-policy`.
 
 Use PAM to Restrict Access to Compute Nodes

doc/man1/flux-jobs.rst

@@ -56,6 +56,10 @@ OPTIONS
 
    List jobs in a specific queue or queues. Multiple queues may be separated
    by a comma or by using the :option:`-q, --queue` option multiple times.
+   If a named queue has virtual queues configured (RFC 33), then jobs
+   submitted to any of its virtual queues are also listed, since they are
+   scheduled as part of the named queue. Naming a virtual queue directly
+   lists only the jobs submitted to that virtual queue.
 
 .. option:: -i, --include=HOSTS|RANKS
 

doc/man1/flux-queue.rst

@@ -32,6 +32,15 @@ The :program:`flux queue` command operates on Flux job queue(s).
 By default, Flux has one anonymous queue.  Multiple named queues may be
 configured - see :man5:`flux-config-queues`.
 
+A named queue may be a virtual queue, as described in RFC 33: an alternate
+submission name for a parent queue's resources with different policy,
+such as raised limits for a subset of users. Jobs submitted to a virtual
+queue are scheduled as part of the parent queue. The :program:`flux queue`
+administrative commands below operate on a virtual queue's own state
+without affecting its parent or sibling queues, but note that a virtual
+queue's jobs are only eligible for scheduling when both the virtual queue
+and its parent are started.
+
 COMMANDS
 ========
 
@@ -42,7 +51,9 @@ list
 
 .. program:: flux queue list
 
-List queue status, defaults, and limits.
+List queue status, defaults, and limits. When one or more virtual
+queues are configured, a ``PARENT`` column is also shown, listing each
+virtual queue's parent queue (blank for a non-virtual queue).
 
 .. option:: -q, --queue=QUEUE,...
 
@@ -223,23 +234,35 @@ The following field names can be specified:
 **queuem**
    queue name, but default queue is marked up with an asterisk
 
+**parent**
+   parent queue name if this is a virtual queue (RFC 33), otherwise empty.
+   This field is only shown by default if at least one configured queue
+   has a parent.
+
 **submission**
    Description of queue submission status: ``enabled`` or ``disabled``
 
 **scheduling**
-   Description of queue scheduling status: ``started`` or ``stopped``
+   Description of queue scheduling status: ``started``, ``stopped``, or,
+   for a virtual queue whose own state is started but whose parent queue
+   is stopped, ``stopped (parent)``.
 
 **enabled**
    Single character submission status: ``✔`` if enabled, ``✗`` if disabled.
 
 **started**
    Single character scheduling status: ``✔`` if started, ``✗`` if stopped.
+   A virtual queue whose own state is started but whose parent queue is
+   stopped shows ``⏸`` instead: scheduling will resume automatically once
+   the parent is started.
 
 **enabled.ascii**
    Single character submission status: ``y`` if enabled, ``n`` if disabled.
 
 **started.ascii**
-   Single character scheduling status: ``y`` if started, ``n`` if stopped.
+   Single character scheduling status: ``y`` if started, ``n`` if stopped,
+   or ``p`` if started but paused because its parent is stopped (see
+   **started** above).
 
 **defaults.timelimit**
    default timelimit for jobs submitted to the queue

doc/man1/flux-resource.rst

@@ -493,7 +493,9 @@ The following field names can be specified for the **list** subcommand:
 
 **queue**
    Queue(s) associated with resources. When ``-q, --queue`` is used,
-   only the specified queues are shown.
+   only the specified queues are shown. Virtual queues (RFC 33) do not
+   appear in this field, since they share their parent queue's resources
+   rather than being associated with any resources of their own.
 
 **properties**
    Properties associated with resources.

doc/man5/flux-config-policy.rst

@@ -12,7 +12,10 @@ and limits, as described in RFC 33.
 Each queue defined in the ``queues`` table described in
 :man5:`flux-config-queues` may have a ``policy`` sub-table that follows the
 same rules as the main table.  The per-queue table overrides the general table
-for jobs submitted to that queue.
+for jobs submitted to that queue. A virtual queue (RFC 33) adds a further
+level: its own ``policy`` table, where set, overrides its parent queue's
+effective policy on a per-key basis, which in turn overrides the general
+table.
 
 DEFAULTS
 ========

doc/man5/flux-config-queues.rst

@@ -25,11 +25,44 @@ policy (table)
    A policy table as described in :man5:`flux-config-policy` that overrides
    the general system policy for jobs submitted to this queue.
 
+parent (string)
+   Declares this queue to be a virtual queue, as described in RFC 33. The
+   value names another queue in the ``[queues]`` table from which this
+   queue inherits its resource subset and policy. A virtual queue MUST
+   NOT set ``requires`` or ``policy.scheduler``: it always shares its
+   parent's resources, and thus its parent's scheduler configuration.
+   The named parent must not itself be a virtual queue. A virtual queue
+   may be configured as the default queue.
+
 A default queue name may be configured by setting
 ``policy.jobspec.defaults.system.queue`` as described in
 :man5:`flux-config-policy`.
 
 
+VIRTUAL QUEUES
+==============
+
+A virtual queue is an alternate name under which jobs may be submitted to
+a parent queue's resources, with different policy applied at job
+ingest. A job submitted to a virtual queue keeps the virtual queue's name
+for listing, accounting, and per-queue policy purposes, but is scheduled
+as part of its parent queue: it competes for the parent's resources in the
+same priority order as jobs submitted directly to the parent.
+
+A virtual queue inherits every policy key from its parent, and any key it
+sets itself overrides only that key. For example, a virtual queue that
+sets ``policy.limits.duration`` but not ``policy.limits.job-size`` still
+inherits the parent's job-size limits. Job defaults
+(``policy.jobspec.defaults.system``) are inherited the same way.
+
+A virtual queue may be enabled/disabled and started/stopped
+independently of its parent. However, because a virtual queue's jobs are
+scheduled as part of its parent, they are only eligible for scheduling
+when both the virtual queue and its parent are started. Starting a
+virtual queue whose parent is stopped does not release its jobs; they
+become eligible only once the parent is also started.
+
+
 EXAMPLE
 =======
 
@@ -51,8 +84,13 @@ EXAMPLE
 
    [queues.batch]
    policy.limits.duration = "8h"
+   policy.limits.job-size.max.nnodes = 16
    requires = [ "batch" ]
 
+   [queues.expedite]
+   parent = "batch"
+   policy.limits.duration = "1h"
+
    [policy.jobspec.defaults.system]
    queue = "batch"
 
@@ -63,6 +101,11 @@ EXAMPLE
    match-policy = "lonodex"
    match-format = "rv1_nosched"
 
+In this configuration, ``expedite`` is a virtual queue of ``batch``: jobs
+submitted with ``--queue=expedite`` run on ``batch``'s resources, subject
+to a shorter 1 hour duration limit, but still inherit ``batch``'s
+16 node job-size limit since ``expedite`` does not override it.
+
 
 CAVEATS
 =======

src/bindings/python/flux/job/frobnicator/plugins/constraints.py

@@ -25,9 +25,25 @@ def __init__(self, config={}):
 
     def queue_properties(self, name):
         try:
-            return self.queues[name]["requires"]
+            entry = self.queues[name]
         except KeyError:
             return None
+        # A virtual queue (RFC 33) has no 'requires' of its own -
+        # inject the parent's instead, since that is what makes the
+        # job schedule as part of the parent's job list. Inheritance
+        # is one level (validated by conf_policy.c), so no chain to
+        # walk. An unresolvable parent is a fatal error (fail closed):
+        # silently dropping the parent's constraint would place the
+        # job outside the parent's resource slice.
+        parent = entry.get("parent")
+        if parent is not None:
+            try:
+                entry = self.queues[parent]
+            except KeyError:
+                raise ValueError(
+                    f"queue '{name}': parent queue '{parent}' is not configured"
+                )
+        return entry.get("requires")
 
     def apply_constraints(self, jobspec):
         """Apply queue-specific constraints to jobspec"""

src/bindings/python/flux/job/frobnicator/plugins/defaults.py

@@ -49,18 +49,42 @@ def validate_config(self):
             self.queue_defaults(queue)
 
     def queue_defaults(self, name):
-        """Create a copy of self.defaults updated with queue-specific values"""
+        """Create a copy of self.defaults updated with queue-specific values
+
+        Effective defaults are layered: global defaults, then (if 'name'
+        is a virtual queue, RFC 33) the parent queue's own defaults,
+        then this queue's own defaults - each layer overlaid per-key
+        over the last. Inheritance is one level (validated by
+        conf_policy.c), so there is no chain to walk beyond the parent.
+        """
+
+        def queue_system_defaults(qconf):
+            try:
+                return qconf["policy"]["jobspec"]["defaults"]["system"]
+            except KeyError:
+                return None
+
         defaults = copy.deepcopy(self.defaults)
         if name and self.queues:
             if name not in self.queues:
                 raise ValueError(f"Invalid queue '{name}' specified")
             qconf = self.queues[name]
-            try:
-                qdefaults = qconf["policy"]["jobspec"]["defaults"]["system"]
+            # A virtual queue (RFC 33) inherits the parent's defaults
+            # beneath its own. An unresolvable parent is a fatal error
+            # (fail closed): silently skipping the parent layer would
+            # give the vqueue the wrong effective defaults.
+            parent = qconf.get("parent")
+            if parent is not None:
+                if parent not in self.queues:
+                    raise ValueError(
+                        f"queue '{name}': parent queue '{parent}' is not configured"
+                    )
+                pdefaults = queue_system_defaults(self.queues[parent])
+                if pdefaults is not None:
+                    defaults.update(pdefaults)
+            qdefaults = queue_system_defaults(qconf)
+            if qdefaults is not None:
                 defaults.update(qdefaults)
-                return defaults
-            except KeyError:
-                return defaults
         return defaults
 
     def setattr_default(self, jobspec, attr, value):