sub queues or queue groups

[dimonnwc3]

Let's say I have a queue "encode" which has long running tasks. Many users can submit many tasks. I want to load balance tasks between users, by e.g a special "group" key so that one user can't block others for a long time.

10:30 user A submits 10 "encode" tasks
10:31 user B submits 10 "encode" tasks
Then user B will wait for all 10 tasks from user A to finish <- I want to prevent this

here is an example of such feature in bullmq: https://docs.bullmq.io/bullmq-pro/groups

is this possible to do and how?

[timgit]

Since v11, if you use a singleton or stately queue with singletonKeys and batching (> 1 job fetched at a time), only the 1st job per key is pulled into the batch, effectively load balancing as you described. This was built specifically to address a batching bug for these policies, but something similar could be done for any queue types using singletonKeys like this.

[lexuzieel]

Does this allow for round-robin load balancing if the number of singleton keys is larger than batch size?

Say, I have batch size of 3 and 5 unique keys A, B, C, D, E. Will the current implementation first fetch A, B and C in the first batch and later remaining D and E?

Also, on a somewhat unrelated note, how would this play along with priorities? I’m asking since I’m considering queue level prioritisation instead of job level prioritisation by the way of putting all of the jobs into a single global queue and using a combination of priorities and singleton keys

[timgit]

Does this allow for round-robin load balancing if the number of singleton keys is larger than batch size?
Say, I have batch size of 3 and 5 unique keys A, B, C, D, E. Will the current implementation first fetch A, B and C in the first batch and later remaining D and E?

No, it wouldn't.

Also, on a somewhat unrelated note, how would this play along with priorities?

Priorities would be handled before key "groups", so having a high priority attached to a key would guarantee that key would be pulled before others.