max_parallelism not linked to Manager.job_limit — BO concurrency silently capped at 3

[blttkgl]

Description

AxBackend has a hardcoded DEFAULT_BO_CONCURRENCY = 3 that caps how many trials Ax suggests per cycle. Manager has an independent job_limit that caps how many simulation jobs run concurrently. These two values are never linked, causing a silent resource underutilization bug:

Example: User sets job_limit=8. Ax only ever suggests 3 candidates. 5 cluster slots sit idle every cycle.

Root cause

DEFAULT_BO_CONCURRENCY mirrors Ax's internal DEFAULT_BAYESIAN_CONCURRENCY and is applied in AxBackend.initialize() when max_parallelism is None (the default). There is no code path that automatically propagates job_limit → max_parallelism.

Expected behavior

AxBackend.max_parallelism should default to Manager.job_limit unless explicitly overridden by the user.

Proposed fix

Wire them together at session construction time, e.g.:

if backend.max_parallelism is None:
    backend.max_parallelism = manager.job_limit

Or pass job_limit into AxBackend directly during Session setup.

[blttkgl]

Another option is to explicitly define max_parallelism in the job script:

    session.backend.initialization_trials = 4  # Ax recommends 2 * dim_search_space
    session.backend.max_parallelism= job_limit  

But I think that explicitly specifying the number of concurrent runs via Session, instead of interacting with backend, could be better.

Another issue is with how max_paralellism is handled. Ideally since Sobol sampling is quasi-random, it would make sense to apply max_parallelism only to BO samples, and set the job_limit as a limiter for Sobol samples. At the moment there is no distinction between the two. @499602D2 Any ideas how to move forward?

[499602D2]

@blttkgl Good catch! I think the sensible way is to just keep it mostly as-is, like you recommended: manager job limit as a hard resource utilization cap and use that during the Sobol phase, then treat the BO concurrency as another toggle (which hypothetical future backends might not support). Latter especially because you're really balancing resource efficiency and wall-clock time with parallel / async BO.

Also agreed on making this a Session toggle, so there's no need to reach into Session.backend internals.

[blttkgl]

Perfect, you can either throw a Claude at it, or I can do it good old fashioned way :)

At the moment efficient utilization of resources are handled manually, we usually start with an explicit max_parallelism that is equal to job limit, that is equal to number of initialization trials, since we can (and want) to sample initial points simultaneously. Seperating initialization job limit from optimization job limit via two different parameters that can be controlled from the Session should mitigate this issue.

[499602D2]

@blttkgl feel free to grab it ;)