Skip to content

[ADH-8475] Support engine profiles#24

Open
tigrulya-exe wants to merge 2 commits into
develop/4.3.0/1.11.1.1from
feature/1.11/ADH-8475
Open

[ADH-8475] Support engine profiles#24
tigrulya-exe wants to merge 2 commits into
develop/4.3.0/1.11.1.1from
feature/1.11/ADH-8475

Conversation

@tigrulya-exe

Copy link
Copy Markdown
Member

This PR adds named static engine profiles. An admin declares a profile once in Kyuubi server configuration (engine type + env + Kyuubi session vars + engine-native config); a client selects it by name (or gets one by user/group/engine-type default), and the server launches the right engine without the client setting any low-level details.
Backward compatible: with no profiles declared, behavior is unchanged.

Main changes

  • Profile model & registryEngineProfile, EngineProfileRegistry materialize
    kyuubi.engine.profile.<name>.* declarations into engine conf at startup.
  • ResolutionEngineProfileResolver picks the profile at session open with precedence:
    explicit session param → user/group default → per-engine-type default → none.
  • Session applyKyuubiSessionImpl applies the profile as config defaults with client-supplied
    options overrides.
  • IsolationEngineRef folds the profile name into the engine subdomain part, so engines
    from different profiles never share an instance/lock/pool.

    Note: it means that subdomain part of the engines created from profiles will contain profile name prefix, so it should be confidered, when operating with client methods accepting subdomain as an input parameter.

  • REST — new GET /api/v1/admin/engine/profile endpoint groups live engines by profile + engine
    type. Idle (declared but not running) profiles are listed too.
  • Web UI — new management view for engines grouped by profile.

Added configs

Key Default Description
kyuubi.engine.profile (none) Session param selecting the named profile to use for this session.
kyuubi.engine.<TYPE>.profile.default (none) Per-engine-type default profile, used when the session requests no profile and no user/group default applies.
kyuubi.engine.profiles.unknown.strategy FAIL Behavior when a resolved profile name is not defined: FAIL (error) or LOG (warn and apply none).

Profiles themselves are declared (not config entries, but a namespace) as
kyuubi.engine.profile.<name>.<bucket>...:

Bucket Maps to Meaning
.type kyuubi.engine.type engine type (validated)
.env.<VAR> kyuubi.engineEnv.<VAR> environment variable
.session.<rest> kyuubi.session.<rest> Kyuubi session variable
.conf.<rest> <rest> verbatim engine-native config

UI changes

  • New view views/management/engine-profile — table of profiles (profile, engine type,
    version, instance count, status) with per-instance and per-profile Kill, and Refresh.
Screenshot 2026-07-01 at 19 32 30 Screenshot 2026-07-01 at 19 31 43

Example

Declare profiles in kyuubi-defaults.conf:

# Spark 4 profile
kyuubi.engine.profile.spark4.type=SPARK_SQL
kyuubi.engine.profile.spark4.env.SPARK_HOME=/usr/lib/spark4
kyuubi.engine.profile.spark4.conf.spark.executor.memory=2g

# Postgres (JDBC) profile
kyuubi.engine.profile.pg.type=JDBC
kyuubi.engine.profile.pg.conf.kyuubi.engine.jdbc.type=postgresql
kyuubi.engine.profile.pg.conf.kyuubi.engine.jdbc.driver.class=org.postgresql.Driver
kyuubi.engine.profile.pg.conf.kyuubi.engine.jdbc.connection.url=jdbc:postgresql://localhost:5432/demo
kyuubi.engine.profile.pg.conf.kyuubi.engine.jdbc.connection.user=kyuubi
kyuubi.engine.profile.pg.conf.kyuubi.engine.jdbc.connection.password=kyuubi

# Optional defaults
kyuubi.engine.SPARK_SQL.profile.default=spark4
___analysts___.kyuubi.engine.profile=pg

Connect and select a profile via the JDBC url:

jdbc:hive2://localhost:10009/default;#kyuubi.engine.profile=spark4
jdbc:hive2://localhost:10009/default;#kyuubi.engine.profile=pg

Comment thread kyuubi-server/src/main/scala/org/apache/kyuubi/session/KyuubiSessionManager.scala Outdated
@tigrulya-exe tigrulya-exe force-pushed the feature/1.11/ADH-8475 branch from 3293086 to 5f57edf Compare July 2, 2026 22:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants