feat(dev): multi-worktree isolation and remote runner plugin for datahub-dev (#17550)

Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

Author: shirshanka

AGENTS.md

@@ -468,8 +468,9 @@ A stdlib-only Python CLI for agent-driven development. No venv needed — runs w
 scripts/dev/datahub-dev.sh <command>
 ```
 
-Run `scripts/dev/datahub-dev.sh --help` to see all available subcommands (`start`, `stop`, `setup`,
-`frontend`, `docs`, `status`, `wait`, `rebuild`, `test`, `flag list/get`, `env`, `sync-flags`, `reset`, `nuke`).
+Run `scripts/dev/datahub-dev.sh --help` to see all available subcommands (`start`, `stop`, `suspend`,
+`setup`, `frontend`, `docs`, `status`, `wait`, `rebuild`, `test`, `flag list/get`, `env`,
+`sync-flags`, `reset`, `nuke`, `instances list/clean`, `shell-env`).
 
 ### End-to-End Workflow
 
@@ -529,6 +530,95 @@ or after a fresh clone.
 When starting, `datahub-dev start` automatically detects and stops conflicting DataHub instances
 from other worktrees/compose projects that occupy the same ports.
 
+### Remote Runners
+
+`datahub-dev.sh` supports a **runner plugin** that proxies operations to a remote machine
+(EC2, Kubernetes pod, or any SSH-accessible host) instead of running Docker locally.
+
+**Configure a runner** in `~/.datahub/dev/config.json`:
+
+```json
+{
+  "max_local_instances": 2,
+  "max_remote_instances": 10,
+  "runner": "/path/to/your-runner.sh"
+}
+```
+
+Or export `DATAHUB_RUNNER=/path/to/runner.sh` in your shell for a one-off session.
+
+**Remote lifecycle** (all commands work identically to local once a runner is set):
+
+```bash
+# One-time bootstrap — provisions the remote environment
+scripts/dev/datahub-dev.sh setup --remote
+
+# Start — syncs changed local files, runs quickstartDebug on the remote,
+#          then sets up port tunnels so local ports reach the remote instance
+scripts/dev/datahub-dev.sh start
+
+# Stop containers only (remote compute keeps running)
+scripts/dev/datahub-dev.sh stop
+
+# Stop containers AND halt the remote compute (no billing while suspended).
+# 'start' will automatically resume the instance when needed.
+scripts/dev/datahub-dev.sh suspend
+
+# All other commands (status, wait, rebuild, test, flag, env, nuke, …)
+# proxy through the runner transparently — use them exactly as you would locally.
+scripts/dev/datahub-dev.sh status
+```
+
+**Multi-instance management** — each git worktree gets its own isolated instance
+(separate Docker project, volumes, and port assignment):
+
+```bash
+# List all registered instances (local and remote) with their ports and status
+scripts/dev/datahub-dev.sh instances list
+
+# Remove stale entries for worktrees that no longer exist
+scripts/dev/datahub-dev.sh instances clean
+
+# Print export statements for the current instance's CLI environment
+eval $(scripts/dev/datahub-dev.sh shell-env)
+# → sets DATAHUB_GMS_URL to the correct local port (tunnel or direct)
+```
+
+**Port assignment** — each instance gets a slot; ports = base + slot × 1000:
+
+| Slot | GMS   | Frontend | Notes                     |
+| ---- | ----- | -------- | ------------------------- |
+| 0    | 8080  | 9002     | First local instance      |
+| 1    | 9080  | 10002    | Second local instance     |
+| 2    | 10080 | 11002    | First remote instance     |
+| …    | …     | …        | Each worktree is isolated |
+
+**Backwards compatibility / opting out of isolation** — if the new per-worktree
+project names cause problems (lost data in old volumes, tooling that expects
+`datahub-*` container names, CI environments that don't need isolation), set
+`compose_project` in `~/.datahub/dev/config.json`:
+
+```json
+{ "compose_project": "datahub" }
+```
+
+This reverts to the old single-instance behaviour: one `datahub` Docker project,
+same container names, existing volumes fully accessible. The env var
+`COMPOSE_PROJECT_NAME=datahub` has the same effect without touching the config file.
+
+**Runner interface** — a runner is any executable that speaks four verbs:
+
+```bash
+runner init                        # one-time environment bootstrap
+runner sync                        # push changed local files to the remote
+runner exec -- <cmd> [args...]     # execute a command in the remote workspace
+runner tunnel <local:remote> ...   # set up port forwarding
+runner resume                      # start compute if stopped (no-op if running)
+runner suspend                     # stop containers + halt compute
+```
+
+A reference Kubernetes runner is at `scripts/dev/runners/k8s.sh`.
+
 ### Recovery Escalation
 
 **When to use each:**

docker/build.gradle

@@ -10,7 +10,7 @@ apply from: "../gradle/versioning/versioning.gradle"
 
 ext {
     compose_base = "profiles/docker-compose.yml"
-    project_name = "datahub"
+    project_name = System.getenv("COMPOSE_PROJECT_NAME") ?: "datahub"
 
     backend_profile_modules = [
             ':datahub-upgrade',

docker/profiles/docker-compose.gms.yml

@@ -209,7 +209,7 @@ x-datahub-mae-consumer-service: &datahub-mae-consumer-service
   hostname: datahub-mae-consumer
   image: ${DATAHUB_MAE_CONSUMER_IMAGE:-${DATAHUB_REPO:-acryldata}/datahub-mae-consumer}:${DATAHUB_MAE_VERSION:-${DATAHUB_VERSION:-quickstart}}
   ports:
-    - 9091:9091
+    - ${DATAHUB_MAPPED_MAE_CONSUMER_PORT:-9091}:9091
   expose:
     - 4319
   env_file:
@@ -240,7 +240,7 @@ x-datahub-mce-consumer-service: &datahub-mce-consumer-service
   hostname: datahub-mce-consumer
   image: ${DATAHUB_MCE_CONSUMER_IMAGE:-${DATAHUB_REPO:-acryldata}/datahub-mce-consumer}:${DATAHUB_MCE_VERSION:-${DATAHUB_VERSION:-quickstart}}
   ports:
-    - 9090:9090
+    - ${DATAHUB_MAPPED_MCE_CONSUMER_PORT:-9090}:9090
   expose:
     - 4319
   env_file:
@@ -258,8 +258,8 @@ x-datahub-mce-consumer-service-dev: &datahub-mce-consumer-service-dev
   <<: *datahub-mce-consumer-service
   image: ${DATAHUB_MCE_CONSUMER_IMAGE:-${DATAHUB_REPO:-acryldata}/datahub-mce-consumer}:${DATAHUB_MCE_VERSION:-${DATAHUB_VERSION:-debug}}
   ports:
-    - 9090:9090
-    - 5009:5009
+    - ${DATAHUB_MAPPED_MCE_CONSUMER_PORT:-9090}:9090
+    - ${DATAHUB_MAPPED_MCE_DEBUG_PORT:-5009}:5009
   environment: &datahub-mce-consumer-env-dev
     <<: [*datahub-dev-telemetry-env, *datahub-mce-consumer-env]
     ENTITY_VERSIONING_ENABLED: ${ENTITY_VERSIONING_ENABLED:-true}

metadata-integration/java/datahub-client/src/main/resources/MetadataChangeProposal.avsc

@@ -172,11 +172,6 @@
         "type" : [ "null", "string" ],
         "doc" : "Aspect version\n   Initial implementation will use the aspect version's number, however stored as\n   a string in the case where a different aspect versioning scheme is later adopted.",
         "default" : null
-      }, {
-        "name" : "schemaVersion",
-        "type" : [ "null", "long" ],
-        "doc" : "Schema version of the aspect data model. Used to determine aspects that need migrations.\nDefaults to 1 when not present as a baseline. Incremented when an aspect is modified.",
-        "default" : null
       }, {
         "name" : "aspectCreated",
         "type" : [ "null", {

metadata-integration/java/datahub-event/src/main/resources/MetadataChangeProposal.avsc

@@ -172,11 +172,6 @@
         "type" : [ "null", "string" ],
         "doc" : "Aspect version\n   Initial implementation will use the aspect version's number, however stored as\n   a string in the case where a different aspect versioning scheme is later adopted.",
         "default" : null
-      }, {
-        "name" : "schemaVersion",
-        "type" : [ "null", "long" ],
-        "doc" : "Schema version of the aspect data model. Used to determine aspects that need migrations.\nDefaults to 1 when not present as a baseline. Incremented when an aspect is modified.",
-        "default" : null
       }, {
         "name" : "aspectCreated",
         "type" : [ "null", {

scripts/dev/datahub-dev.sh

@@ -3,4 +3,56 @@
 #   scripts/datahub-dev.sh status
 #   scripts/datahub-dev.sh rebuild --wait
 # etc.
+
+# Read per-user config from ~/.datahub/dev/config.json.
+# Reads two keys (runner, compose_project) in a single Python call to avoid
+# paying the interpreter startup cost twice.
+if [ "$DATAHUB_REMOTE_EXEC" != "1" ]; then
+  _dh_cfg=$(python3 - <<'EOF' 2>/dev/null
+import json, pathlib, sys
+cfg = pathlib.Path.home() / ".datahub/dev/config.json"
+try:
+    d = json.loads(cfg.read_text())
+    r = d.get("runner", "")
+    p = d.get("compose_project", "")
+    if r:
+        r = str(pathlib.Path(r).expanduser())
+    print(r)
+    print(p)
+except Exception:
+    print("")
+    print("")
+EOF
+)
+  if [ -z "$DATAHUB_RUNNER" ]; then
+    DATAHUB_RUNNER=$(echo "$_dh_cfg" | sed -n '1p')
+    [ -n "$DATAHUB_RUNNER" ] && export DATAHUB_RUNNER
+  fi
+  if [ -z "$COMPOSE_PROJECT_NAME" ]; then
+    COMPOSE_PROJECT_NAME=$(echo "$_dh_cfg" | sed -n '2p')
+    [ -n "$COMPOSE_PROJECT_NAME" ] && export COMPOSE_PROJECT_NAME
+  fi
+  unset _dh_cfg
+fi
+
+# When a runner is configured and we're not already on the remote side,
+# proxy ALL commands through the runner — EXCEPT the ones that are inherently
+# local (managing the local registry or printing local connection info).
+# 'start' and 'setup --remote' are handled specially inside datahub_dev.py
+# itself, so they also skip this proxy and reach Python directly.
+if [ -n "$DATAHUB_RUNNER" ] && [ "$DATAHUB_REMOTE_EXEC" != "1" ]; then
+  case "${1:-}" in
+    instances|shell-env|start|setup|suspend)
+      # Local-aware commands: let Python handle them (start/setup/suspend have
+      # their own runner logic; instances/shell-env read the local registry).
+      ;;
+    *)
+      # Everything else (stop, reset, nuke, status, wait, rebuild, flag, env,
+      # sync-flags, test, docs, frontend) runs on the remote.
+      exec "$DATAHUB_RUNNER" exec -- \
+        env DATAHUB_REMOTE_EXEC=1 "$(realpath "$0")" "$@"
+      ;;
+  esac
+fi
+
 exec uv run --python 3.11 --no-project "$(dirname "$0")/datahub_dev.py" "$@"