guestSocket forwards die with the SSH ControlMaster and are never re-created (silent on vz)

Description

Unix-socket forwards declared in portForwards (guestSocket rules, e.g. the docker socket in the docker template, or colima's docker.sock) are created exactly once at instance start, as ssh -O forward registrations against the SSH ControlMaster. If the master later dies, every forward it owned dies with it. The hostagent never detects this and never re-creates the forwards, so the host-side socket file is left behind as a stub that refuses every connection until the instance is restarted.

Host sleep is the reliable trigger (the master's TCP connection does not survive it). We also see the same failure on machines that have not slept, which points at idle-flow reaping in the usernet stack or network transitions killing a long-idle, keepalive-less connection — flagged as suspected rather than proven; the sleep case reproduces deterministically below.

On vz the failure is completely silent: the guest agent connection rides vsock (GuestAgentConn), which survives sleep, so the reconnect logic in watchGuestAgentEvents never fires. The hostagent keeps logging routine activity (time sync, guest agent events) with no error while the socket is dead. Inside the VM everything is healthy — when we first hit this in the field, dockerd had 24 h of uptime and 15 healthy containers while the host-side socket had been dead for hours. Users typically "fix" it with a full instance restart, which is how it hides inside vaguer reports.

Steps to reproduce

Any instance with a guestSocket port forward works; we used a docker runtime instance on vz.

Start the instance and verify the socket forward works:

$ curl --unix-socket ~/.lima/<inst>/sock/docker.sock http://localhost/_ping
OK

Find the ControlMaster:

$ ssh -F ~/.lima/<inst>/ssh.config -O check lima-<inst>
Master running (pid=13185)

Simulate what host sleep does to the master's connection:
```
$ kill -9 13185
```
The socket is now dead and stays dead. We probed every 10 s for 120 s: 13/13 attempts refused, no recovery, and ha.stderr.log shows no error, no reconnect attempt, no re-forward — the hostagent has not noticed.
limactl shell <inst> still works, and (since limactl shell --reconnect fails when the previous ssh master crashed (stale ssh.sock not cleaned up) #4913) revives a fresh master — but the socket forward stays dead even after that. Reviving the master does not re-issue the forward registrations; they must be re-issued explicitly.

Root cause (from reading master @ `0990395`)

watchGuestAgentEvents sets up the guestSocket forwards once, outside any retry or reconcile path: https://github.com/lima-vm/lima/blob/0990395/pkg/hostagent/hostagent.go#L750-L758
The only re-forward logic on reconnect covers the guest agent's own ga.sock, and it only runs when the guest agent connection breaks. On vz the guest agent uses vsock (https://github.com/lima-vm/lima/blob/0990395/pkg/driver/vz/vz_driver_darwin.go#L542-L548), so a dead SSH master never surfaces there.
Nothing keeps the master alive or notices its death: SSHOpts sets ControlMaster=auto / ControlPersist=yes with no ServerAliveInterval (https://github.com/lima-vm/lima/blob/0990395/pkg/sshutil/sshutil.go#L595-L598), and the guest sshd has ClientAliveInterval 0. A connection that exchanges no traffic for hours is exposed to everything that reaps idle flows.
The stale-master helpers added for limactl shell --reconnect fails when the previous ssh master crashed (stale ssh.sock not cleaned up) #4913 (IsControlMasterRunning, RemoveStaleControlMaster) are only wired into limactl shell — the hostagent never consults them.
Two sharp edges for anyone attempting a fix: ssh -O forward cannot create a master (it fails if none exists, which is exactly the post-sleep state), and combining -f -N with -O fails silently.

Proposed fix

Two small host-side changes; no guest changes needed.

1. Keepalives (prevention). The master should notice its connection is gone instead of persisting as a zombie, and regular traffic protects it from idle reaping:

// pkg/sshutil/sshutil.go, SSHOpts()
opts = append(opts,
    fmt.Sprintf("User=%s", username),
    "ControlMaster=auto",
    controlPath,
    "ControlPersist=yes",
    "ServerAliveInterval=15",
    "ServerAliveCountMax=4",
)

2. Reconcile loop in the hostagent (heal). Pseudo-code:

every 10s, while socket forwarding is active:
    if IsControlMasterRunning(instDir):
        continue                      # near-free in the healthy case
    RemoveStaleControlMaster(instDir)
    run a trivial remote command over the normal ssh path
        # re-spawns the master via ControlMaster=auto;
        # `ssh -O` alone cannot create one
    for each rule in portForwards where rule.GuestSocket != "":
        forwardSSH(verbCancel, rule)  # best-effort, releases stale registration
        forwardSSH(verbForward, rule)

We have built and field-tested exactly this (plus the keepalives) on a fork, based on v2.2.0: https://github.com/MartyPine/lima/tree/fix/hostagent-socket-forward-reconcile — including a unit test in the style of the existing ga.sock forward tests. Measured results: the baseline above never recovers; with the patch, detection happens within one 10 s tick and restore takes about a second (kill-to-healed of 3–9 s across repeated runs, 5 s in production use; the reconciler triggers exactly once per kill and never while healthy). Sharing the branch as evidence that the approach works, not as a finished PR — happy for it to be adapted or rewritten however the maintainers prefer.

Longer term, vz could route guestSocket forwards over the gRPC forwarder the way TCP ports already go — pkg/portfwd already has host-side unix listener support — which would remove SSH from the socket path entirely on vz. The reconciler would still matter for qemu and any fallback path, so the two are complementary.

Related issues

Lima's X11 forwarding breaks after a MacBook goes to sleep #2099 — X11 forwarding breaks after sleep. Same mechanism, different symptom; the first maintainer comment there ("We need a way to restore the SSH connections after sleep") describes this issue's subject.
limactl shell --reconnect fails when the previous ssh master crashed (stale ssh.sock not cleaned up) #4913 — fixed stale ssh.sock handling for limactl shell only; the forwards were left unhealed, and this issue is the remaining half.
VM randomly loses network #442 — "VM randomly loses network" likely absorbs some reports of this fault (host-side forwards dead, VM fine).
Colima and docker instability: randomly freezing, requiring reset abiosoft/colima#1033 — collects many docker-flavoured reports of this same fault, mixed with unrelated VM freezes. Worth noting for triage there: colima currently pins LIMA_SSH_PORT_FORWARDER=true, so those users also lose all TCP forwards when the master dies, making the failure look much bigger than the socket forwards alone.
Colima seems to "die" after sleep on Mac. abiosoft/colima#460 — "Colima seems to die after sleep", open since 2022: the same symptom pattern (works again after colima stop && colima start), consistent with dead forwards rather than a dead VM.

Full disclosure - this issue content was generated with AI, but the proposed fix seems to be working well on the fork I have running locally so far...

Field update after running the proposed fix in production for about a week.

Setup: lima 2.2.0 on Apple Silicon, vz + virtiofs, portForwarder: grpc (via colima). Patched with both halves of the fix — the hostagent ControlMaster reconciler, and ServerAliveInterval=15 / ServerAliveCountMax=4 in SSHOpts.

Over an uninterrupted four-day stretch the machine went through roughly two dozen sleep/wake cycles with no ControlMaster death, no reconciler activation, and no false triggers. The reconciler's periodic liveness check sat as a quiet no-op the whole time — no spurious re-forwards, no noticeable overhead.

Two honest caveats so this isn't read as more than it is:

I shipped the keepalives and the reconciler together, so I can't attribute the quiet period to either one specifically. It's equally consistent with "keepalives prevent the idle-timeout death" and with "this trigger is just rarer than the thread suggests."
In this window the fault never occurred on its own. The only way I've reproduced it is by killing the ControlMaster deliberately — after which stock lima leaves the guestSocket forwards dead indefinitely (silent on vz, since the guest-agent link is vsock and the reconnect loop never notices), while the patched build re-establishes them in a few seconds. So I can speak to the reconciler being correct and safe in steady state, but I haven't caught it rescuing a real, un-forced death yet.
Takeaways I'd offer:

The reconciler looks safe to ship as a steady-state no-op.
The ServerAliveInterval change is one line and might carry a lot of the benefit on its own. It would be worth testing keepalive-only (no reconciler) to isolate whether it alone stops the idle-death variant.
If others hitting this want to help isolate it: try adding ServerAliveInterval 15 / ServerAliveCountMax 4 to your lima ssh config, keep your normal workflow, and report whether the "dead after sleep/idle" recurrence stops. If it does for several people, keepalives likely cover the idle variant and the reconciler stays as the backstop for genuine master deaths (sleep sever, network change) that keepalives can't prevent.

(generated with AI)

Marty here - Things have been pretty stable all week. Still keeping an eye on things, but the random outages have definitely stopped.

Month-one field update, plus one finding that may help triage the reports that land here and in the downstream issues.

Extended field result (same setup as before: lima 2.2.0, vz + virtiofs, portForwarder: grpc, patched with the reconciler + ServerAliveInterval=15 / ServerAliveCountMax=4):

31 days in production, 33 host sleep/wake transitions, hostagent currently 4+ days continuous.
Zero reconciler activations in every surviving hostagent log, zero false triggers. The 10 s liveness poll has stayed a silent no-op the whole month.

Same caveat as before, unchanged: keepalives and reconciler shipped together, so this is a confounded null — it can't say which one prevents the death, only that with both in place the master never died on its own across a month of sleeps. If it would help isolate the keepalive's contribution, I have a throwaway instance and can run keepalive-only (no reconciler) across sleep cycles and report back.

A triage discriminator, from the outages we did see. Over the same month the docker VM went down four times — and none were this bug. Every one was Received SIGTERM, shutting down the host agent in ha.stderr.log: the hostagent was killed by something on the host (a local port-freeing script that killed whatever held a forwarded port, a timeout watchdog that fired kill at a PID the OS had recycled, and two OS shutdowns). On vz the VM runs inside the hostagent process, so killing limactl takes the whole VM down — which reads, from the outside, exactly like "docker died."

That's a different failure from #5420, and I suspect the two get conflated in the downstream threads. A cheap way to tell them apart, for anyone landing here:

What you see	Likely cause
`docker.sock` dead, VM still running, hostagent process alive, no `Received SIG…` in `ha.stderr.log`	#5420 — the forward was stranded when the ControlMaster died; the reconciler heals it
hostagent process gone, `Received SIGTERM` (or SIGKILL) in `ha.stderr.log`	Not #5420 — something on the host killed limactl; look for kill-by-port scripts (`lsof -ti :PORT \| xargs kill`) or delayed `kill $pid` watchdogs on the host

Worth noting for the second row: with portForwarder: grpc, limactl itself binds every forwarded container port on the host, so any "free this port by killing its listener" script hits limactl instead of the app it meant to kill. That was the mechanism behind most of our four.

(Generated with the help of AI)

@MartyPine

Hey @MartyPine , I also hit this issue a lot with the guestSocket forwards dying. I can get it to reproduce consistently with my setup, but I haven't bothered to make a minimum reproducible example yet, since the code is very proprietary.
While I didn't really try out your fork, I went an alternate route and made vz route guestSockets over the GRPC forwarder. If you want to take a look, the commit is in my fork: KayleMaster@1d80d32
Looks like we can't open a pull request until one of the lima contributors marks this as ready to work, so I guess I'll be running with a developer version until then. But this does seem to fix the root issue that I encountered in rancher-sandbox/rancher-desktop#9839

@KayleMaster

@KayleMaster — thanks, that's the structural route I'd hoped someone would take, and it's the better end-state on vz: with guestSocket rules going through the gRPC/vsock tunnel there is no ControlMaster in the socket path at all, so there is nothing to strand. I read the commit — small, and the unix tests are a nice touch.

For @jandubois, I think these two are complementary rather than competing, and they touch different lines (yours changes the initial forward's condition; mine adds a liveness poll + re-forward after it), so they'd merge together with only a trivial conflict at the if rule.GuestSocket != "" guard:

gRPC unix sockets (KayleMaster): removes the failure class on vz + portForwarder: grpc. The gaps it leaves are exactly the ones the commit keeps on SSH — rule.Reverse sockets, the SSH forwarder mode, and qemu-on-macOS where the gRPC tunnel itself rides an SSH-forwarded ga.sock.
Reconciler + keepalives (mine): the backstop for those gaps — heals any ControlMaster death regardless of cause, and the ServerAliveInterval half turns a hung master into a detected one. Steady-state it's a no-op (a month in production, zero activations, zero false triggers).

So the picture I'd suggest: gRPC sockets as the default on vz, reconciler as the safety net everywhere else. I'll include @KayleMaster's changes in my fork and see how they go and report back if a decision hasn't been made by then. If docker.sock stays up with the reconciler never firing, then we're on to a winner.

(Marty - this is my AI talking. This stuff is way outside my comfort zone, so take it as you will :) )

@MartyPine

@MartyPine I'm currently overloaded, but I already started looking into this last week. The gRPC setting is obviously separate and does not affect the SSH ControlMaster. You should not experience the ControlMaster disconnect while using gRPC port forwarding because the only connections using it will be Unix sockets.

I think I found the reason for the ControlMaster disconnecting: on macOS OpenSSH uses select instead of poll, and fails when dealing with more than 1024 connections (because Lima raises the limit from 256 to unlimited). So I think just keeping the file descriptor limit at 1024 will prevent it from disconnecting to begin with. But I haven't found the time yet to test this properly. Hopefully later this week, or maybe next.

@KayleMaster

@KayleMaster I don't understand why you need to modify Lima to set portforwarding to gRPC; it should be the default. You would have to explicitly select SSH by setting LIMA_SSH_PORT_FORWARDER=true before calling limactl start.

It is not currently the default in Rancher Desktop (but probably should be), but you can configure this with:

rdctl set --experimental.virtual-machine.ssh-port-forwarder=false

@jandubois

@jandubois — that matches the data from this setup: under gRPC forwarding, a month in production has shown zero organic ControlMaster disconnects, consistent with the fd-limit explanation.

On my earlier note about trying @KayleMaster's commit: tested on my set up on vz with portForwarder: grpc — three synthetic ControlMaster kills, 0 failed docker.sock probes across 60 s each, lsof showing limactl rather than ssh owning the socket. Same kill on the stock/reconciler build drops the socket for 5–9 s.

If I can be of any assistance, let me know.

Hi mate, I know you're busy so I got curious and tried to give you some more info which might help move the ball.

Field test of the proposed 1024 fd cap on macOS/vz. Short version: the theory holds, and a 1024 soft limit does keep the ControlMaster alive — but on vz a cap set in adjustNofileRlimit() never reaches the ControlMaster, because Virtualization.framework raises the hostagent's soft RLIMIT_NOFILE about 60 ms later.

Setup. Lima v2.2.0 + a one-line patch (limit.Cur = min(limit.Max, 1024) in adjustNofileRlimit), macOS 26 (Darwin 25.6), /usr/bin/ssh = OpenSSH_10.3p1 (nm -u shows it imports _pselect and no poll/ppoll, so the poll compat is select-backed), vz, a throwaway colima instance in ssh-forwarder mode. Load = N idle TCP connections held open through one forwarded port (redis PING/PONG), N = 500 / 1000 / 1024 / 1100, then released.

One confound had to be removed first: the Ubuntu guest's sshd session has soft NOFILE 1024, so the guest refuses new channels at ~1017 (open failed: connect failed) at almost the same count where the host would break. The guest sshd was given LimitNOFILE=65536 so only the host limit varies below.

Results

ControlMaster	soft NOFILE at spawn	1000 conns	≥1024 conns
hand-started ssh, `ulimit -n 1048576` (no Lima code involved)	1048576	ok	process gone at 1017 conns; nothing logged at `LogLevel=DEBUG1 -E`, no crash report
hand-started ssh, `ulimit -n 1024`	1024	ok	survives: `accept: Too many open files`, fds plateau at 1024, full recovery once connections are released
Lima hostagent, cap in `adjustNofileRlimit()` (startup)	3328	ok	dies exactly like the uncapped case
Lima hostagent, same patch, cap re-applied immediately before `waitForRequirements("essential")` (diagnostic only, to isolate the variable)	1024	ok	survives; docker.sock and the TCP forward recover after release

Rows 2 and 4 were repeated with Lima's exact ssh option set and at Lima's ControlPath — same outcome.

Why the startup cap is lost. Sampling the hostagent's own RLIMIT_NOFILE every 50 ms:

15:00:13.517 cur=1024            (after adjustNofileRlimit)
15:00:13     Mounting disk ...
15:00:13.574 cur=3328  (changed) <- inside VZ start
15:00:13     [VZ] vm state change: VirtualMachineStateStarting
15:00:19.748 cur=1024            (diagnostic re-apply, row 4 only)
15:00:20     Explicitly start ssh ControlMaster

The ControlMaster is spawned by the hostagent itself (essential requirement 3), ~6 s after the raise, and inherits 3328. Lima does not import fsnotify or anything else that calls setrlimit; the raise comes from Virtualization.framework during VZVirtualMachine.start.

One side effect of the cap worth knowing. At the ceiling the mux listener EMFILEs too (channel_post_mux_listener: accept: Too many open files), so ssh -O check / -O forward from the hostagent can stall until connections drain. It recovered in every run once load dropped; a reconnect tick with a timeout handles it.

Raw CSVs and the ssh -E logs for every run are available if useful.

@jandubois

edit: reposting because I accidentally posted from my work account earlier
@jandubois My commit extends the same gRPC-tunnel forwarding to GuestSocket rules too, so they stop depending on the ControlMaster. The GuestSocket rule seems to be added by rancher-desktop by default:

portForwards:
  - guestPortRange: [1, 65535]
    guestIPMustBeZero: true
    hostIP: 0.0.0.0
    proto: any
  - guestSocket: /var/run/docker.sock # does not get grpc forwarded
    hostSocket: /Users/myuser/.rd/docker.sock

and yes, I do have rancher desktop configured to with grpc port fwd. But that's not enough to fix the issue when the guestSocket dies, also experienced by users here rancher-sandbox/rancher-desktop#9839 (comment)

I just reproduced the issue on two more Macs - Tahoe, and to solve it I'm currently running my fork of lima with the guestSocket over gRPC fix. The issue reproduces 100% otherwise. I do agree that the SSH itself should also not crash even without gRPC port fwd enabled. They're probably two separate issues, although both fix the overarching problem.

One finding from running the guestSocket-over-gRPC change in production for five days, for anyone else carrying it on a v2.2.x base:

docker exec, docker run and docker attach through the gRPC-forwarded docker.sock returned no output at all (exit 0, empty stdout). Plain requests such as docker ps and docker logs were fine, and the same exec through an SSH-forwarded socket was fine. The only shape that worked was docker exec -i with stdin held open.

Cause: the Docker CLI half-closes its side after sending the request; the host relays that as CloseSend, and the v2.2.0 guest agent's portfwdserver treats it as a full close and tears the tunnel down before the response is relayed. That is exactly #5264 / #5329, fixed by #5401 (98ae93ac, merged 2026-08-24, after the v2.2.0 tag). Cherry-picking that one commit on top of the guestSocket change restored all of it (exec, run, 200 KB streams, stdin), with the guest agent reinstalled at the next boot. Note that the commit at KayleMaster/lima@1d80d32 is based on 0990395 (2026-08-12), which also predates #5401, so a build from that branch would see the same symptom on TCP-style half-close clients until it is rebased.

Not a defect in the guestSocket change itself — it just moves docker.sock onto a path that needs #5401 — but worth knowing for backports and for anyone on Rancher Desktop testing that fork.

Thanks Marty, I will rebase it to include that fix when I get some free time.

docs(colima): root-cause analysis of the 2026-10-08 wedged forward; c…

docs(next-steps): upgrade lima when lima-vm/lima#5420 ships

@MartyPine

This comment was drafted by Claude Code from the Lima hostagent and OpenSSH sources.

Keeping ssh's soft fd limit at 1024 (#5631) should stop the master from dying under load on macOS, but nothing brings the forwards back after a death from any other cause. Here's a proposal for that part; objections and simpler ideas are welcome before anyone writes code for it.

The hostagent starts the ControlMaster as a long-lived child (ssh -M -N with ControlMaster=yes and ControlPersist=no), and pipes its stderr into the hostagent log. It then learns at once when the master exits, and why. Today the master detaches with stderr on /dev/null, so nothing records why it dies.
Every other Lima ssh client (limactl shell, copy, tunnel, the hostagent's own commands, and any client that uses the generated ssh.config) switches to ControlMaster=no but keeps the ControlPath. Such a client still multiplexes over a running master and connects directly when none runs, but it never creates a master of its own. Today any of them can replace a dead master with a new one that doesn't own any forwards.
When the master exits, the hostagent removes the stale ssh.sock, restarts the master with a backoff, and replays the forwards that the hostagent owns onto the new master. Those are the guestSocket rules, the static TCP forwards, and the SSH forwarder's dynamic ports, which it takes from the guest agent's current port list. Reverse-sshfs mounts either remount after a restart or stop using the master, which they already do on Windows.
ServerAliveInterval on that one master would turn a hung connection into an exit that the hostagent sees.

A restart loses the forwards that Lima doesn't own, such as limactl tunnel's SOCKS forward or a user's own ssh -O forward; we'd document that.

I wouldn't take the 10 s reconciler from @MartyPine's branch as is, though it's useful prior art. It checks that some master answers on ssh.sock, not that the master that holds the forwards is still running. So a limactl shell run between the death and the next tick creates a fresh master, and the check passes while the forwards stay dead. It also replays only guestSocket rules, so with the SSH port forwarder, every TCP forward stays dead.

This comment was drafted by Claude Code from a test on a throwaway VM.

Lima itself can kill the master. limactl shell --reconnect runs ssh -O exit on it, which drops every forward that the master holds, and the shell's own ssh then starts a new master that has none of them.

I checked this on vz with #5631 applied and LIMA_SSH_PORT_FORWARDER=true, using a guestSocket rule and a TCP port forwarded by the SSH forwarder. Both answered before limactl shell --reconnect <instance> -- true. Five and 35 seconds after it, the host socket file was still there but refused connections, the TCP port refused too, and the hostagent logged nothing about it.

So the fd limit alone doesn't fix this issue, and a test for the recovery proposed above can trigger the death with --reconnect instead of killing a process.

@KayleMaster

I've filed #5631 to lower the fd limit to 1024 to avoid the ControlMaster from getting killed by too many open connections.

I've also filed #5635 to mount reverse-sshfs mounts over their own connections, so they are not affected by ControlMaster restarts, e.g. via limactl shell --reconnect.

The maintainers have discussed on Slack that we do not want to add the complexity of reconnecting sockets when the ControlMaster gets reconnected; you will have to restart the instance instead, or use the default gRPC port forwarding. #5639 has been closed without getting merged.

I expect #5631 and #5635 to be merged for Lima 2.3.0.

This leaves @KayleMaster's changes as the last open question from this issue. It will take docker.sock off the ControlMaster as well (just like #5635 takes sshfs mounts off), so the docker socket will also survive a ControlMaster reconnect.

@KayleMaster can you please open a PR with your changes (I can't remember if you special-cased docker.sock; I think all Linux sockets should use their own connections). Let me know if you lack the time and want me to create the PR instead. We only have about another week until the 2.3.0 release.

@jandubois

@jandubois I'll try to open it by end of the week

feat: support Unix sockets with gRPC port forwarding

portfwd: support Unix socket forwarding over gRPC

Navigation Menu

Repository navigation

guestSocket forwards die with the SSH ControlMaster and are never re-created (silent on vz) #5420

Description

Steps to reproduce

Root cause (from reading master @ `0990395`)

Proposed fix

Related issues

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

guestSocket forwards die with the SSH ControlMaster and are never re-created (silent on vz) #5420

Description

Description

Steps to reproduce

Root cause (from reading master @ 0990395)

Proposed fix

Related issues

Activity

MartyPine commented on Aug 17, 2026

MartyPine commented on Aug 25, 2026

MartyPine commented on Sep 18, 2026

KayleMaster commented on Sep 22, 2026

MartyPine commented on Sep 23, 2026

jandubois commented on Sep 23, 2026

jandubois commented on Sep 23, 2026

MartyPine commented on Sep 23, 2026

MartyPine commented on Sep 23, 2026

KayleMaster commented on Sep 23, 2026

KayleMaster commented on Sep 25, 2026

MartyPine commented on Sep 28, 2026

KayleMaster commented on Sep 29, 2026

jandubois commented on Oct 15, 2026

jandubois commented on Oct 16, 2026

jandubois commented on Oct 19, 2026

KayleMaster commented on Oct 20, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Root cause (from reading master @ `0990395`)