guestSocket forwards die with the SSH ControlMaster and are never re-created (silent on vz)

[MartyPine]

Description

Unix-socket forwards declared in portForwards (guestSocket rules, e.g. the docker socket in the docker template, or colima's docker.sock) are created exactly once at instance start, as ssh -O forward registrations against the SSH ControlMaster. If the master later dies, every forward it owned dies with it. The hostagent never detects this and never re-creates the forwards, so the host-side socket file is left behind as a stub that refuses every connection until the instance is restarted.

Host sleep is the reliable trigger (the master's TCP connection does not survive it). We also see the same failure on machines that have not slept, which points at idle-flow reaping in the usernet stack or network transitions killing a long-idle, keepalive-less connection — flagged as suspected rather than proven; the sleep case reproduces deterministically below.

On vz the failure is completely silent: the guest agent connection rides vsock (GuestAgentConn), which survives sleep, so the reconnect logic in watchGuestAgentEvents never fires. The hostagent keeps logging routine activity (time sync, guest agent events) with no error while the socket is dead. Inside the VM everything is healthy — when we first hit this in the field, dockerd had 24 h of uptime and 15 healthy containers while the host-side socket had been dead for hours. Users typically "fix" it with a full instance restart, which is how it hides inside vaguer reports.

Steps to reproduce

Any instance with a guestSocket port forward works; we used a docker runtime instance on vz.

1. Start the instance and verify the socket forward works:

   $ curl --unix-socket ~/.lima/<inst>/sock/docker.sock http://localhost/_ping
   OK

2. Find the ControlMaster:

   $ ssh -F ~/.lima/<inst>/ssh.config -O check lima-<inst>
   Master running (pid=13185)

3. Simulate what host sleep does to the master's connection:

   $ kill -9 13185

4. The socket is now dead and stays dead. We probed every 10 s for 120 s: 13/13 attempts refused, no recovery, and ha.stderr.log shows no error, no reconnect attempt, no re-forward — the hostagent has not noticed.
5. limactl shell <inst> still works, and (since limactl shell --reconnect fails when the previous ssh master crashed (stale ssh.sock not cleaned up) #4913) revives a fresh master — but the socket forward stays dead even after that. Reviving the master does not re-issue the forward registrations; they must be re-issued explicitly.

Root cause (from reading master @ 0990395)

• watchGuestAgentEvents sets up the guestSocket forwards once, outside any retry or reconcile path: https://github.com/lima-vm/lima/blob/0990395/pkg/hostagent/hostagent.go#L750-L758
• The only re-forward logic on reconnect covers the guest agent's own ga.sock, and it only runs when the guest agent connection breaks. On vz the guest agent uses vsock (https://github.com/lima-vm/lima/blob/0990395/pkg/driver/vz/vz_driver_darwin.go#L542-L548), so a dead SSH master never surfaces there.
• Nothing keeps the master alive or notices its death: SSHOpts sets ControlMaster=auto / ControlPersist=yes with no ServerAliveInterval (https://github.com/lima-vm/lima/blob/0990395/pkg/sshutil/sshutil.go#L595-L598), and the guest sshd has ClientAliveInterval 0. A connection that exchanges no traffic for hours is exposed to everything that reaps idle flows.
• The stale-master helpers added for limactl shell --reconnect fails when the previous ssh master crashed (stale ssh.sock not cleaned up) #4913 (IsControlMasterRunning, RemoveStaleControlMaster) are only wired into limactl shell — the hostagent never consults them.
• Two sharp edges for anyone attempting a fix: ssh -O forward cannot create a master (it fails if none exists, which is exactly the post-sleep state), and combining -f -N with -O fails silently.

Proposed fix

Two small host-side changes; no guest changes needed.

1. Keepalives (prevention). The master should notice its connection is gone instead of persisting as a zombie, and regular traffic protects it from idle reaping:

// pkg/sshutil/sshutil.go, SSHOpts()
opts = append(opts,
    fmt.Sprintf("User=%s", username),
    "ControlMaster=auto",
    controlPath,
    "ControlPersist=yes",
    "ServerAliveInterval=15",
    "ServerAliveCountMax=4",
)

2. Reconcile loop in the hostagent (heal). Pseudo-code:

every 10s, while socket forwarding is active:
    if IsControlMasterRunning(instDir):
        continue                      # near-free in the healthy case
    RemoveStaleControlMaster(instDir)
    run a trivial remote command over the normal ssh path
        # re-spawns the master via ControlMaster=auto;
        # `ssh -O` alone cannot create one
    for each rule in portForwards where rule.GuestSocket != "":
        forwardSSH(verbCancel, rule)  # best-effort, releases stale registration
        forwardSSH(verbForward, rule)

We have built and field-tested exactly this (plus the keepalives) on a fork, based on v2.2.0: https://github.com/MartyPine/lima/tree/fix/hostagent-socket-forward-reconcile — including a unit test in the style of the existing ga.sock forward tests. Measured results: the baseline above never recovers; with the patch, detection happens within one 10 s tick and restore takes about a second (kill-to-healed of 3–9 s across repeated runs, 5 s in production use; the reconciler triggers exactly once per kill and never while healthy). Sharing the branch as evidence that the approach works, not as a finished PR — happy for it to be adapted or rewritten however the maintainers prefer.

Longer term, vz could route guestSocket forwards over the gRPC forwarder the way TCP ports already go — pkg/portfwd already has host-side unix listener support — which would remove SSH from the socket path entirely on vz. The reconciler would still matter for qemu and any fallback path, so the two are complementary.

Related issues

• Lima's X11 forwarding breaks after a MacBook goes to sleep #2099 — X11 forwarding breaks after sleep. Same mechanism, different symptom; the first maintainer comment there ("We need a way to restore the SSH connections after sleep") describes this issue's subject.
• limactl shell --reconnect fails when the previous ssh master crashed (stale ssh.sock not cleaned up) #4913 — fixed stale ssh.sock handling for limactl shell only; the forwards were left unhealed, and this issue is the remaining half.
• VM randomly loses network #442 — "VM randomly loses network" likely absorbs some reports of this fault (host-side forwards dead, VM fine).
• Colima and docker instability: randomly freezing, requiring reset abiosoft/colima#1033 — collects many docker-flavoured reports of this same fault, mixed with unrelated VM freezes. Worth noting for triage there: colima currently pins LIMA_SSH_PORT_FORWARDER=true, so those users also lose all TCP forwards when the master dies, making the failure look much bigger than the socket forwards alone.
• Colima seems to "die" after sleep on Mac. abiosoft/colima#460 — "Colima seems to die after sleep", open since 2022: the same symptom pattern (works again after colima stop && colima start), consistent with dead forwards rather than a dead VM.

[MartyPine]

Full disclosure - this issue content was generated with AI, but the proposed fix seems to be working well on the fork I have running locally so far...

[MartyPine]

Field update after running the proposed fix in production for about a week.

Setup: lima 2.2.0 on Apple Silicon, vz + virtiofs, portForwarder: grpc (via colima). Patched with both halves of the fix — the hostagent ControlMaster reconciler, and ServerAliveInterval=15 / ServerAliveCountMax=4 in SSHOpts.

Over an uninterrupted four-day stretch the machine went through roughly two dozen sleep/wake cycles with no ControlMaster death, no reconciler activation, and no false triggers. The reconciler's periodic liveness check sat as a quiet no-op the whole time — no spurious re-forwards, no noticeable overhead.

Two honest caveats so this isn't read as more than it is:

I shipped the keepalives and the reconciler together, so I can't attribute the quiet period to either one specifically. It's equally consistent with "keepalives prevent the idle-timeout death" and with "this trigger is just rarer than the thread suggests."
In this window the fault never occurred on its own. The only way I've reproduced it is by killing the ControlMaster deliberately — after which stock lima leaves the guestSocket forwards dead indefinitely (silent on vz, since the guest-agent link is vsock and the reconnect loop never notices), while the patched build re-establishes them in a few seconds. So I can speak to the reconciler being correct and safe in steady state, but I haven't caught it rescuing a real, un-forced death yet.
Takeaways I'd offer:

The reconciler looks safe to ship as a steady-state no-op.
The ServerAliveInterval change is one line and might carry a lot of the benefit on its own. It would be worth testing keepalive-only (no reconciler) to isolate whether it alone stops the idle-death variant.
If others hitting this want to help isolate it: try adding ServerAliveInterval 15 / ServerAliveCountMax 4 to your lima ssh config, keep your normal workflow, and report whether the "dead after sleep/idle" recurrence stops. If it does for several people, keepalives likely cover the idle variant and the reconciler stays as the backstop for genuine master deaths (sleep sever, network change) that keepalives can't prevent.

(generated with AI)

Marty here - Things have been pretty stable all week. Still keeping an eye on things, but the random outages have definitely stopped.