Repository navigation
Guest agent time sync counts host-to-guest RPC latency as clock drift and steps the guest clock backwards under host load #5543
Description
Description
GuestServer.SyncTime (pkg/guestagent/api/server/server.go, v2.1.2) computes drift := now.Sub(hostTime) where hostTime is the timestamp the host put in the request, and calls timesync.SetSystemTime(hostTime) when |drift| > 100ms. The host's stamp is already latency old when the guest reads it, so the measured "drift" is the guest's real drift plus the host-to-guest transport latency, and the correction sets the guest clock to a time that is latency in the past. Whenever the RPC takes more than 100 ms (a loaded host), every 10-second sync steps the guest clock backwards by the latency, even if the guest clock was correct.
On a busy macOS host we see this at every sync for minutes at a time. Log excerpt from ~/.colima/_lima/colima/ha.stderr.log (11,226 guest clock adjusted entries since 2026-09-19 on this machine):
{"level":"info","msg":"Time sync: guest clock adjusted (was 113ms off)","time":"2026-09-25T15:08:48+08:00"}
{"level":"info","msg":"Time sync: guest clock adjusted (was 100ms off)","time":"2026-09-25T15:11:48+08:00"}
{"level":"info","msg":"Time sync: guest clock adjusted (was 102ms off)","time":"2026-09-25T15:13:29+08:00"}
{"level":"info","msg":"Time sync: guest clock adjusted (was 102ms off)","time":"2026-09-25T15:15:58+08:00"}
The largest adjustments logged that day were 1388 ms, 4102 ms and 9577 ms. A probe inside a container that read CLOCK_REALTIME every 10 ms recorded the guest clock jumping back by 47 ms to 853 ms at the sync instants, and the guest journal shows matching "Time jumped backwards" entries. The sign is always the same: the drift is positive (guest ahead of the stale host stamp), so the clock only ever moves back.
Impact
Anything in the guest that orders events by wall-clock time breaks. Our case: GHDL records each design file's analysis time in its library file and reports "X is obsoleted by Y" when a dependency's stamp is newer, so a file analysed just after a step looks older than its dependencies and the whole build fails at random under host load. We had to add a wrapper that waits until the clock passes the newest stamp before each analysis.
Environment
- Lima 2.1.2 (via Colima 0.10.1, vz driver), macOS 26.5.1, Apple silicon (arm64 guest)
- Reproduces whenever the host's load average exceeds roughly the core count; no reproduction on an idle host.
Suggested fix
Measure the round trip on the guest side and treat half of it as transport, i.e. compare now with hostTime + rtt/2 (or have the host send both a send stamp and the guest reply with its receive stamp, NTP-style), and never step the clock backwards by less than the measured latency. Alternatively only adjust when the same-signed drift persists across several samples, which a latency spike does not.
Workaround we use
A ghdl wrapper that keeps analysis stamps monotonic (waits past the newest library stamp, re-analyses when its own stamp comes out older). Happy to share it or test a fix.
Activity
mvanhorn commentedon Oct 3, 2026
The host agent captured time.Now() at send time and passed that instant to SyncTime as host_time.
startTimeSync keeps the two most recent completed round-trip durations and passes them into each tick. syncGuestClock sends host_time as the send time plus oneWayDelay, half the smaller of those two samples, and that delay is zero while the latest sample is not positive, so the first sync goes out uncompensated and one long sample stays out of the next estimate.
fix: stop guest time sync from stepping the clock back by RPC latency
fix: keep stale delay estimates from stepping the guest clock
jandubois commentedon Oct 5, 2026
This comment is by Claude Code, from a test Jan asked it to run on his Mac.
We tried to reproduce this on an M1 with vz VMs running Ubuntu 26.04. Even with heavy CPU load on both host and guest, latency above 100 ms never lasted past a single sync. Two backward steps did happen, 114 ms and 134 ms, each from a one-sync latency spike, with the host clock 32 to 86 ms behind NTP adding to the drift. That is far rarer than what you see. Master stepped twice in 46 minutes of full load, and you logged 11,226 adjustments in six days. So our CPU-only load probably differs from yours in some way that matters. The details are in #5565.
Which fix helps depends on what your steps look like, so could you check four things on the machine that shows them?
sntp time.apple.comon the host while the steps happen. It prints how far the host clock is from NTP, and an offset of 50 ms already accounts for half of the 100 ms threshold.timedatectl timesync-statusorchronyc tracking. If it does, the host clock's offset from NTP adds to every drift Lima measures.ha.stderr.logaround a step with debug logging on (with Lima directly,limactl --debug start). Debug logging records every sync's drift as "Time sync: drift 10ms within threshold", which shows whether the drift rose on one sync or over several.sysctl vm.swapusageon the host,free -min the guest). A VM stalled under memory pressure could explain both your rate and the multi-second adjustments.#5565 helps only when the latency stays high across several syncs. If you can try a build of it on that machine, that would answer the question directly.