Skip to content

Add trackpad mode for relative pointer input on touch devices#2065

Open
benjaminfrombe wants to merge 29 commits into
novnc:masterfrom
benjaminfrombe:trackpad-mode
Open

Add trackpad mode for relative pointer input on touch devices#2065
benjaminfrombe wants to merge 29 commits into
novnc:masterfrom
benjaminfrombe:trackpad-mode

Conversation

@benjaminfrombe

Copy link
Copy Markdown

Summary

noVNC maps touch input to absolute framebuffer coordinates, which is
awkward on phones/tablets: your finger covers the target, you can't
reach screen edges easily, and there is no way to zoom/pan locally.
This PR adds an opt-in trackpad mode that behaves like the iOS RDP
app or a laptop trackpad: one finger moves a virtual cursor relatively,
pinch zooms the view locally (without sending anything to the remote),
and two-finger drag either pans the magnified view or scrolls the remote.

What it does

Toggle via RFB.trackpadMode = true or the new "Trackpad mode (touch)"
checkbox in the settings panel.

One finger

  • Moves the virtual cursor relatively (scaled by trackpadSensitivity,
    default 1.5) — driven from raw touchmove events so there is no dead
    zone before movement is visible
  • Tap = left click at the virtual cursor
  • Long-press = right click (and drag = drag with button held)
  • Tap immediately followed by drag ("tap-and-a-half") = drag with left
    button held

Two fingers

  • Pinch = local magnification (1×–5×) via a CSS transform on the canvas
    (GPU-composited, smooth, the remote is never touched)
  • While magnified: two-finger drag pans the view toward the fingers
  • At fit (1×): two-finger drag sends mouse-wheel scroll events
  • Zoom is anchored at the midpoint between the fingers (RDP-style)

Cursor visibility

  • A dot cursor is always shown in trackpad mode (falls back to the dot
    when the server sends no visible cursor), so you can always see where
    the virtual cursor is

iOS Safari

  • Blocks the browser's own pinch-to-zoom (which ignores user-scalable=no
    since iOS 10) by preventing the proprietary gesturestart/gesturechange
    events (detected via their .scale property, so noVNC's own synthetic
    gesture CustomEvents are unaffected)

Bug fix included

gesturehandler.js _twoTouchTimeout: the existing code chose between
PINCH and TWODRAG by comparing finger movement per axis against the
inter-finger distance change. A diagonal drag splits its movement across
both axes (~70% each), so it would lose to distance noise and be
misclassified as PINCH even when the fingers stayed parallel. Fixed by
comparing the 2D movement magnitude (hypot) instead.

Implementation notes

  • All two-finger and one-finger movement handling is in raw touchstart/ move/end handlers that run alongside the gesture handler. The gesture
    handler retains responsibility for tap classification only.
  • The CSS transform (translate()+scale(), origin 0 0) is purely visual;
    it does not affect the canvas layout size, so absX/absY still map
    pointer coordinates correctly to framebuffer pixels.
  • The virtual cursor position is kept in untransformed canvas CSS pixels;
    _trackpadUpdateCursor maps it through the bounding rect's effective
    scale so the dot follows the logical position even while magnified.
  • trackpadMode and trackpadSensitivity are public properties on RFB.
    Leaving trackpad mode resets the CSS transform.

Test plan

  • Enable "Trackpad mode (touch)" on a mobile browser and verify:
    • One-finger drag moves the cursor from the first pixel (no dead zone)
    • Tap clicks, two-finger tap right-clicks
    • Long-press holds the button; tap-then-drag does the same
    • Pinch zooms in/out around the midpoint between fingers
    • While zoomed: two-finger drag pans the view (including diagonally)
    • At fit: two-finger drag scrolls the remote
    • Dot cursor is visible without moving first
    • iOS Safari does not zoom the page on pinch
  • Verify existing desktop behaviour is unaffected (trackpadMode defaults to false)
  • npm test passes (654 tests)

🤖 Generated with Claude Code

Benjamin Jacobs and others added 6 commits June 7, 2026 21:29
noVNC maps touch input to absolute framebuffer coordinates, which is
awkward on phones/tablets where the finger covers the target. This adds
an opt-in trackpad mode that treats touch as a relative pointing device,
the way native mobile VNC clients do.

When enabled (RFB.trackpadMode, or the "Trackpad mode" UI setting):
- one finger moves a virtual cursor relatively (scaled by
  trackpadSensitivity), clamped to the framebuffer bounds
- a tap is a left click at the virtual cursor
- a two-finger tap is a right click
- a two-finger drag scrolls
- a tap immediately followed by a drag ("tap-and-a-half") drags with the
  left button held, and a long-press then drag does the same

The two-finger scroll and pinch-zoom step loops are factored out of
_handleGesture into _gestureTwoDragScroll/_gesturePinchZoom so both the
absolute and trackpad paths share them. Absolute-mode behaviour is
unchanged. Adds unit tests covering each trackpad gesture.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…zoom

Follow-ups to trackpad mode based on on-device testing:

- Always show a cursor in trackpad mode. The finger isn't on the pointer,
  so _shouldShowDotCursor now also honours trackpadMode (falls back to the
  dot when the server has no visible cursor). trackpadMode is now a
  getter/setter that refreshes the cursor and seeds the virtual cursor at
  the centre on enable, so you see it from the first movement.

- Pinch now magnifies the view locally (RDP-style) instead of sending
  Ctrl+wheel to the remote: a virtual zoom factor (1.0..5.0) drives
  clipViewport + display scaling, and while zoomed a two-finger drag pans
  the viewport. _updateScale keeps the zoom across resizes; leaving
  trackpad mode resets it. Pointer coordinates stay correct because absX/Y
  already account for scale and viewport offset.

- Block iOS Safari's own pinch-zoom (it ignores user-scalable=no) by
  preventing the proprietary gesturestart/gesturechange events, detected
  via their `scale` property so noVNC's synthetic gesture CustomEvents are
  left untouched.

Adds unit tests for local magnification and zoom clamping.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Diagonal two-finger drags were being misclassified as pinch (zoom) instead
of twodrag (pan): the gesture handler's ambiguous-case tiebreak compared
finger movement to inter-finger distance change per axis, so a diagonal
drag (movement split across both axes) lost to the distance noise even
when the fingers stayed parallel. Compare the 2D movement magnitude
(hypot) against the distance change instead, so diagonal parallel drags
are reliably treated as twodrag.

Also pan the viewport opposite the drag on both axes so the magnified
image follows the fingers ("drag the map").

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The previous local-zoom used noVNC's discrete pinch/twodrag classification
and clip-based scaling, which felt janky (per-frame framebuffer re-render),
mixed up pan vs zoom, and could barely zoom on diagonal/imperfect pinches.

Rework it the way the iOS RDP app behaves:

- Handle the two-finger case directly from raw touch events (touchstart/
  move/end) instead of noVNC's gesture classifier, so pan and zoom happen
  together and there's no pinch-vs-twodrag ambiguity. While two fingers are
  down, the gesture handler's two-finger output is ignored.
- Magnify with a CSS transform on the canvas (GPU-composited, smooth, no
  framebuffer re-render) instead of clip + display scaling. The transform
  is purely visual, so pointer coordinates still map correctly (the virtual
  cursor stays in untransformed canvas CSS pixels).
- Zoom is anchored at the midpoint between the fingers (zoom toward the
  fingers); the midpoint movement pans when magnified, or scrolls the
  remote (mouse wheel) when at fit. Pan is clamped to the content. Cursor
  speed is divided by the zoom so it tracks the finger 1:1 on screen.

Tests updated to drive raw TouchEvents for pinch/zoom/scroll and to verify
single-finger gestures are ignored mid-pinch.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… zone

One-finger cursor movement was driven by noVNC's gesture handler, which
only emits a drag after the finger crosses a ~50px (~1cm) recognition
threshold. The movement was applied, but only became visible after that
dead zone, so the first centimetre of dragging showed nothing.

Drive one-finger cursor movement directly from raw touch events instead,
so it tracks from the first pixel. The gesture handler keeps responsibility
for click semantics only (tap = left, two-finger tap = right, long-press /
tap-and-a-half = hold the left button); it no longer positions the cursor
in trackpad mode. Two-finger pan/zoom/scroll continue to be handled from
raw touch.

Tests updated to drive one-finger movement and clamping via raw TouchEvents.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
@tobfah

tobfah commented Jun 22, 2026

Copy link
Copy Markdown

Hi, thanks for your contribution. The first impression is that the modularization of this new trackpad feature is lacking -- see for instance how keyboard and cursor are integrated with rfb.

Benjamin Jacobs and others added 23 commits June 27, 2026 08:44
When magnified, pushing the virtual cursor into the viewport edge band now
pans the view continuously at a speed proportional to how far the cursor is
pushed past the edge (like the Microsoft RDP mobile client), revealing
off-screen content without a separate two-finger pan. Runs on its own
requestAnimationFrame loop; the cursor stays fixed on screen while its
logical framebuffer position moves onto the revealed content.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Confine the virtual cursor to the visible region and drive edge panning from
finger overshoot past that edge (accumulated while pushing), instead of a wide
margin band. Fixes panning starting well before the edge, and makes the pan
speed scale with how hard you keep pushing past the border (RDP-style); holding
past the edge keeps panning. Cursor rides the leading edge onto revealed content.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Edge-pan bounds now come from the on-screen viewport (_screen container) via
getBoundingClientRect instead of the canvas footprint, so the pan-trigger edges
follow the actual screen edges after zooming (the canvas is centred with margins
at fit, which put the edges in the middle). Pan speed gets a non-zero floor so
the slowest pan isn't sluggish.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…vershoot bleed)

The cursor jumped because edge bounds were read via getBoundingClientRect mid-
transform every frame, giving noisy values the cursor was snapped to. Cache the
canvas screen origin + viewport rect once (invariant to pan/zoom) and compute
bounds analytically. Overshoot now bleeds toward zero by inward finger motion
instead of hard-resetting when the auto-pan shifts the bounds, so panning is
continuous instead of stuttering on/off. Base invalidated on resize/gesture end.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Pan direction now follows the cursor's offset from the visible centre (speed
still from overshoot magnitude), so pushing into a corner or sliding along an
edge pans diagonally toward the cursor instead of snapping to pure horizontal/
vertical. Cursor is kept fixed on screen during the pan.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Handle the two-finger tap from raw touch events (a quick two-finger down/up with
no pinch/pan) so the right click lands on the virtual cursor and opens the remote
context menu there. The gesture-handler path was blocked by the multitouch guard
and could fire at the screen centre; remove its twotap case to avoid duplicates.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On iOS Safari the async clipboard auto-sync is unsupported, so noVNC falls back
to the manual clipboard panel. Add explicit buttons that call navigator.clipboard
read/write inside the click gesture (the only context Safari permits), bridging
the remote clipboard to/from the device clipboard with one tap.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Move the selection-drag button hold into the raw touch handlers: a one-finger
drag that starts shortly after a tap (double-tap-drag) now presses and holds the
left button from the first pixel of movement, so text selection grabs from the
start instead of after the ~50px gesture threshold. A plain drag just moves; a
single tap clicks (deselect). Longpress-drag still handled via the gesture path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Port the o365-bridge HiDPI patches: _screenSize requests the remote framebuffer
at CSS px * devicePixelRatio, and _rescale presents the device-pixel backing at
CSS size (/dpr) when factor===1. Only active in resize=remote mode (no-op under
resize=scale). noVNC 1.7.0 is otherwise not devicePixelRatio-aware.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A two-finger gesture now commits to either zoom OR move once it leaves a
small deadzone, and keeps that intent until the fingers lift. Previously
every frame applied both zoom (on finger-distance change) and pan/scroll
(on midpoint change), so a pinch also panned and a pan also zoomed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Zoom-intent now pans on midpoint movement within the same gesture, so the
user can pinch-and-drag without lifting. Move-intent still never zooms, so a
pure pan can't wobble the magnification.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants