Add trackpad mode for relative pointer input on touch devices#2065
Open
benjaminfrombe wants to merge 29 commits into
Open
Add trackpad mode for relative pointer input on touch devices#2065benjaminfrombe wants to merge 29 commits into
benjaminfrombe wants to merge 29 commits into
Conversation
noVNC maps touch input to absolute framebuffer coordinates, which is
awkward on phones/tablets where the finger covers the target. This adds
an opt-in trackpad mode that treats touch as a relative pointing device,
the way native mobile VNC clients do.
When enabled (RFB.trackpadMode, or the "Trackpad mode" UI setting):
- one finger moves a virtual cursor relatively (scaled by
trackpadSensitivity), clamped to the framebuffer bounds
- a tap is a left click at the virtual cursor
- a two-finger tap is a right click
- a two-finger drag scrolls
- a tap immediately followed by a drag ("tap-and-a-half") drags with the
left button held, and a long-press then drag does the same
The two-finger scroll and pinch-zoom step loops are factored out of
_handleGesture into _gestureTwoDragScroll/_gesturePinchZoom so both the
absolute and trackpad paths share them. Absolute-mode behaviour is
unchanged. Adds unit tests covering each trackpad gesture.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…zoom Follow-ups to trackpad mode based on on-device testing: - Always show a cursor in trackpad mode. The finger isn't on the pointer, so _shouldShowDotCursor now also honours trackpadMode (falls back to the dot when the server has no visible cursor). trackpadMode is now a getter/setter that refreshes the cursor and seeds the virtual cursor at the centre on enable, so you see it from the first movement. - Pinch now magnifies the view locally (RDP-style) instead of sending Ctrl+wheel to the remote: a virtual zoom factor (1.0..5.0) drives clipViewport + display scaling, and while zoomed a two-finger drag pans the viewport. _updateScale keeps the zoom across resizes; leaving trackpad mode resets it. Pointer coordinates stay correct because absX/Y already account for scale and viewport offset. - Block iOS Safari's own pinch-zoom (it ignores user-scalable=no) by preventing the proprietary gesturestart/gesturechange events, detected via their `scale` property so noVNC's synthetic gesture CustomEvents are left untouched. Adds unit tests for local magnification and zoom clamping. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Diagonal two-finger drags were being misclassified as pinch (zoom) instead
of twodrag (pan): the gesture handler's ambiguous-case tiebreak compared
finger movement to inter-finger distance change per axis, so a diagonal
drag (movement split across both axes) lost to the distance noise even
when the fingers stayed parallel. Compare the 2D movement magnitude
(hypot) against the distance change instead, so diagonal parallel drags
are reliably treated as twodrag.
Also pan the viewport opposite the drag on both axes so the magnified
image follows the fingers ("drag the map").
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The previous local-zoom used noVNC's discrete pinch/twodrag classification and clip-based scaling, which felt janky (per-frame framebuffer re-render), mixed up pan vs zoom, and could barely zoom on diagonal/imperfect pinches. Rework it the way the iOS RDP app behaves: - Handle the two-finger case directly from raw touch events (touchstart/ move/end) instead of noVNC's gesture classifier, so pan and zoom happen together and there's no pinch-vs-twodrag ambiguity. While two fingers are down, the gesture handler's two-finger output is ignored. - Magnify with a CSS transform on the canvas (GPU-composited, smooth, no framebuffer re-render) instead of clip + display scaling. The transform is purely visual, so pointer coordinates still map correctly (the virtual cursor stays in untransformed canvas CSS pixels). - Zoom is anchored at the midpoint between the fingers (zoom toward the fingers); the midpoint movement pans when magnified, or scrolls the remote (mouse wheel) when at fit. Pan is clamped to the content. Cursor speed is divided by the zoom so it tracks the finger 1:1 on screen. Tests updated to drive raw TouchEvents for pinch/zoom/scroll and to verify single-finger gestures are ignored mid-pinch. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… zone One-finger cursor movement was driven by noVNC's gesture handler, which only emits a drag after the finger crosses a ~50px (~1cm) recognition threshold. The movement was applied, but only became visible after that dead zone, so the first centimetre of dragging showed nothing. Drive one-finger cursor movement directly from raw touch events instead, so it tracks from the first pixel. The gesture handler keeps responsibility for click semantics only (tap = left, two-finger tap = right, long-press / tap-and-a-half = hold the left button); it no longer positions the cursor in trackpad mode. Two-finger pan/zoom/scroll continue to be handled from raw touch. Tests updated to drive one-finger movement and clamping via raw TouchEvents. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
|
Hi, thanks for your contribution. The first impression is that the modularization of this new trackpad feature is lacking -- see for instance how keyboard and cursor are integrated with rfb. |
When magnified, pushing the virtual cursor into the viewport edge band now pans the view continuously at a speed proportional to how far the cursor is pushed past the edge (like the Microsoft RDP mobile client), revealing off-screen content without a separate two-finger pan. Runs on its own requestAnimationFrame loop; the cursor stays fixed on screen while its logical framebuffer position moves onto the revealed content. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Confine the virtual cursor to the visible region and drive edge panning from finger overshoot past that edge (accumulated while pushing), instead of a wide margin band. Fixes panning starting well before the edge, and makes the pan speed scale with how hard you keep pushing past the border (RDP-style); holding past the edge keeps panning. Cursor rides the leading edge onto revealed content. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Edge-pan bounds now come from the on-screen viewport (_screen container) via getBoundingClientRect instead of the canvas footprint, so the pan-trigger edges follow the actual screen edges after zooming (the canvas is centred with margins at fit, which put the edges in the middle). Pan speed gets a non-zero floor so the slowest pan isn't sluggish. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…vershoot bleed) The cursor jumped because edge bounds were read via getBoundingClientRect mid- transform every frame, giving noisy values the cursor was snapped to. Cache the canvas screen origin + viewport rect once (invariant to pan/zoom) and compute bounds analytically. Overshoot now bleeds toward zero by inward finger motion instead of hard-resetting when the auto-pan shifts the bounds, so panning is continuous instead of stuttering on/off. Base invalidated on resize/gesture end. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Pan direction now follows the cursor's offset from the visible centre (speed still from overshoot magnitude), so pushing into a corner or sliding along an edge pans diagonally toward the cursor instead of snapping to pure horizontal/ vertical. Cursor is kept fixed on screen during the pan. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Handle the two-finger tap from raw touch events (a quick two-finger down/up with no pinch/pan) so the right click lands on the virtual cursor and opens the remote context menu there. The gesture-handler path was blocked by the multitouch guard and could fire at the screen centre; remove its twotap case to avoid duplicates. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On iOS Safari the async clipboard auto-sync is unsupported, so noVNC falls back to the manual clipboard panel. Add explicit buttons that call navigator.clipboard read/write inside the click gesture (the only context Safari permits), bridging the remote clipboard to/from the device clipboard with one tap. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Move the selection-drag button hold into the raw touch handlers: a one-finger drag that starts shortly after a tap (double-tap-drag) now presses and holds the left button from the first pixel of movement, so text selection grabs from the start instead of after the ~50px gesture threshold. A plain drag just moves; a single tap clicks (deselect). Longpress-drag still handled via the gesture path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Port the o365-bridge HiDPI patches: _screenSize requests the remote framebuffer at CSS px * devicePixelRatio, and _rescale presents the device-pixel backing at CSS size (/dpr) when factor===1. Only active in resize=remote mode (no-op under resize=scale). noVNC 1.7.0 is otherwise not devicePixelRatio-aware. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A two-finger gesture now commits to either zoom OR move once it leaves a small deadzone, and keeps that intent until the fingers lift. Previously every frame applied both zoom (on finger-distance change) and pan/scroll (on midpoint change), so a pinch also panned and a pan also zoomed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Zoom-intent now pans on midpoint movement within the same gesture, so the user can pinch-and-drag without lifting. Move-intent still never zooms, so a pure pan can't wobble the magnification. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
noVNC maps touch input to absolute framebuffer coordinates, which is
awkward on phones/tablets: your finger covers the target, you can't
reach screen edges easily, and there is no way to zoom/pan locally.
This PR adds an opt-in trackpad mode that behaves like the iOS RDP
app or a laptop trackpad: one finger moves a virtual cursor relatively,
pinch zooms the view locally (without sending anything to the remote),
and two-finger drag either pans the magnified view or scrolls the remote.
What it does
Toggle via
RFB.trackpadMode = trueor the new "Trackpad mode (touch)"checkbox in the settings panel.
One finger
trackpadSensitivity,default 1.5) — driven from raw
touchmoveevents so there is no deadzone before movement is visible
button held
Two fingers
(GPU-composited, smooth, the remote is never touched)
Cursor visibility
when the server sends no visible cursor), so you can always see where
the virtual cursor is
iOS Safari
user-scalable=nosince iOS 10) by preventing the proprietary
gesturestart/gesturechangeevents (detected via their
.scaleproperty, so noVNC's own syntheticgesture
CustomEvents are unaffected)Bug fix included
gesturehandler.js_twoTouchTimeout: the existing code chose betweenPINCH and TWODRAG by comparing finger movement per axis against the
inter-finger distance change. A diagonal drag splits its movement across
both axes (~70% each), so it would lose to distance noise and be
misclassified as PINCH even when the fingers stayed parallel. Fixed by
comparing the 2D movement magnitude (
hypot) instead.Implementation notes
touchstart/ move/endhandlers that run alongside the gesture handler. The gesturehandler retains responsibility for tap classification only.
translate()+scale(), origin 0 0) is purely visual;it does not affect the canvas layout size, so
absX/absYstill mappointer coordinates correctly to framebuffer pixels.
_trackpadUpdateCursormaps it through the bounding rect's effectivescale so the dot follows the logical position even while magnified.
trackpadModeandtrackpadSensitivityare public properties onRFB.Leaving trackpad mode resets the CSS transform.
Test plan
npm testpasses (654 tests)🤖 Generated with Claude Code