Use the screenshot command to check for red pills - #852
Open
WillMartin wants to merge 12 commits into
Open
Conversation
… verification Add a desktop tool that screenshots the whole Tableau Desktop window (shelf chrome included) and returns the main-window PNG inline, so a red error pill or empty view that a command-level success receipt cannot see gets caught before the agent reports Done. This is the only capture that shows chrome — a viz-only render never does. It drives the monolith debug command tabui:take-all-screenshots (kebab wire name, verified against a live 2026.x build), whose TempFilePath out-param is the generated temp DIRECTORY of per-widget ScreenShot_<n>.png files; resolveImageFiles expands that directory and chooseMainWindowImage picks the largest file (the main window). Over the inline-image cap the bytes spill to a cache file, mirroring buildSheetImageToolResult. Wire it into the dynamic-authoring profile and the new verify-visible-state route, which treats RED anywhere as broken (shelf pill -> get-worksheet-xml; schema-viewer / Data-pane red -> list-available-fields / list-workbook- datasources). Re-pin the tools/list surface budgets and instruction length accordingly (all still well under the 46k auto-deferral cliff). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Scan the captured PNG server-side for Tableau's error-pill red and lead capture-window-screenshot's result with a one-line triage note, so an obvious red indicator is flagged before the model looks — the image still rides in the same result. It is a nudge to inspect, never a verdict. errorRedScan.ts decodes 8-bit RGB/RGBA PNGs with Node's built-in zlib (no image dependency) and buckets error-red pixels into a coarse grid. The signal is the densest cell's red fraction, not raw count: a solid pill saturates its cell while red scattered through a viz does not. Threshold 0.30 calibrated against a real capture whose lone error pill scored 0.47 at 0.15% total footprint. Color intentionally does NOT gate: the error red (#E84050) sits only ~26 away from Tableau's default categorical red (#E15759), so a hue tolerance loose enough to catch real pills would also catch common viz red. Location (shelf band vs canvas) and density carry the separation; hue does not. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…lled parser Replace the hand-written PNG decoder (IHDR/IDAT walk + Paeth unfiltering over Node zlib) with fast-png. decodePng is now a thin adapter that maps fast-png's output to DecodedImage and still returns null for layouts the scan does not handle (16-bit, palette). scanForErrorRed and its threshold are unchanged. fast-png is pure JS (MIT; deps fflate + iobuffer, both MIT, no native binary), so it bundles into the desktop single-executable — the reason the original avoided an image lib was native modules like sharp, not pure-JS ones. Same result on the real capture (densest cell 0.466) at ~20ms; drops ~90 lines of format code we should not own. Tests now build fixtures with fast-png's encoder instead of a hand-rolled PNG. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… report Post-apply verification now speaks one contract for all strategies, not just readback. A VerificationReport carries a single findings list; each finding is a source-tagged, pre-rendered VerificationFinding with optional evidence. This is the pipe the upcoming visual (red-pill) check writes into — it appends findings instead of adding a sibling `visualCheck` field for consumers to fan out over. Pure refactor, no behavior change: - ReadbackVerificationResult -> VerificationReport (+ optional findings channel) - ReadbackVerificationStatus -> VerificationStatus - add VerificationFinding / VerificationSource + readbackFindingsToVerification() - ReadbackFinding stays the readback verifier's rich internal type (promise-check sort-loss + formatters unchanged); publicVerificationReport() projects it onto the public report with findings tagged source:'readback' - migrate all consumers; full unit suite green (5795 tests) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…apply Wire a pluggable VISUAL verifier into the same post-apply verification channel readback uses, gated behind the new AUTO_VISUAL_CHECK config flag (off by default). On a cached-file worksheet apply, when the flag is on, capture the Desktop window and scan for a dense error-red cluster (a likely error pill / broken render that XML readback cannot see); a hit becomes a source-agnostic warning finding that nudges the model to capture-window-screenshot and inspect the shelves before reporting Done. The check is best-effort: an unavailable or undecodable capture yields no finding rather than failing the apply, and a hit is only ever a warning. It carries no pinned evidence handle yet — an error pill persists until fixed, so a live re-capture shows the same state; VerificationFinding.evidence is left for a later refinement that pins the exact apply-time pixels. captureMainWindowImage is factored out of the capture-window-screenshot tool callback so the visual check can capture mid-apply without the read harness. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… applies Fold the visual finding into the structured VerificationReport the artifact and template-plan apply modes already return, so all three apply-worksheet modes run the same off-by-default (AUTO_VISUAL_CHECK) rendered-window check through one verification channel — not just the cached-file mode. New withVerificationFinding() appends a finding to a report's findings list, escalates status/ok to reflect it (a warning lifts passed/skipped to warning, an error to failed; nothing downgrades a worse status), and surfaces its message so a consumer reading only status/message still sees what to look at. Applied at the same consumer layer as mode 3; verificationRan keys off the readback status alone so the receipt's "read back" claim stays honest when only the visual check fires. runVisualErrorCheck now catches internally so its "best-effort, never fails the apply" contract holds at every call site. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…k applies apply-dashboard and apply-workbook have no structural readback, so when AUTO_VISUAL_CHECK is on the rendered-window scan is their only post-apply signal: a dense error-red cluster (a likely error pill / broken element) rides the success message as a warning nudging the model to capture and look, never as a hard failure. New runVisualErrorCheckText() is the text-channel form of the check for applies with no VerificationReport to fold into — it renders a hit as the same warning text the worksheet text-mode apply uses. The shared acceptedNoReadbackApplyResult receipt gains an optional visualWarning appended after the host verification line. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
apply-storyboard shares the no-readback apply path with dashboard and workbook, so give it the same treatment: when AUTO_VISUAL_CHECK is on, a dense error-red cluster in the rendered window rides the success message as a warning nudging the model to capture and look, never as a hard failure. Completes visual-check coverage across every apply-worksheet/dashboard/workbook/storyboard entry point. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
bind-template and run-dashboard-batch apply through loadWorkbookXml, whose readback is XML-only and blind to a red error pill in the rendered window. Wire the visual check at the consumer/tool layer (loadWorkbookXml has no config access), matching the existing five apply sites: - bind-template: append runVisualErrorCheckText to the success guidance, riding the text exactly like readbackEvidence. It does NOT feed the needsFollowUp/done decision — a soft visual signal must never flip the terminal receipt that closes the re-bind loop. - run-dashboard-batch: widen the dashboard applied step's verification to a full VerificationReport and fold a visual finding via withVerificationFinding, as a warning — never a hard failure, never flips `applied`. Off by default (AUTO_VISUAL_CHECK=true); flag-off path is byte-identical. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… fired The visual error check was fully silent: a clean window appends nothing, so its absence from an apply's output told you nothing about whether it ran, whether AUTO_VISUAL_CHECK was on, or what the scan measured. Add one log line at the runVisualErrorCheck chokepoint (all sites call it), under logger "visualCheck": - info on a clean scan (visible at default verbosity once the flag is on), with the scan metrics behind the verdict - warning on a flagged dense cluster - debug when the capture is unavailable/undecodable (expected/transient) - warning when the scan throws (still no finding — best-effort contract intact) Behavior is otherwise unchanged; the finding/return contract is identical. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tector The post-apply visual check now defaults ON (AUTO_VISUAL_CHECK=false is the kill-switch), and its red field-pill detector keys on SHAPE, not density. The old density scan (max red fraction over a fixed cell grid) gave a false negative on a real pill: a 296x40 pill straddling cells diluted below the 0.30 threshold. The new detector groups error-red pixels into 8-connected components and matches the pill's invariant capsule profile — a flat plateau plus at least one rounded, convex end-cap — expressed entirely as ratios of the plateau height, so it is invariant to capture DPI and to how long the pill is. A square-cornered bar, a round mark, or scattered red does not match. - config.desktop.ts: autoVisualCheck defaults true; only AUTO_VISUAL_CHECK=false disables it. - errorRedScan.ts: rewrite from density to connected-component cap-profile matching; redPixels now reports total error-red (context only), pillFound is the trigger. - visualErrorCheck.ts / captureWindowScreenshot.ts: pill-shape messaging and logging fields. - bindTemplate.test.ts: bump executeCommand counts for the added post-apply window capture. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ctor Add two fixtures cropped at native resolution from live Tableau Desktop captures, plus two synthetic edge cases, and refresh stale density-era wording in the verification-formatting tests. - real-window-red-pill.png: the exact window the OLD density scan scored as "no dense red cluster (29% red)" — a false negative. The cap-profile detector must flag its pill; this pins the fix to real anti-aliased pixels. - real-window-clean.png: a genuinely clean capture (real chrome, gridlines, valid non-red pills) — guards false positives the synthetic negatives can't. - caps=1: a pill clipped at the image edge, exercising the caps<1 gate. - thin red reference line: the "red is overloaded" false-positive case. - readback-verify.test.ts: sample visual-warning messages now say "a red field pill shape" instead of the removed density wording. Downscaling was rejected as a fixture format: it corrupted the signal under test (a real pill dropped to plateau height 9, caps 2->1, and a second pill was erased entirely), so both fixtures are cropped at native resolution. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In an ideal world all warnings in Tableau would be exposed from the Client API. That is what we need to strive for. However, we accept a lot of errors with only visual feedback given the visual roots of the product. This uses that fact today to heuristically check for if a red pill is on the shelves (or data pane) after application of any changes. If it's there we warn in our tool responses. We also add a command that can be used to do the screenshot itself for that agent to debug.
A downside of the approach is that the command (via Qt) actually uses accessibility tooling to take the window screenshot. On my mac that required additional perms being given. I think that's a reasonable trade off for now.
I've tested this locally but not as thoroughly as I'd like to. If I weren't going on vacation I'd try and do some more but putting it out for review.