Fix show_neighborhoods tool call reliability - #30
Open
AndrewSepic wants to merge 2 commits into
Open
Conversation
The agent frequently wrote full neighborhood recommendations in chat text but called show_neighborhoods with an empty or missing neighborhoods array, so nothing rendered on the map. Diagnostic logging showed the model duplicating its answer (full prose, then a redundant tool call) and getting no feedback when that second call came back empty. - Move the show_neighborhoods instruction to the top of the system prompt, make it apply to every follow-up turn, and tell the model to keep chat text brief and put recommendation detail only in the tool call's reason field instead of duplicating it in prose. - Have execute() return a corrective error (asking the model to retry) instead of a canned success message when neighborhoods is empty or invalid, instead of silently reporting success. - Add logging (raw tool args, per-step tool inputs, and a client-side warning on unexpected tool-part states) to make failures visible instead of silent. - Fix two pre-existing eslint config gaps surfaced by linting these files: add a node env override for server.js (the first Node/Express file in this repo) so process.env isn't flagged as undefined, and turn off react/prop-types for .ts/.tsx files since this codebase uses TypeScript types instead of PropTypes.
Now that the show_neighborhoods-specific logging (raw args and validation result inside execute()) has already surfaced the actual failure, the blanket per-tool-call JSON dump in onStepFinish is redundant and would clutter server logs with large payloads from the Mapbox tools during normal use.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
While testing this demo, the agent would frequently write full neighborhood recommendations in chat text but call
show_neighborhoodswith an empty or missingneighborhoodsarray, so nothing ever rendered on the map. Diagnostic logging showed the model duplicating its answer (full prose, then a redundant tool call) and getting no feedback when that second call came back empty.show_neighborhoodsinstruction to the top of the system prompt, apply it to every follow-up turn, and tell the model to keep chat text brief and put recommendation detail only in the tool call'sreasonfield instead of duplicating it in prose.execute()return a corrective error (asking the model to retry) instead of a canned success message whenneighborhoodsis empty or invalid.server.js(the first Node/Express file in this repo) soprocess.envisn't flagged as undefined, and turn offreact/prop-typesfor.ts/.tsxfiles since this codebase uses TypeScript types instead of PropTypes.Test plan
show_neighborhoodswith non-empty data on first ask and on follow-up refinementsnpx eslintclean on the modified filescw/demo-nyc-neighborhood-finder🤖 Generated with Claude Code