Skip to content

Fix show_neighborhoods tool call reliability - #30

Open
AndrewSepic wants to merge 2 commits into
cw/demo-nyc-neighborhood-finderfrom
as/fix-show-neighborhoods-reliability
Open

Fix show_neighborhoods tool call reliability#30
AndrewSepic wants to merge 2 commits into
cw/demo-nyc-neighborhood-finderfrom
as/fix-show-neighborhoods-reliability

Conversation

@AndrewSepic

Copy link
Copy Markdown
Contributor

Summary

While testing this demo, the agent would frequently write full neighborhood recommendations in chat text but call show_neighborhoods with an empty or missing neighborhoods array, so nothing ever rendered on the map. Diagnostic logging showed the model duplicating its answer (full prose, then a redundant tool call) and getting no feedback when that second call came back empty.

  • Move the show_neighborhoods instruction to the top of the system prompt, apply it to every follow-up turn, and tell the model to keep chat text brief and put recommendation detail only in the tool call's reason field instead of duplicating it in prose.
  • Have execute() return a corrective error (asking the model to retry) instead of a canned success message when neighborhoods is empty or invalid.
  • Add logging (raw tool args, per-step tool inputs, and a client-side warning on unexpected tool-part states) so failures are visible instead of silent.
  • Fix two pre-existing eslint config gaps surfaced while linting these files: add a node env override for server.js (the first Node/Express file in this repo) so process.env isn't flagged as undefined, and turn off react/prop-types for .ts/.tsx files since this codebase uses TypeScript types instead of PropTypes.

Test plan

  • Ran the app locally and confirmed the agent reliably calls show_neighborhoods with non-empty data on first ask and on follow-up refinements
  • npx eslint clean on the modified files
  • Chris to review/merge into cw/demo-nyc-neighborhood-finder

🤖 Generated with Claude Code

The agent frequently wrote full neighborhood recommendations in chat
text but called show_neighborhoods with an empty or missing
neighborhoods array, so nothing rendered on the map. Diagnostic
logging showed the model duplicating its answer (full prose, then a
redundant tool call) and getting no feedback when that second call
came back empty.

- Move the show_neighborhoods instruction to the top of the system
  prompt, make it apply to every follow-up turn, and tell the model to
  keep chat text brief and put recommendation detail only in the tool
  call's reason field instead of duplicating it in prose.
- Have execute() return a corrective error (asking the model to retry)
  instead of a canned success message when neighborhoods is empty or
  invalid, instead of silently reporting success.
- Add logging (raw tool args, per-step tool inputs, and a client-side
  warning on unexpected tool-part states) to make failures visible
  instead of silent.
- Fix two pre-existing eslint config gaps surfaced by linting these
  files: add a node env override for server.js (the first Node/Express
  file in this repo) so process.env isn't flagged as undefined, and
  turn off react/prop-types for .ts/.tsx files since this codebase
  uses TypeScript types instead of PropTypes.
@AndrewSepic
AndrewSepic requested a review from a team as a code owner August 24, 2026 18:49
Now that the show_neighborhoods-specific logging (raw args and
validation result inside execute()) has already surfaced the actual
failure, the blanket per-tool-call JSON dump in onStepFinish is
redundant and would clutter server logs with large payloads from the
Mapbox tools during normal use.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant