Simplified Tool Call Accuracy V1 #40710
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
the
Evaluation
There was a problem hiding this comment.
Pull Request Overview
This PR introduces a simplified mechanism for predicting and evaluating tool call accuracy by adding a new tool call predictor module and a corresponding evaluator.
- Added the predict_tools function in _tool_call_predictor.py to generate ground truth based on a prompt model.
- Exposed predict_tools via an init.py file for easier consumption.
- Included a new evaluator in _tool_accuracy_new.py that implements scoring logic for tool calls and tool results.
Reviewed Changes
Copilot reviewed 3 out of 4 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_tool_call_accuracy/_tool_calls_predictor/_tool_call_predictor.py | Implements the tool call predictor using prompt models (contains duplicate imports that could lead to ambiguity). |
| sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_tool_call_accuracy/_tool_calls_predictor/init.py | Exposes the predict_tools function for the caller. |
| sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_tool_call_accuracy/_tool_call_accuracy_new.py | Introduces new evaluation logic for tool call accuracy including extraction and match scoring. |
Files not reviewed (1)
- sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_tool_call_accuracy/_tool_calls_predictor/tool_call_predictor.prompty: Language not supported
Comments suppressed due to low confidence (2)
sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_tool_call_accuracy/_tool_calls_predictor/_tool_call_predictor.py:29
- Duplicate import of AsyncPrompty detected (also imported on line 10). Consider consolidating the imports to a single source to avoid ambiguity.
from promptflow.core import AsyncPrompty
sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_tool_call_accuracy/_tool_call_accuracy_new.py:151
- The function 'generate_ground_truth' is referenced but not defined or imported; please add its definition or import it.
ground_truth = generate_ground_truth(query, response, tool_definitions)
|
API change check API changes are not detected in this pull request. |
|
@ghyadav Can you please share context on requirements for a new tool call accuracy evaluator ? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Please add an informative description that covers that changes made by the pull request and link all relevant issues.
If an SDK is being regenerated based on a new swagger spec, a link to the pull request containing these swagger spec changes has been included above.
All SDK Contribution checklist:
General Guidelines and Best Practices
Testing Guidelines