Skip to content

fix: correct double swap of parameters in AnswerAccuracy metric templ… #2007

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

DmitrySirakov
Copy link

Overview

This PR addresses the double swap bug in the AnswerAccuracy metric implementation. In the original implementation, both the meta-labels (User Answer / Reference Answer) and their corresponding values (sample.response and sample.reference) were swapped, leading to an unintended behavior. According to the documentation, only the labels should be swapped while the values must remain in the original order.

Changes Made

  • Modified the second prompt generation for AnswerAccuracy to maintain the original order of values, while swapping only the meta-labels.
  • Ensured consistency with the documentation by keeping sentence_inference as sample.response and sentence_true as sample.reference in both prompts.

Issue Reference

This fixes issue #2000.

Additional Notes

Please review the changes and let me know if further modifications are needed.

@dosubot dosubot bot added the size:XS This PR changes 0-9 lines, ignoring generated files. label Apr 13, 2025
@sahusiddharth sahusiddharth self-requested a review April 14, 2025 03:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
size:XS This PR changes 0-9 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Nvidia Accuracy calculation code is not consistent with the docs
1 participant