Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Feature: add a qna evaluation for post training checkpoint evals #474

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

nerdalert
Copy link
Member

@nerdalert nerdalert commented Jan 15, 2025

  • Give the user the ability to evaluate their QnA from the dashboard against a checkpoint on the server. This requires the same API server being used in the end to end fine-tuning demo.
  • Add endpoints for a checkpoint listing and the execution of a qna eval.
  • The backend go api server will call a container that loads the requested checkpoint and qna.yaml into vLLM and queries the checkpoint with the seed questions from the QnA to get a sense of the fidelity of the checkpoint from the orignal seed examples.

Tasks:

  • Remove hardcodes for qna and checkpoint paths via ENVs.
  • Add the option to side by side the qna and the eval results for easy comps.

Quick snippet to give a general idea. Will change with the above tasks.

image

@nerdalert nerdalert marked this pull request as draft January 15, 2025 07:44
@nerdalert nerdalert added demo PR that contains Demo related changes enhancement New feature or request labels Jan 15, 2025
- Add endpoints for a checkpoint listing and the execution of
a qna eval.
- The backend go api will call a container that loads the
requested checkpoint and qna.yaml into vLLM and queries the
checkpoint with the seed questions from the QnA to get a sense
of the fidelity of the checkpoint from the orignal seed examples.

Signed-off-by: Brent Salisbury <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
demo PR that contains Demo related changes enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant