Skip to content

Feat/max error rate #171

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 18 commits into
base: main
Choose a base branch
from
Draft

Feat/max error rate #171

wants to merge 18 commits into from

Conversation

markVaykhansky
Copy link
Collaborator

@markVaykhansky markVaykhansky commented May 21, 2025

Addressing this issue:
#105

--max-error-rate flag was added.
It is only applicable in either or the the following scenarios:

  1. Fixed length dataset
  2. max_requests is set
  3. rate_type is constant and max_duration is set
  • If error rate is not applicable (i.e non of the above scenarios) we just ignore it (throwing an error would break sweep for example)
  • Implemented using multiprocessing.Event which is used to trigger a shutdown once we hit max_error_rate.
  • max_error_rate - added to report
  • error_rate in the final report is calculated as total errored / total requests finalized i.e we exclude incomplete requests

@markVaykhansky markVaykhansky self-assigned this May 21, 2025

📦 Build Artifacts Available
The build artifacts (.whl and .tar.gz) have been successfully generated and are available for download: https://github.com/neuralmagic/guidellm/actions/runs/15167220831/artifacts/3170182036.
They will be retained for up to 30 days.

📦 Build Artifacts Available
The build artifacts (.whl and .tar.gz) have been successfully generated and are available for download: https://github.com/neuralmagic/guidellm/actions/runs/15167280131/artifacts/3170205369.
They will be retained for up to 30 days.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant