Skip to content

fix(charxiv): lazy-init OpenAI client and make model version configur…#1252

Merged
Luodian merged 1 commit intoEvolvingLMMs-Lab:mainfrom
MaxwellJryao:fix/charxiv-lazy-client-configurable-model
Mar 15, 2026
Merged

fix(charxiv): lazy-init OpenAI client and make model version configur…#1252
Luodian merged 1 commit intoEvolvingLMMs-Lab:mainfrom
MaxwellJryao:fix/charxiv-lazy-client-configurable-model

Conversation

@MaxwellJryao
Copy link
Copy Markdown
Contributor

@MaxwellJryao MaxwellJryao commented Mar 13, 2026

Summary

  • Lazy-initialize OpenAI client to avoid SSLContext pickling errors in datasets.map() multiprocessing
  • Make model version configurable via MODEL_VERSION env var for both descriptive and reasoning grading

In scope

  • lmms_eval/tasks/charxiv/utils.py: replace module-level client with _get_client() lazy initializer; remove num_proc=1 from dataset.map()
  • lmms_eval/tasks/charxiv/reasoning_utils.py: add model parameter to get_reasoning_result_gpt()

Out of scope

  • No changes to grading logic, prompts, or evaluation metrics
  • No changes to other tasks or model integrations

Validation

Risk / Compatibility

  • Low risk: lazy init is functionally equivalent; default model value preserves existing behavior
  • MODEL_VERSION env var now takes effect for reasoning grading (previously ignored)

Type of Change

  • Bug fix (non-breaking change)
  • New feature
  • New benchmark/task
  • New model integration
  • Breaking change
  • Documentation update
  • Refactoring (no functional changes)

…able

- Replace module-level OpenAI client with lazy _get_client() to avoid
  SSLContext pickling errors in datasets.map() multiprocessing
- Remove num_proc=1 from dataset.map() (no longer needed)
- Add model parameter to get_reasoning_result_gpt() so MODEL_VERSION
  env var is respected for both descriptive and reasoning grading
@Luodian Luodian merged commit a130a0c into EvolvingLMMs-Lab:main Mar 15, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants