Skip to content

Add VSI-SUPER benchmark#1267

Merged
Luodian merged 6 commits intoEvolvingLMMs-Lab:mainfrom
akawincent:task/vsisuper
Mar 26, 2026
Merged

Add VSI-SUPER benchmark#1267
Luodian merged 6 commits intoEvolvingLMMs-Lab:mainfrom
akawincent:task/vsisuper

Conversation

@akawincent
Copy link
Copy Markdown
Contributor

@akawincent akawincent commented Mar 25, 2026

Summary

  • add VSI-SUPER Recall task configs and scoring utilities
  • add VSI-SUPER Count task configs and scoring utilities
  • add VSI-SUPER Count Streaming task configs and scoring utilities

Dataset pages:

In scope

  • add vsisuper/recall task configs and vsr_utils.py
  • add vsisuper/count task configs and vsc_utils.py
  • add vsisuper/count_streaming task configs and vsc_streaming_utils.py

Out of scope

  • model integrations
  • dataset content changes
  • changes outside the VSI-SUPER task family

Validation

  • pre-commit run --files lmms_eval/tasks/vsisuper/count/vsc_utils.py lmms_eval/tasks/vsisuper/count_streaming/vsc_streaming_utils.py lmms_eval/tasks/vsisuper/recall/vsr_utils.py | result: pass

Risk / Compatibility

  • low risk: changes are limited to new task configs and task-local utilities
  • no intended behavior changes outside the new VSI-SUPER tasks

Type of Change

  • Bug fix (non-breaking change)
  • New feature
  • New benchmark/task
  • New model integration
  • Breaking change
  • Documentation update
  • Refactoring (no functional changes)

@akawincent akawincent changed the title Add VSI-SUPER count and recall tasks with streaming count variants Add VSI-SUPER benchmark Mar 25, 2026
Copy link
Copy Markdown
Contributor

@Luodian Luodian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — clean additive benchmark. Task configs and utilities are well-structured across recall, count, and count_streaming variants.

@Luodian Luodian merged commit d927265 into EvolvingLMMs-Lab:main Mar 26, 2026
3 checks passed
@akawincent akawincent deleted the task/vsisuper branch March 27, 2026 08:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants