Skip to content

[Installation] lmms-eval --tasks list returns empty list after pip install #1215

@YirongWho

Description

@YirongWho

Checklist

  • I have searched for similar issues before opening this one.
  • I am using the latest version of lmms-eval.

Bug Description

Describe the bug
After installing lmms-eval via pip install lmms-eval==0.6.1, running the command lmms-eval --tasks list returns an empty list for available tasks.

Root Cause Analysis
I investigated the installation directory in site-packages/lmms_eval/tasks and found that no .yaml configuration files were included in the pip package. It seems the resource files (YAML) are missing from the PyPI distribution, causing the TaskManager to find zero tasks.

Questions
Is this a known packaging issue where .yaml files are excluded from the wheel file?

Is it mandatory to install from source (pip install -e .) to get the task configurations?

If I want to keep the pip installation, is there a recommended way to sync the tasks/ directory separately?

Steps to Reproduce

1. pip install lmms-eval==0.6.1
2. lmms-eval --tasks list

Error Message / Traceback

Environment

  • Python 3.11.14
  • lmms-eval 0.6.1

Additional Context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions