Skip to content

[Feature]: make token count accessible without loading the model #584

@joein

Description

@joein

What feature would you like to request?

The new token_count method currently requires a model to be loaded before using it.
However, the better approach might be to allow loading just a tokenizer without loading a model

Is there any additional information you would like to provide?

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions