[BUG] Mistral-Small-24B-Instruct-2501 - Tensor Parallel outputs garbled text.

[mindkrypted]

OS

Linux

GPU Library

CUDA 12.x

Python version

3.10

Pytorch version

2.5.1

Model

No response

Describe the bug

Running inference using Tensor Parallel will output garbled text.
Tried with two different quanted models:

• matatonic/Mistral-Small-24B-Instruct-2501-6.5bpw-h8-exl2
• MikeRoz/mistralai_Mistral-Small-24B-Instruct-2501-8.0bpw-h8-exl2

Reproduction steps

Example

examples/inference.py:

Loading: /home/user/models/Mistral-Small-24B-Instruct-2501-6.5bpw-h8-exl2 ━━━━━━━━━━━━━━━━ 100% 0:00:08 0:00:00
Loading tokenizer...
-----------------------------------------------------------------------------------
- Single completion
-----------------------------------------------------------------------------------
Once upon a time, in a land not so far away, there was a small village. In this village, there were a few families who were known by all the villagers as being very generous people. They always helped each other, and they were known for their kindness and compassion towards everyone they met.

One day, one of the families decided to have a big celebration. They invited all the villagers to come and join them for a feast. The family had been saving up for months, and they wanted to show their generosity by providing a wonderful meal for everyone.

(remaining output is as clean)

examples/inference_tp.py:

Loading: /home/user/models/Mistral-Small-24B-Instruct-2501-6.5bpw-h8-exl2 ━━━━━━━━━━━━━━━━ 100% 0:00:09 0:00:00
Loading tokenizer...
-----------------------------------------------------------------------------------
- Single completion
-----------------------------------------------------------------------------------
Our story begins in the Scottish town of�.

���������



�.

����

 [��'.��.

�./ . [ as..,

.\ .

 [ etc।

,

(continues like this until context is filled)

Expected behavior

Tensor Parallel should be working like the other models~

While this model is sufficiently small, 8bpw 8h with full context doesn't fit in 24GB VRAM

Logs

Installed packages

pip list

Package                                  Version                              Editable project location
---------------------------------------- ------------------------------------ -----------------------------------------------------------
absl-py                                  2.1.0
accelerate                               1.3.0
accelerated-scan                         0.2.0
addict                                   2.4.0
aenum                                    3.1.15
aiocache                                 0.12.3
aiofiles                                 23.2.1
aiohappyeyeballs                         2.4.0
aiohttp                                  3.11.10
aiohttp-cors                             0.7.0
aiosignal                                1.3.1
aiostream                                0.5.2
airportsdata                             20241001
alabaster                                1.0.0
albucore                                 0.0.19
albumentations                           1.4.20
alembic                                  1.13.2
altair                                   5.4.1
aniso8601                                9.0.1
annotated-types                          0.7.0
anthropic                                0.39.0
antlr4-python3-runtime                   4.9.3
anyascii                                 0.3.2
anyio                                    3.7.1
APScheduler                              3.10.4
arabic-reshaper                          3.0.0
archspec                                 0.2.3
argon2-cffi                              23.1.0
argon2-cffi-bindings                     21.2.0
asciitree                                0.3.3
asgiref                                  3.8.1
asn1crypto                               1.5.1
asteroid-filterbanks                     0.4.0
astor                                    0.8.1
asttokens                                2.4.1
astunparse                               1.6.3
async-lru                                2.0.4
async-timeout                            4.0.3
attrdict                                 2.0.1
attrs                                    24.2.0
audioread                                3.0.1
audiosr                                  0.0.7
Authlib                                  1.3.2
auto_gptq                                0.7.1
av                                       12.3.0
babel                                    2.16.0
backoff                                  2.2.1
bangla                                   0.0.2
bcrypt                                   4.2.0
beautifulsoup4                           4.12.3
bibtexparser                             2.0.0b8
bidict                                   0.23.1
bitarray                                 3.0.0
bitsandbytes                             0.43.3
black                                    24.8.0
blake3                                   1.0.1
blendmodes                               2022
blinker                                  1.9.0
blis                                     0.7.11
bnnumerizer                              0.0.2
bnunicodenormalizer                      0.1.7
bokeh                                    1.4.0
boltons                                  24.0.0
boto3                                    1.35.90
botocore                                 1.35.90
braceexpand                              0.1.7
Brotli                                   1.1.0
build                                    1.2.2.post1
CacheControl                             0.14.1
cachetools                               5.5.0
catalogue                                2.0.10
cbor                                     1.0.0
ccimport                                 0.4.4
cdifflib                                 1.2.6
certifi                                  2024.12.14
cffi                                     1.17.0
chardet                                  5.2.0
charset-normalizer                       3.3.2
chroma-hnswlib                           0.7.6
chromadb                                 0.5.15
clean-fid                                0.1.35
cleo                                     2.1.0
click                                    8.1.7
clip                                     1.0
clip-interrogator                        0.6.0
clldutils                                3.24.0
cloudpathlib                             0.16.0
cloudpickle                              3.1.0
cmake                                    3.31.2
colbert-ai                               0.2.21
colorama                                 0.4.6
colorclass                               2.2.2
coloredlogs                              15.0.1
colorful                                 0.5.6
colorlog                                 6.8.2
comm                                     0.2.2
compressed_rtf                           1.0.6
compressed-tensors                       0.8.1
compressed-tensors-nightly               0.8.0.20241121
conda                                    24.11.3
conda-libmamba-solver                    24.11.1
conda-package-handling                   2.4.0
conda_package_streaming                  0.11.0
confection                               0.1.5
contourpy                                1.3.0
coqpit                                   0.0.17
coqpit-config                            0.1.2
coqui-tts                                0.25.1
coqui-tts-trainer                        0.2.0
cramjam                                  2.9.0
crashtest                                0.4.1
cryptography                             43.0.0
cssselect2                               0.7.0
csvw                                     3.5.1
ctranslate2                              4.5.0
cumm-cu120                               0.4.11
cutlet                                   0.4.0
cycler                                   0.12.1
cymem                                    2.0.8
Cython                                   3.0.11
cytoolz                                  0.12.3
dask                                     2024.11.1
dataclasses-json                         0.6.7
datasets                                 2.21.0
dateparser                               1.1.8
DateTime                                 5.5
decorator                                5.1.1
decord                                   0.6.0
deepdiff                                 8.0.1
deepspeed                                0.14.4
defusedxml                               0.7.1
Deprecated                               1.2.14
deprecation                              2.1.0
depth_anything                           2024.1.22.0
depth_anything_v2                        2024.7.1.0
depyf                                    0.18.0
diff_gaussian_rasterization              0.0.0
diffoctreerast                           0.0.0
diffusers                                0.31.0
dill                                     0.3.8
diskcache                                5.6.3
Distance                                 0.1.3
distlib                                  0.3.9
distributed                              2024.11.1
distro                                   1.9.0
dlinfo                                   1.2.1
dnspython                                2.7.0
docker                                   7.1.0
docker-pycreds                           0.4.0
docopt                                   0.6.2
docstring-parser                         0.15
docutils                                 0.21.2
docx2txt                                 0.8
duckdb                                   0.9.2
duckduckgo_search                        6.3.5
dulwich                                  0.21.7
durationpy                               0.9
dynamicprompts                           0.31.0
easydict                                 1.13
easygui                                  0.98.3
ebcdic                                   1.1.1
ecdsa                                    0.19.0
editdistance                             0.8.1
einops                                   0.4.1
einops-exts                              0.0.4
email_validator                          2.2.0
embreex                                  2.17.7.post5
emoji                                    2.14.0
en_core_web_sm                           3.8.0
encodec                                  0.1.1
entrypoints                              0.4
environs                                 9.5.0
et_xmlfile                               2.0.0
eval_type_backport                       0.2.0
Events                                   0.5
exceptiongroup                           1.2.2
executing                                2.1.0
exllamav2                                0.2.7
extract-msg                              0.52.0
facexlib                                 0.3.0
faiss-cpu                                1.9.0.post1
fake-useragent                           1.5.1
fal_client                               0.5.0
fastapi                                  0.104.1
fastapi-cli                              0.0.5
fastapi-slim                             0.115.5
fasteners                                0.19
faster-whisper                           1.0.3
fastjsonschema                           2.20.0
fastparquet                              2024.11.0
fasttext                                 0.9.3
ffmpeg                                   1.4
ffmpy                                    0.4.0
fiddle                                   0.3.0
filelock                                 3.16.1
filetype                                 1.2.0
filterpy                                 1.4.5
fire                                     0.7.0
FlagEmbedding                            1.3.2
flash-attn                               2.7.3
flashinfer                               0.1.6+cu121torch2.4
Flask                                    3.0.3
flask-cloudflared                        0.0.14
Flask-Cors                               5.0.0
Flask-RESTful                            0.3.10
flatbuffers                              24.3.25
fonttools                                4.53.1
formatron                                0.4.10
fpdf2                                    2.7.9
frozendict                               2.4.6
frozenlist                               1.4.1
fsspec                                   2024.6.1
ftfy                                     6.2.3
fugashi                                  1.4.0
future                                   1.0.0
fvcore                                   0.1.5.post20221221
g2p-en                                   2.1.0
g2pkk                                    0.1.2
gast                                     0.6.0
gcsfs                                    2023.12.2.post1
gdown                                    5.2.0
gekko                                    1.2.1
general_sam                              1.0.1
gguf                                     0.10.0
ghp-import                               2.1.0
git-python                               1.0.3
gitdb                                    4.0.11
GitPython                                3.1.32
glcontext                                3.0.0
google-ai-generativelanguage             0.6.6
google-api-core                          2.23.0
google-api-python-client                 2.153.0
google-auth                              2.34.0
google-auth-httplib2                     0.2.0
google-auth-oauthlib                     1.0.0
google-cloud-core                        2.4.1
google-cloud-storage                     2.18.2
google-crc32c                            1.6.0
google-generativeai                      0.7.2
google-pasta                             0.2.0
google-resumable-media                   2.7.2
googleapis-common-protos                 1.63.2
GPUtil                                   1.4.0
gradio                                   4.40.0
gradio_client                            1.2.0
gradio_imageslider                       0.0.20
gradio_litmodel3d                        0.0.1
gradio_rangeslider                       0.0.6
graphviz                                 0.8.4
greenlet                                 3.0.3
griffe                                   1.4.1
grpcio                                   1.66.1
grpcio-status                            1.62.3
grpcio-tools                             1.62.3
grpclib                                  0.4.7
gruut                                    2.4.0
gruut-ipa                                0.13.0
gruut_lang_de                            2.0.1
gruut_lang_en                            2.0.1
gruut_lang_es                            2.0.1
gruut_lang_fr                            2.0.2
gunicorn                                 21.2.0
h11                                      0.14.0
h2                                       4.1.0
h5py                                     3.12.1
handrefinerportable                      2024.2.12.0
hangul-romanize                          0.1.0
hf_transfer                              0.1.8
hjson                                    3.1.0
hnswlib                                  0.8.0
hpack                                    4.0.0
html5lib                                 1.1
httpcore                                 1.0.7
httplib2                                 0.22.0
httptools                                0.6.4
httpx                                    0.28.1
httpx-sse                                0.4.0
huepy                                    1.2.1
huggingface-hub                          0.26.2
humanfriendly                            10.0
hydra-core                               1.3.2
hyperframe                               6.0.1
HyperPyYAML                              1.2.2
idna                                     3.10
igraph                                   0.11.8
ijson                                    3.3.0
imageio                                  2.36.0
imageio-ffmpeg                           0.5.1
imagesize                                1.4.1
importlib_metadata                       8.4.0
importlib_resources                      6.4.5
inflect                                  7.3.1
inflection                               0.5.1
iniconfig                                2.0.0
inscriptis                               2.5.0
insightface                              0.7.3
installer                                0.7.0
instructor                               0.4.8
interegular                              0.3.3
intervaltree                             3.1.0
iopath                                   0.1.10
ipycanvas                                0.13.3
ipyevents                                2.0.2
ipython                                  8.27.0
ipywidgets                               8.1.5
ir_datasets                              0.5.9
isodate                                  0.7.2
isort                                    5.13.2
itsdangerous                             2.2.0
jaconv                                   0.4.0
jamo                                     0.4.1
Janome                                   0.5.0
jaraco.classes                           3.4.0
jax                                      0.4.36
jaxlib                                   0.4.36
jedi                                     0.19.1
jeepney                                  0.8.0
jieba                                    0.42.1
Jinja2                                   3.1.5
jiter                                    0.5.0
jiwer                                    3.0.4
jmespath                                 1.0.1
joblib                                   1.4.2
jsonlines                                1.2.0
jsonmerge                                1.8.0
jsonpatch                                1.33
jsonpath-python                          1.0.6
jsonpointer                              3.0.0
jsonschema                               4.23.0
jsonschema-specifications                2024.10.1
julius                                   0.2.7
jupyter_client                           7.4.9
jupyter_core                             5.7.2
jupyterlab_widgets                       3.0.13
kaldi-python-io                          1.2.2
kaldiio                                  2.18.0
kaolin                                   0.17.0
kbnf                                     0.4.1
keras                                    3.6.0
keyring                                  24.3.1
kiwisolver                               1.4.5
kornia                                   0.6.7
kornia_rs                                0.1.5
kubernetes                               31.0.0
langchain                                0.3.5
langchain-chroma                         0.1.4
langchain-community                      0.3.3
langchain-core                           0.3.19
langchain-text-splitters                 0.3.2
langcodes                                3.4.1
langdetect                               1.0.9
langfuse                                 2.44.0
langsmith                                0.1.143
language_data                            1.2.0
language-tags                            1.2.0
lark                                     1.1.2
lark-parser                              0.12.0
latexcodec                               3.0.0
lazy_loader                              0.4
ldap3                                    2.9.1
Levenshtein                              0.26.1
lhotse                                   1.27.0
libclang                                 18.1.1
libcst                                   1.5.1
libmambapy                               2.0.2
librosa                                  0.10.2.post1
lightning                                2.4.0
lightning-utilities                      0.11.6
lilcom                                   1.8.0
linkify-it-py                            2.0.3
livereload                               2.7.0
llama-cpp-agent                          0.0.17
llama_cpp_python                         0.3.1+cpuavx2
llama_cpp_python_cuda                    0.3.1+cu121
llama_cpp_python_cuda_tensorcores        0.3.1+cu121
llama-cpp-scripts                        0.0.0
llmcompressor                            0.3.0
llvmlite                                 0.41.1
lm-format-enforcer                       0.10.9
lmdeploy                                 0.6.4                                /home/user/apps/lmdeploy
loadimg                                  0.1.2
locket                                   1.0.0
loguru                                   0.7.2
loky                                     3.4.1
lxml                                     5.3.0
lz4                                      4.3.3
Mako                                     1.3.5
mamba-ssm                                2.2.2
manifold3d                               3.0.0
mapbox_earcut                            1.0.2
marisa-trie                              1.2.1
Markdown                                 3.7
markdown-it-py                           3.0.0
markdown2                                2.5.1
MarkupSafe                               2.1.5
marshmallow                              3.23.1
matplotlib                               3.9.2
matplotlib-inline                        0.1.7
matrix-client                            0.4.0
mdit-py-plugins                          0.4.2
mdurl                                    0.1.2
mecab-python3                            1.0.10
mediapipe                                0.10.18
megatron-energon                         4.0.0
memray                                   1.15.0
menuinst                                 2.2.0
mergedeep                                1.3.4
milvus-lite                              2.4.10
mistral_common                           1.5.2
mistral_inference                        1.5.0
mistralai                                1.5.0
mkdocs                                   1.6.1
mkdocs-autorefs                          1.2.0
mkdocs-get-deps                          0.2.0
mkdocs-material                          9.5.40
mkdocs-material-extensions               1.3.1
mkdocstrings                             0.26.2
mkdocstrings-python                      1.12.1
ml-dtypes                                0.4.1
mmengine-lite                            0.10.5
mmh3                                     5.0.1
modal                                    0.67.7
moderngl                                 5.12.0
mojimoji                                 0.0.13
monotonic                                1.6
monotonic-alignment-search               0.1.1
more-itertools                           10.4.0
moviepy                                  1.0.3
mpmath                                   1.3.0
mpv                                      1.0.7
msgpack                                  1.0.8
msgspec                                  0.18.6
msoffcrypto-tool                         5.4.2
multidict                                6.0.5
multiprocess                             0.70.16
murmurhash                               1.0.10
mxnet-cu102                              1.9.1
mypy-extensions                          1.0.0
namex                                    0.0.8
narwhals                                 1.13.3
nemo_text_processing                     1.1.0
nemo_toolkit                             2.0.0
nerfacc                                  0.5.3
nest-asyncio                             1.6.0
networkx                                 2.8.8
ninja                                    1.11.1.3
nltk                                     3.9.1
num2words                                0.5.14
numba                                    0.58.1
numcodecs                                0.13.1
numpy                                    1.26.4
nvdiffrast                               0.3.3
nvidia-cublas-cu12                       12.4.5.8
nvidia-cuda-cupti-cu12                   12.4.127
nvidia-cuda-nvcc-cu12                    12.5.82
nvidia-cuda-nvrtc-cu12                   12.4.127
nvidia-cuda-runtime-cu12                 12.4.127
nvidia-cudnn-cu12                        9.1.0.70
nvidia-cufft-cu12                        11.2.1.3
nvidia-curand-cu12                       10.3.5.147
nvidia-cusolver-cu12                     11.6.1.9
nvidia-cusparse-cu12                     12.3.1.170
nvidia-ml-py                             12.560.30
nvidia-nccl-cu12                         2.21.5
nvidia-nvjitlink-cu12                    12.4.127
nvidia-nvtx-cu12                         12.4.127
nvitop                                   1.3.2
oauthlib                                 3.2.2
objaverse                                0.1.7
olefile                                  0.47
oletools                                 0.60.2
omegaconf                                2.2.3
onnx                                     1.16.2
onnxruntime                              1.20.0
onnxruntime-gpu                          1.19.2
open-clip-torch                          2.20.0
open-webui                               0.4.1
openai                                   1.59.8
OpenCC                                   1.1.6
opencensus                               0.11.4
opencensus-context                       0.1.3
opencv-contrib-python                    4.10.0.84
opencv-python                            4.10.0.84
opencv-python-headless                   4.10.0.84
openpyxl                                 3.1.5
opensearch-py                            2.7.1
opentelemetry-api                        1.27.0
opentelemetry-exporter-otlp-proto-common 1.27.0
opentelemetry-exporter-otlp-proto-grpc   1.27.0
opentelemetry-instrumentation            0.48b0
opentelemetry-instrumentation-asgi       0.48b0
opentelemetry-instrumentation-fastapi    0.48b0
opentelemetry-proto                      1.27.0
opentelemetry-sdk                        1.27.0
opentelemetry-semantic-conventions       0.48b0
opentelemetry-util-http                  0.48b0
opt_einsum                               3.4.0
optimum                                  1.23.3
optree                                   0.13.0
optuna                                   3.6.1
orderly-set                              5.2.2
orjson                                   3.10.10
oscrypto                                 1.3.0
outlines                                 0.1.11
outlines_core                            0.1.26
overrides                                7.7.0
packaging                                24.1
paginate                                 0.5.7
pandas                                   1.5.3
pangu                                    4.0.6.1
parameterized                            0.9.0
parso                                    0.8.4
partd                                    1.4.2
partial-json-parser                      0.2.1.1.post4
passlib                                  1.7.4
pathspec                                 0.12.1
pccm                                     0.4.16
pcodedmp                                 1.2.6
pdf2audio                                0.1.0                                /home/user/apps/01_audio/Nvidia_Nemo_FastPitch_TTS_Example
pdfminer.six                             20231228
pdfplumber                               0.11.4
peewee                                   3.17.6
peewee-migrate                           1.12.2
peft                                     0.13.2
pesq                                     0.0.4
pexpect                                  4.9.0
pgvector                                 0.3.5
phonemizer                               3.3.0
piexif                                   1.1.3
pillow                                   10.4.0
pillow-avif-plugin                       1.4.3
pip                                      24.2
piper-phonemize                          1.1.0
piper-tts                                1.2.0
pkginfo                                  1.11.2
plac                                     1.4.3
platformdirs                             4.2.2
pluggy                                   1.5.0
plyfile                                  1.1
poetry                                   1.8.4
poetry-core                              1.9.1
poetry-plugin-export                     1.8.0
pooch                                    1.8.2
portalocker                              2.10.1
posthog                                  3.7.2
preshed                                  3.0.9
prettytable                              3.11.0
primePy                                  1.3
primp                                    0.7.0
proglog                                  0.1.10
progress                                 1.6
progressbar                              2.5
prometheus_client                        0.21.0
prometheus-fastapi-instrumentator        7.0.0
prompt_toolkit                           3.0.47
propcache                                0.2.0
proto-plus                               1.25.0
protobuf                                 3.20.0
psutil                                   5.9.5
psycopg2-binary                          2.9.9
ptyprocess                               0.7.0
pure_eval                                0.2.3
py-cpuinfo                               9.0.0
py-markdown-table                        1.2.0
py-spy                                   0.4.0
pyairports                               2.1.1
pyannote.audio                           3.3.1
pyannote.core                            5.0.0
pyannote.database                        5.1.0
pyannote.metrics                         3.2.1
pyannote.pipeline                        3.0.1
pyarrow                                  17.0.0
pyarrow-hotfix                           0.6
pyasn1                                   0.6.0
pyasn1_modules                           0.4.0
PyAudio                                  0.2.14
pybind11                                 2.13.6
pybtex                                   0.24.0
pybtex-docutils                          1.0.3
pyclipper                                1.3.0.post6
pycollada                                0.8
pycosat                                  0.6.6
pycountry                                24.6.1
pycparser                                2.22
pydantic                                 2.10.6
pydantic_core                            2.27.2
pydantic-settings                        2.6.1
pydeck                                   0.9.1
pydub                                    0.25.1
PyGithub                                 2.4.0
pygltflib                                1.16.3
Pygments                                 2.19.1
pyHanko                                  0.25.3
pyhanko-certvalidator                    0.26.5
PyJWT                                    2.9.0
pylatexenc                               2.10
pyloudnorm                               0.1.1
PyMatting                                1.1.13
PyMCubes                                 0.1.6
pymdown-extensions                       10.11.2
pymeshfix                                0.17.0
pymilvus                                 2.4.9
pymongo                                  4.10.1
PyMySQL                                  1.1.1
PyNaCl                                   1.5.0
pynini                                   2.1.6.post1
pynndescent                              0.5.13
pynvml                                   11.5.3
pypandoc                                 1.13
pyparsing                                3.1.4
pypdf                                    4.3.1
pypdfium2                                4.30.0
PyPika                                   0.48.9
pypinyin                                 0.53.0
pypinyin-dict                            0.8.0
pyproject_hooks                          1.2.0
PyQt5                                    5.15.11
PyQt5-Qt5                                5.15.15
PyQt5_sip                                12.15.0
pysbd                                    0.3.4
pySmartDL                                1.3.4
PySocks                                  1.7.1
pystoi                                   0.4.1
pytest                                   8.3.3
pytest-docker                            3.1.1
pytest-mock                              3.14.0
pytest-runner                            6.0.1
python-bidi                              0.6.3
python-crfsuite                          0.9.11
python-dateutil                          2.9.0.post0
python-dotenv                            1.0.1
python-engineio                          4.10.1
python-iso639                            2024.10.22
python-jose                              3.3.0
python-magic                             0.4.27
python-multipart                         0.0.20
python-oxmsg                             0.0.1
python-pptx                              1.0.0
python-socketio                          5.11.3
python_speech_features                   0.6
pytorch-lightning                        1.9.4
pytorch-metric-learning                  2.6.0
pyttsx3                                  2.90
pytube                                   15.0.0
pytz                                     2024.1
pyvista                                  0.44.2
PyWavelets                               1.8.0
pyxlsb                                   1.0.10
PyYAML                                   6.0.2
pyyaml_env_tag                           0.1
pyzmq                                    26.2.0
qdrant-client                            1.12.1
qrcode                                   8.0
qwen-vl-utils                            0.0.8
rank-bm25                                0.2.2
rapidfuzz                                3.9.6
rapidocr-onnxruntime                     1.3.24
ray                                      2.38.0
rdflib                                   7.1.1
red-black-tree-mod                       1.20
redis                                    5.1.1
referencing                              0.35.1
regex                                    2024.7.24
rembg                                    2.0.60
replicate                                1.0.4
reportlab                                4.2.5
requests                                 2.32.3
requests-oauthlib                        2.0.0
requests-toolbelt                        1.0.0
resampy                                  0.4.3
resize-right                             0.0.2
rfc3986                                  1.5.0
rich                                     13.9.4
rotary-embedding-torch                   0.8.6
rouge                                    1.0.1
rouge_score                              0.1.2
rpds-py                                  0.21.0
rsa                                      4.9
RTFDE                                    0.1.2
Rtree                                    1.3.0
ruamel.yaml                              0.18.10
ruamel.yaml.clib                         0.2.8
ruff                                     0.7.0
s3fs                                     0.4.2
s3transfer                               0.10.3
sacrebleu                                2.4.3
sacremoses                               0.1.1
safehttpx                                0.1.6
safetensors                              0.5.2
scenedetect                              0.6.5.2
scikit-build                             0.18.1
scikit-image                             0.21.0
scikit-learn                             1.3.2
scipy                                    1.13.1
scooby                                   0.10.0
seaborn                                  0.13.2
SecretStorage                            3.3.3
segment-anything                         1.0
segments                                 2.2.1
semantic-version                         2.10.0
semver                                   3.0.2
sentence-transformers                    3.2.0
sentencepiece                            0.2.0
sentry-sdk                               2.13.0
setproctitle                             1.3.3
setuptools                               69.5.1
setuptools-scm                           8.1.0
shapely                                  2.0.6
shellingham                              1.5.4
shortuuid                                1.0.13
sigtools                                 4.0.1
simple-parsing                           0.1.6
simple-websocket                         1.1.0
simpleeval                               1.0.3
six                                      1.16.0
smart-open                               6.4.0
smmap                                    5.0.1
sniffio                                  1.3.1
snowballstemmer                          2.2.0
sortedcontainers                         2.4.0
sounddevice                              0.5.1
soundfile                                0.12.1
soupsieve                                2.6
sox                                      1.5.0
soxr                                     0.3.7
spacy                                    3.7.2
spacy-legacy                             3.0.12
spacy-loggers                            1.0.5
spandrel                                 0.3.4
spandrel_extra_arches                    0.1.1
spconv-cu120                             2.3.6
speechbrain                              1.0.0
SpeechRecognition                        3.10.0
Sphinx                                   8.1.3
sphinxcontrib-applehelp                  2.0.0
sphinxcontrib-bibtex                     2.6.3
sphinxcontrib-devhelp                    2.0.0
sphinxcontrib-htmlhelp                   2.1.0
sphinxcontrib-jsmath                     1.0.1
sphinxcontrib-qthelp                     2.0.0
sphinxcontrib-serializinghtml            2.0.0
SQLAlchemy                               2.0.32
srsly                                    2.4.8
sse-starlette                            2.2.1
stack-data                               0.6.3
stanza                                   1.9.2
starlette                                0.27.0
stream2sentence                          0.3.0
streamdiffusion                          0.1.1
streamlit                                1.40.1
stringzilla                              3.10.5
SudachiDict-core                         20241021
SudachiPy                                0.6.9
surrealist                               1.0.2
svg.path                                 6.3
svglib                                   1.5.1
sympy                                    1.13.1
symusic                                  0.5.4
synchronicity                            0.9.3
tabbyAPI                                 0.0.1
tabulate                                 0.9.0
taming-transformers                      0.0.1
tblib                                    3.0.0
tenacity                                 8.5.0
tensorboard                              2.18.0
tensorboard-data-server                  0.7.2
tensorboardX                             2.6.2.2
tensorflow                               2.18.0
tensorflow-io-gcs-filesystem             0.37.1
tensorrt                                 10.5.0
tensorrt-cu12                            10.5.0
tensorrt-cu12-bindings                   10.5.0
tensorrt-cu12-libs                       10.5.0
tensorstore                              0.1.45
termcolor                                2.5.0
text-unidecode                           1.3
textdistance                             4.6.3
texterrors                               0.5.1
texttable                                1.7.0
textual                                  1.0.0
tf_keras                                 2.18.0
thinc                                    8.2.5
threadpoolctl                            3.5.0
tifffile                                 2024.9.20
tiktoken                                 0.7.0
timm                                     1.0.9
tinycss2                                 1.4.0
tokenizers                               0.21.0
tomesd                                   0.1.3
toml                                     0.10.2
tomli                                    2.0.2
tomlkit                                  0.12.0
toolz                                    0.12.1
torch                                    2.5.1+cu124
torch-audiomentations                    0.11.1
torch-pitch-shift                        1.2.4
torchaudio                               2.5.1+cu124
torchdiffeq                              0.2.3
torchinfo                                1.8.0
torchlibrosa                             0.1.0
torchmetrics                             1.4.1
torchsde                                 0.2.6
torchvision                              0.20.1+cu124
tornado                                  6.4.1
tqdm                                     4.66.1
traitlets                                5.14.3
trampoline                               0.1.2
transformers                             4.49.0.dev0
trec-car-tools                           2.6
trimesh                                  4.5.2
triton                                   3.1.0
trove-classifiers                        2024.10.21.16
truststore                               0.10.0
typeguard                                4.3.0
typer                                    0.15.1
types-certifi                            2021.10.8.3
types-psutil                             6.1.0.20241102
types-toml                               0.10.8.20240310
typing                                   3.7.4.3
typing_extensions                        4.12.2
typing-inspect                           0.9.0
tzdata                                   2024.1
tzlocal                                  5.2
uc-micro-py                              1.0.3
ujson                                    5.10.0
ultralytics                              8.3.13
ultralytics-thop                         2.0.9
umap-learn                               0.5.7
Unidecode                                1.3.8
unidic-lite                              1.0.8
unlzw3                                   0.2.2
unstructured                             0.15.9
unstructured-client                      0.27.0
uritemplate                              4.1.1
uritools                                 4.0.3
urllib3                                  2.2.2
usd-core                                 24.11
utils3d                                  0.0.2
uuid                                     1.30
uvicorn                                  0.30.6
uvloop                                   0.21.0
validators                               0.33.0
vhacdx                                   0.0.8.post1
virtualenv                               20.27.1
vite                                     1.5.2
vllm                                     0.1.dev4188+gb5b57e3.d20250118.cu124 /mnt/LLM_Models/vllm_pixtral
voluptuous                               0.15.2
vox2seq                                  0.0.0
vtk                                      9.3.1
wandb                                    0.17.8
warc3-wet                                0.2.5
warc3-wet-clueweb09                      0.2.5
warp-lang                                1.5.0
wasabi                                   1.1.3
watchdog                                 5.0.3
watchfiles                               0.24.0
wcwidth                                  0.2.13
weasel                                   0.3.4
webcolors                                24.8.0
webdataset                               0.2.100
webencodings                             0.5.1
websocket-client                         1.8.0
websockets                               11.0.3
Werkzeug                                 3.0.4
wget                                     3.2
wheel                                    0.45.1
widgetsnbextension                       4.0.13
wrapt                                    1.16.0
wsproto                                  1.2.0
xatlas                                   0.0.9
xformers                                 0.0.29.post1
xgrammar                                 0.1.9
xhtml2pdf                                0.2.16
xlrd                                     2.0.1
XlsxWriter                               3.2.0
xtts-api-server                          0.9.0
xxhash                                   3.5.0
yacs                                     0.1.8
yamlargparse                             1.31.1
yapf                                     0.43.0
yarl                                     1.18.3
youtokentome                             1.0.6
youtube-transcript-api                   0.6.2
zarr                                     2.18.3
zict                                     3.0.0
zipp                                     3.20.2
zlib-state                               0.1.9
zope.interface                           7.2
zstandard                                0.23.0

Additional context

Cheers! :)

Acknowledgements

• I have looked for similar issues before submitting this one.
• I understand that the developers have lives and my issue will be answered when possible.
• I understand the developers of this program are human, and I will ask my questions politely.