### Ticket [Link to Github Issue](tenstorrent/tt-xla#5217) ### Problem description [`microsoft/VibeVoice-1.5B`](https://huggingface.co/microsoft/VibeVoice-1.5B) — a long-form, generation-based text-to-speech (multimodal TTS) model — has no tt-forge-models loader. It cannot be brought up as-is: the HuggingFace repo ships **weights only** (no remote code), and the architecture (`model_type: vibevoice`) is **not in transformers**, so there is no off-the-shelf way to instantiate the model. The full model is also ~2.7B params (the "1.5B" refers only to the Qwen2.5 LM backbone; the rest is acoustic/semantic VAE tokenizers + diffusion head + connectors), and its dataclass forward output contains a non-tensor int leaf that breaks the inference harness's pytree comparator. ### What's changed Adds a PyTorch loader for VibeVoice-1.5B in **port mode**: the core model code is vendored from [github.com/microsoft/VibeVoice](https://github.com/microsoft/VibeVoice) `@303b2833` (provenance + file list in `vibevoice/pytorch/src/SRC_VENDORED_FROM.txt`). - `ModelLoader` / `ModelVariant` return the full VibeVoice model with **random weights** built from a vendored real 1.5B `config.json` (config-only — no weight download). - `_VibeVoiceLogitsWrapper` exposes a clean **tensors-only** `forward()` returning just `logits`, so the harness's pytree comparator doesn't trip on the dataclass int leaf (`speech_token_num`) under `torch.equal`. - `transformers>=5` compat shims, documented inline: `tie_weights(self, *args, **kwargs)` (v5 passes `recompute_mapping=`) and idempotent `AutoModel(.ForCausalLM).register(exist_ok=True)` (v5 pre-registers configs). - Bringup forward path uses `speech_tensors=None` → reduces to a clean Qwen2.5 LM forward → logits. `speech_semantic_tensors (1,1,128)` is required (semantic connector is called unconditionally) but its result is unused. **Impact:** VibeVoice-1.5B can now be loaded and run through the standard tt-forge-models flow. Brought up single-device on **n150 (inference)** via tt-xla, enabling the companion tt-xla test-config PR. Layout: ``` vibevoice/ ├── __init__.py └── pytorch/ ├── __init__.py ├── loader.py ├── config.json # vendored real 1.5B arch (config-only; random weights) └── src/ └── vibevoice/{modular,schedule}/... # 6 core files vendored from upstream ``` Notes: vendored `dpm_solver.py` keeps its TSAIL/HuggingFace Apache-2.0 notice (SPDX tag added); other ported files carry the standard Tenstorrent Apache-2.0 header per repo convention (matches the `maptr`/`transfuser`/`bevformer` vendoring pattern). The empty `__init__.py`s under `src/vibevoice/` intentionally override upstream ones (which pull in streaming/processor + gradio/av/aiortc and double-register Auto classes). ### Checklist - [x] New/Existing tests provide coverage for changes — single-device n150 inference passes via `test_all_models_torch[vibevoice/pytorch-1.5B-single_device-inference]` (`1 passed`, default PCC); CPU sanity produces finite logits `(1, 32, 151936)`. ### Logs [bringup_steps.txt](https://github.com/user-attachments/files/29241006/bringup_steps.txt) [iter_2_run.log](https://github.com/user-attachments/files/29241008/iter_2_run.log) [model_overview.md](https://github.com/user-attachments/files/29241010/model_overview.md) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

[Model Bringup]VibeVoice 1.5B #5217

Description

Activity

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions