[QUESTION] When will multimodal evaluation support pipeline parallelism ?

[wangxiaoyang-dev]

In currently, multimodal evaluation does not support pipeline parallelism, refer to

Megatron-LM/examples/multimodal/run_text_generation.py

Line 944 in db7d37b

logging.getLogger(__name__).warning("Models using pipeline parallelism are not supported yet.")

In some multimodals such as InternVL2-Llama3-76B, its vision model has 25 attention heads which does not support tensor parallelism, so it needs pipeline parallelism to evaluation. so i want to ask when will multimodal evaluation support pipeline parallelism ?

thank.

[Advait251206]

This is a good question, and the answer mostly comes down to architecture limitations rather than a missing feature toggle.

---

🚫 Why pipeline parallelism is currently not supported in multimodal evaluation

In Megatron-LM/examples/multimodal/run_text_generation.py, the warning:

logging.getLogger(__name__).warning(
    "Models using pipeline parallelism are not supported yet."
)

exists because the current multimodal inference stack assumes:

• a single-stage forward pass
• tightly coupled vision + language model execution
• no inter-stage activation scheduling across pipeline stages

Pipeline parallelism breaks this assumption because it introduces:

• micro-batch scheduling
• activation transfers between stages
• stage-wise execution boundaries

Multimodal models (like InternVL2 / LLaVA-style stacks) currently fuse vision encoder + projector + LLM in a way that is not yet pipeline-stage partition friendly.

---

⚠️ Why tensor parallelism is also tricky here

You already pointed out an important constraint:

vision model has 25 attention heads → does not divide cleanly for tensor parallelism

This is exactly why pipeline parallelism is theoretically needed, but:

• pipeline parallelism requires model partition annotations
• multimodal blocks (vision + projector + LLM) are not yet partition-aware
• cross-module tensor dependencies are not cleanly segmented

So even though pipeline parallelism seems like the natural fallback, it requires a redesign of the multimodal module boundaries.

---

🧠 Current design reality in Megatron-LM

Today, multimodal evaluation supports:

• ✅ Tensor Parallel (when dimensions align)
• ❌ Pipeline Parallel (not wired for multimodal graph)
• ❌ Interleaved PP + multimodal vision stacks

So the limitation is not accidental—it is because:

multimodal evaluation is still treated as a monolithic forward graph

not a pipeline-scheduled model.

---

🔮 When will it be supported?

There is no official ETA in the current discussions or codebase.

However, based on Megatron-LM development direction, pipeline parallel support for multimodal models depends on:

1. Refactoring model into stageable components

• vision encoder stage
• projector stage
• LLM decoder stage

2. Aligning multimodal blocks with Megatron pipeline engine

• defining forward/recv/send boundaries
• enabling activation checkpointing across modalities

3. Supporting mixed parallelism

• TP + PP hybrid for multimodal stacks

---

🧭 Practical workaround today

If you must run models like InternVL2-76B:

Option 1: disable pipeline parallelism

• rely on tensor parallel only (if possible)

Option 2: run vision encoder separately

• precompute vision embeddings
• feed into LLM-only Megatron pipeline

Option 3: inference frameworks instead

• vLLM / TensorRT-LLM / custom serving stacks often handle this more flexibly today

---

📌 Bottom line

• Pipeline parallelism is not yet implemented for multimodal evaluation
• The limitation is architectural, not a missing flag
• It requires model restructuring + engine integration work
• No confirmed timeline yet

---

If this answer helped or pointed you in the right direction, I'd appreciate it if you could mark it as the accepted answer so it's easier for others with the same issue to find.

Also, if you found my contribution useful, I'd appreciate it if you could check out my GitHub profile, follow me, and star any repositories you find interesting.

GitHub: https://github.com/Advait251206