What git branch corresponds to the latest version and what Docker version to use?

[karim-manaouil]

Hi,

I am having so many Python error due to version mismatching. I really could not get it right.

Could you please tell me what git branch I should use and what nvidia Docker version works with it?

I think it would be valuable to add this information to the README and update regularly whenever versions change.

[Advait251206]

Unfortunately, there isn't a single "latest branch ↔ Docker image" mapping that always works. Megatron-LM evolves rapidly, and the main branch often depends on recent versions of PyTorch, CUDA, Transformer Engine, Apex, and NVIDIA Container images. Mixing components from different release cycles is one of the most common causes of version mismatch errors.

Which branch should you use?

If you're:

• Developing new features or contributing to Megatron-LM: use the latest main branch, but also use the matching NVIDIA PyTorch container and install dependencies exactly as described in the repository for that revision.
• Training models for production or research: it's generally better to use a tagged release or a specific commit that has been validated for your workload rather than tracking main, since dependency requirements can change frequently.

Using a fixed commit together with its corresponding container provides a much more reproducible environment.

---

Which Docker image should you use?

The safest approach is to use the NVIDIA PyTorch NGC container that matches the version of Megatron-LM you're checking out.

In general, all of the following components should come from the same software stack:

• CUDA
• PyTorch
• Transformer Engine
• NCCL
• Apex
• Megatron-LM

Problems usually occur when, for example:

• Using an older NGC container with the latest main branch.
• Upgrading PyTorch without upgrading Transformer Engine.
• Installing Apex built against a different CUDA or PyTorch version.

Keeping these components aligned avoids many compatibility issues.

---

Recommended workflow

Instead of thinking in terms of:

Latest branch
+
Latest Docker image

it's better to think in terms of a validated software stack:

Megatron commit
↓
NGC PyTorch container
↓
Transformer Engine version
↓
Apex version
↓
CUDA version

This ensures that all components have been built and tested together.

---

If you're encountering version mismatch errors

It would help to know:

• The Megatron-LM commit or branch you're using.
• The Docker image (or NGC PyTorch container tag).
• Your CUDA version.
• Your PyTorch version.
• Your Transformer Engine version.
• The specific Python error or stack trace.

With that information, it's much easier to identify whether the issue stems from an incompatible dependency or an unsupported combination of versions.

---

I also agree that documenting a compatibility matrix (e.g., Megatron commit → recommended NGC container → PyTorch → CUDA → Transformer Engine) in the README would be very helpful. It would make it much easier for users to reproduce working environments and avoid the version mismatch issues that often arise when tracking the latest development branch.

---

If this answer helped or pointed you in the right direction, I'd appreciate it if you could mark it as the accepted answer so it's easier for others with the same issue to find.

Also, if you found my contribution useful, I'd appreciate it if you could check out my GitHub profile, follow me, and star any repositories you find interesting.

GitHub: https://github.com/Advait251206