how to use TikTokenizer during Training?

[bugm]

Hello,
I noticed that ML now support TikTokenizer by setting the --tokenizer-type argument. But I do not know what i should set with --tokenizer-model. I have checked the source code and find that we should pass a json file, and the function below will convert the json file to Tiktoken format.

Megatron-LM/megatron/training/tokenizer/tokenizer.py

Line 581 in 772faca

def reload_mergeable_ranks(path: str, max_vocab: Optional[int] = None) -> Dict[bytes, int]:

The comment says " Reload our tokenizer JSON file and convert it to Tiktoken format." What does "our tokenizer JSON" means? which format should the json file be?

[Advait251206]

This is one of those Megatron-LM “hidden contract” areas where the code is clearer than the documentation.

Let’s break it down properly.

---

🧠 1. What “our tokenizer JSON” actually means

When Megatron says:

“Reload our tokenizer JSON file and convert it to Tiktoken format”

it does NOT mean a generic HuggingFace tokenizer file.

It specifically means:

A BPE merge + vocab JSON exported in OpenAI / TikToken-compatible structure

---

📦 2. Expected JSON format

The file you pass via:

--tokenizer-model path/to/tokenizer.json

should contain mergeable ranks + vocabulary mapping, typically like:

{
  "version": "1",
  "pat_str": "...",
  "mergeable_ranks": {
    "b'Ġt'": 0,
    "b'he'": 1,
    "b'll'": 2
  },
  "special_tokens": {
    "<|endoftext|>": 100257,
    "<|pad|>": 100258
  }
}

---

🔥 3. What Megatron actually uses from it

Inside:

reload_mergeable_ranks(path)

Megatron extracts:

✔️ Required fields:

1. mergeable_ranks

This is the core:

byte_pair → rank

Example:

"b'hello'": 1234

This is equivalent to TikToken’s BPE ranks.

---

2. special_tokens

Used for:

• EOS
• BOS
• PAD
• control tokens

---

3. (optional) pat_str

Regex pattern for pre-tokenization splitting.

---

⚠️ 4. What it is NOT

This is NOT:

• HuggingFace tokenizer.json
• SentencePiece .model
• GPT-NeoX tokenizer config

Those formats will NOT work directly.

---

🧩 5. Why Megatron uses this format

Because TikToken internally expects:

Dict[bytes, int]

So Megatron converts JSON → TikToken BPE ranks:

JSON mergeable_ranks → bytes → integer IDs

That’s what reload_mergeable_ranks() does.

---

🛠️ 6. Where to get such a tokenizer

You usually have 3 options:

---

✅ Option A: Convert from TikToken

If you already have OpenAI tokenizer:

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")

Then export merges (custom script needed).

---

✅ Option B: Use Megatron-provided scripts

In newer Megatron versions:

tools/tokenizers/build_tiktoken_tokenizer.py

(or similar utilities depending on fork)

---

✅ Option C: Convert from SentencePiece / HF

Pipeline:

SentencePiece / HF tokenizer
        ↓
BPE merges extraction
        ↓
Megatron TikTokenizer JSON format

---

🧠 7. What to pass in CLI

Example:

--tokenizer-type TikTokenizer \
--tokenizer-model /path/to/tiktoken.json

NOT:

--tokenizer-model tokenizer.model   ❌ (sentencepiece)
--tokenizer-model tokenizer.json    ❌ (hf format)

unless already converted.

---

⚠️ 8. Common mistake

People often try:

tokenizer.json from HuggingFace

❌ This will fail because:

• structure is different
• no mergeable_ranks
• uses different serialization

---

📌 9. Mental model

Think of Megatron TikTokenizer format as:

“A serialized TikToken BPE table + special tokens registry”

NOT a general tokenizer config.

---

🧾 Final answer

• “our tokenizer JSON” = TikToken-style BPE mergeable ranks JSON
• must contain:
  • mergeable_ranks (required)
  • special_tokens (recommended)
  • optionally pat_str
• it is NOT HuggingFace or SentencePiece format
• it is a direct serialization of TikToken-compatible BPE vocabulary

---

If this answer helped or clarified things, I'd appreciate it if you could mark it as the accepted answer so it's easier for others with the same issue to find.

Also, if you found my contribution useful, I'd appreciate it if you could check out my GitHub profile, follow me, and star any repositories you find interesting.

GitHub: https://github.com/Advait251206