TikTokenizer tiktoken-pattern v1 and v2

[bugm]

I noticed that when using TikTokenizer there are two tiktoken pattern
where

PATTERN_TIKTOKEN = r"[^\r\n\p{L}\p{N}]?+\p{L}+|\p{N}| ?[^\s\p{L}\p{N}]++[\r\n]*|\s*[\r\n]|\s+(?!\S)|\s+"
PATTERN_TIKTOKEN_V2 = "[^\\r\\n\\p{L}\\p{N}]?[\\p{Lu}\\p{Lt}\\p{Lm}\\p{Lo}\\p{M}]*[\\p{Ll}\\p{Lm}\\p{Lo}\\p{M}]+|[^\\r\\n\\p{L}\\p{N}]?[\\p{Lu}\\p{Lt}\\p{Lm}\\p{Lo}\\p{M}]+[\\p{Ll}\\p{Lm}\\p{Lo}\\p{M}]*|\\p{N}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n/]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+"

can you offer some explanation to this two different pattern? Thanks

[Advait251206]

The main difference is that PATTERN_TIKTOKEN_V2 is a refinement of the original pattern, particularly in how it tokenizes Unicode text and words with mixed casing.

At a high level:

PATTERN_TIKTOKEN

[^\r\n\p{L}\p{N}]?+\p{L}+

This essentially treats a sequence of Unicode letters (\p{L}+) as a single token, optionally preceded by one non-letter/non-number character.

For most English text, this works well:

• hello
• Hello
• WORLD

are all matched simply as one sequence of letters.

PATTERN_TIKTOKEN_V2

Instead of simply matching \p{L}+, it explicitly distinguishes between different Unicode letter categories:

• \p{Lu} – uppercase letters
• \p{Ll} – lowercase letters
• \p{Lt} – titlecase letters
• \p{Lm} – modifier letters
• \p{Lo} – other letters
• \p{M} – combining marks

This allows it to better handle:

• CamelCase words
• PascalCase words
• mixed-case identifiers
• languages that rely heavily on combining marks and more complex Unicode casing rules

For example, strings like:

HTTPRequest
myXMLParser
İstanbul
naïve

can be segmented more consistently than with the original pattern.

Other small differences

There are also a couple of other notable changes:

• The newline handling is slightly different ([\r\n/]* vs [\r\n]* in one branch), suggesting adjustments to punctuation/newline tokenization.
• The V2 pattern is generally more explicit about Unicode categories instead of treating all letters uniformly.

Why have two versions?

Typically, tokenizer regexes evolve for one or more of these reasons:

• improve compatibility with newer tiktoken vocabularies,
• better support multilingual text,
• produce more stable tokenization for mixed-case words and identifiers,
• or better match the behavior of newer OpenAI tokenizer versions.

Without a design note or maintainer comment, it's difficult to say which specific motivation led to V2, but it appears to be an evolution of the original pattern rather than a completely different tokenization strategy.

---

If this answer helped or pointed you in the right direction, I'd appreciate it if you could mark it as the accepted answer so it's easier for others with the same issue to find.

Also, if you found my contribution useful, I'd appreciate it if you could check out my GitHub profile, follow me, and star any repositories you find interesting.

GitHub: https://github.com/Advait251206