Use string_view::find() to search for tokenization to speed up

[yumeyao]

The old implementation using std::string::find() consumes a lot of time if the string is very long, because it searches beyond the fragment range, looking up for tokens after the fragment end (and later drop the match result if it is beyond the range).

O(n) wasted.

Furthermore, a long string produces more fragments and for each fragment, then a token is searched again and again in the fragments, i.e. It searches within [offset1, end) [offset2, end) [offset3, end) ... where offsetn is the nth of the fragments.

So actually O(n^2) wasted.

Hope this pic helps understand the issue more easily.

By limiting the search area (esp. the end) using std::string_view::find(), we can avoid such unnecessary looking up.

This PR contains the minimal code change to solve the performance pitfall, to at least make it work.
Maybe someone should refactor the whole fragment stuff with std::string_view for code readability in the future.

[ggerganov]

Could you give some sample numbers to get a feeling about the perf impact of this change?

[yumeyao]

Could you give some sample numbers to get a feeling about the perf impact of this change?

@KiruyaMomochi shared the issue to me to ask for help with the information 'std::string::find() is the hotspot in a profile result' and I instantly noticed the performance pitfall.

In my test on WSL2 Ubuntu 24.04 with gcc 13.3.0 @ AMD 5950x, building with only cmake without any extra flags, this fixes the issue:
@267c139 (before the fix): 2.7s
@63c6042 (the fix commit): 0.426s

To reproduce it:

1. prepare the test case
   printf '<start_of_turn>user\nhello<end_of_turn>\n%.0s' {1..200} > prompt.txt
2. run it with google gemma (in my case 3.4)
   time llama-tokenize --model google_gemma-3-4b-it-IQ2_M.gguf --file prompt.txt

I also tried changing {1..200} to {1..500}, to make the string longer, the result is more obvious:
14.249s vs 0.450s
this also confirms the theoretical O(n^2) 'wasted' time complexity.

[KiruyaMomochi]

My bad, it works like a charm. I didn't recompile after applying the patch.

Also, @LostRuins from LostRuins#1453.

[ggerganov]

I'm not following why this check is no longer needed.

If someone can confirm the correctness - feel free to merge. Otherwise I'll take a detailed look later.

[yumeyao]

it's checking if the match exceeds the length and drop the match if it does, as I have pointed out, it's doing this only because it searches for a match in [offset, end_of_string) instead of the expected range [offset, offset + length), so this extra check is to guarantee the behavior is same as searching within the expected range.

now we use string_view::find() to search the expected range directly hence we dont need this extra check.