Tokenizer while indexing

[chenyn66]

Hi,
I'm wondering how to change the default tokenizer while indexing documents. More specifically, I want to index my documents with a T5 tokenizer. Is it possible to do that?
Thanks!

[lintool]

Current solution is to generate a new version of the corpus that has already been tokenized, and then use the -pretokenized option during indexing. Note that queries need to be similarly pretokenized.

For a full example, see https://github.com/castorini/anserini/blob/master/docs/regressions-msmarco-passage-wp.md - this is MS MARCO with WordPiece tokenization.

[chenyn66]

Thanks for the instruction!