Tokenizer v0.2.2

Changes

• Fix non-breaking space separator (#44) @shekhirin

Thanks again to @LegendreM, and @shekhirin! 🎉

Tokenizer v0.2.1

Changes

• Add release drafter files (#37) @curquiza
• Add bors (#41) @curquiza
• Fix separators: treat cyrillic chars as non-separators (#39) @shekhirin

Thanks again to @shekhirin! 🎉

use HMM feature on jieba

Merge pull request #23 from meilisearch/use-hmm-on-jieba

Use hmm on jieba