This repository was archived by the owner on Apr 23, 2025. It is now read-only.
This repository was archived by the owner on Apr 23, 2025. It is now read-only.
Character instead of String in Alphabet #600
Currently,
Alphabet'sdictionarymaps fromStringrather thanCharacterto support tokens of length > 1 character. UsingCharacterinstead ofStringwould work if we used special Unicode characters or enums instead of"</s>","</w>", and"<pad>".Since this is used in so many places in the WordSeg model, it is potentially worthwhile to make it more efficient.