Code

2x faster MosesTokenizer#149

Code

Open

de9uch1 wants to merge 3 commits intohplt-project:masterhplt-project/sacremoses:masterfrom de9uch1:tokenizer-speedupde9uch1/sacremoses:tokenizer-speedupCopy head branch name to clipboard

+24-14

Commits

Commits on May 26, 2024

• Pre-define the character sets for islower and isanyalpha

  de9uch1

  committed
  271b5f1

  Copy full SHA for 271b5f1

• Compile the dotmulti regex patterns

  de9uch1

  committed
  3e6d5ce

  Copy full SHA for 3e6d5ce

• Compile numeric and token_ends_with_period regex patterns

  de9uch1

  committed
  3a5463a

  Copy full SHA for 3a5463a