Code

Tokenizer fixes#113

Code

Merged

pcuenca merged 9 commits intomainhuggingface/swift-transformers:mainfrom tokenizer-fixeshuggingface/swift-transformers:tokenizer-fixesCopy head branch name to clipboardAug 19, 2024

+129-40

Commits

Commits on Aug 2, 2024

• Bring over hf token envvar from preview branch

  pcuenca

  committed
  01c0f25

  Copy full SHA for 01c0f25

• Add tests for Gemma, including edge cases

  Show description for 4149b55
  pcuenca

  committed
  4149b55

  Copy full SHA for 4149b55

• Sort added tokens by length (descending) to avoid early partial matches

  Show description for 59a5b3a
  pcuenca

  committed
  59a5b3a

  Copy full SHA for 59a5b3a

• Store vocab as NSString to allow multiple tokens with the same Unicode

  Show description for 4631505
  pcuenca

  committed
  4631505

  Copy full SHA for 4631505

• Remove comments

  pcuenca

  authored
  a6637e0

  Copy full SHA for a6637e0

• Go back to making vocab dictionaries private

  pcuenca

  committed
  266bbd0

  Copy full SHA for 266bbd0

• Use ungated copy of Gemma tokenizer

  pcuenca

  committed
  f2044bd

  Copy full SHA for f2044bd

Commits on Aug 3, 2024

• Use NSString in UnigramTokenizer

  pcuenca

  committed
  b610c2d

  Copy full SHA for b610c2d

Commits on Aug 19, 2024

• Switch test to microsoft tokenizer, verify in Python

  pcuenca

  committed
  5d48793

  Copy full SHA for 5d48793