OPENNLP-1885: Add subword API and WordPiece encoder, remove BertTokenizer

[krickert]

Summary

• adds the SubwordTokenizer contract and SubwordPiece to opennlp-api; a piece consists of the spelling, the id, and the UTF-16 span in the original input text, with empty spans for control and fill pieces
• adds WordpieceEncoder, a dependency-free encoder for BERT-style word lists with longest-prefix segmentation, original-text offsets, and Unicode case mapping including the final sigma context
• removes BertTokenizer; opennlp-dl builds a WordpieceEncoder from the model word list and takes the ids from encodeTokens in AbstractDL
• aligns WordpieceTokenizer with the encoder: arguments are validated and copied, the default word limit is 100 code points (it was 50 UTF-16 chars), and the protected createTokenizer(vocab, lowerCase) overload is removed
• documents subword tokenization in the manual, with the example output pinned by a test

This PR contains no SentencePiece model reader, model normalizer, or inference engine. The concrete SentencePiece implementation is apache/opennlp-addons#178.

API

SubwordTokenizer.encode(text) returns List<SubwordPiece>. encodeToIds and encodeToPieces provide the corresponding compact views. WordpieceEncoder accepts a word list, applies longest-prefix segmentation, and preserves original-text offsets. The maximum word length is a number of Unicode code points.

Review of 2026-09-04

All items from the September 4 review are in the code: full Unicode lower casing, checked with Python's str.lower() on the pinned Greek sequences; parity tests that use an independent reference pipeline and pinned sequences; DL ids taken from the encoder; the shared 100 code point limit; validation at the public boundary; trie lookup; per-run normalization; @since 3.0.0; Map.copyOf fields. The earlier inline threads refer to SentencePiece files that moved to the addons PR.

Validation

Offline build with checkstyle and -Dopennlp.forkCount=1:

• opennlp-api: 385 tests
• opennlp-runtime: 2680 tests, 4 pre-existing skips
• opennlp-dl: 90 tests

No failures. The eval build will be re-run on this head before merge.

OPENNLP-1885

[krickert]

Single-thread throughput measurement after 3fb8b6b, for the record.

Machine: AMD Ryzen 9 9950X3D (16 cores, one thread used), Linux, OpenJDK 25.0.3. Workload: 100,100 short texts (77 distinct lines cycled), t5-small unigram model, 32k vocabulary, 1.17M pieces total, 3 warmup passes over the corpus, then 5 timed passes.

• opennlp-subword: 6.47M pieces/s (554k texts/s), producing piece, id, and original-text span for every token
• Reference implementation, sentencepiece 0.2.1 via its Python binding, one encode call per text, ids only: 4.57M pieces/s (391k texts/s)
• opennlp-subword before the optimization commit: 2.83M pieces/s

That is 1.42x the reference on the same corpus and model, measured call for call from a host language, so the binding's per-call overhead is included in the reference number; the raw C++ core inside a batch loop is faster than that number. The Java side also does more work per token, since it maps every piece back to a UTF-16 span of the original input, which the reference does not produce against the original string.

Output is unchanged: the bundled parity fixtures and the T5-small and ALBERT real-model fixtures assert identical pieces, ids, spans, and normalized forms against the reference before and after the optimization commit.

[krickert]

This is now dependent on the embeddings to land. Marking ready for review

[rzo1]

If it depends on #1152 , it should still be "draft" state (since the base PR is also draft)

[krickert]

@rzo1 Correction, my earlier comment had the dependency backwards: this PR is standalone on main, and #1152 is the one that stacks on it (its base is this branch). Nothing in opennlp-subword references the embeddings module, and CI is green on the full matrix with no #1152 code involved. Marking it ready again, sorry for the churn.

[rzo1]

Thanks for the extensive work here. Reviewed file-by-file — no blocking issues; the public API validates at the boundary with IAE, serialVersionUIDs are in order, the module is wired in, and the BertTokenizer removal is fine for a pre-release milestone. A handful of minor items to clean up before merge:

Javadoc — missing @throws

• WordpieceEncoder.java:75 / :87 — the 1-arg and 2-arg public constructors delegate to the throwing map constructor (null/duplicate vocab, missing special token) but lack the @throws IllegalArgumentException clause the 5-arg (:106) and map (:126) constructors already carry. Please add it, matching the 5-arg wording.
• PieceTrie.java:95 — build() propagates IAE on a duplicate piece but omits the @throws both Builder methods carry. Add @throws IllegalArgumentException Thrown if a piece is defined more than once.
• DoubleArrayTrie.java:57 — constructor throws IAE when length is non-positive or not a multiple of four; add @throws IllegalArgumentException Thrown if length is not a positive multiple of four.
• DoubleArrayTrie.java:79 — longestPrefixMatch() maps an out-of-range unit reference (corrupt data) to IAE with no @throws; please document it.

Constants

• ModelProtoReader.java:100 — the tag field-number shift (>>> 3) and wire-type mask (& 7) are raw literals repeated ~14×, while WIRE_*/FIELD_* are already named constants. Please declare TAG_FIELD_SHIFT = 3 and TAG_WIRE_MASK = 7 (or fieldOf(tag)/wireTypeOf(tag) helpers) and use them at every tag-decomposition site.
• PieceTrie.java:70 — the 256-entry dispatch-table width is an unnamed literal at :70 and :76; only DIRECT_THRESHOLD is named. Please add DIRECT_TABLE_SIZE = 256.

Duplication

• IntBuilder.java vs ByteBuilder.java — the 1.5× growth (data.length + (data.length >> 1)), the Math.max(capacity, 16) floor, and truncate() validation are duplicated byte-for-byte. Please name the shared 16 floor and growth policy so the two copies stay in sync. The int/byte split itself is fine as primitive specialization.

Comments

• SentencePieceNormalizer.java:150 / :193 — "heading whitespace" / "heading spaces" should read "leading" in both comments.

Exception convention (discussion)

• SentencePieceTokenizer.java:224 / :240 — load(Path) / load(InputStream) surface a malformed .model as the unchecked IllegalArgumentException, which diverges from the other OpenNLP model loaders that throw the checked InvalidFormatException for bad model content. Would you consider InvalidFormatException for content errors (keeping IAE for null-arg guards)? The class currently uses IAE uniformly by design, so flagging for discussion rather than as a fix.

Process

• BertTokenizer shipped in the opennlp-3.0.0-M4 tag, so describing its removal as "unreleased" is slightly imprecise. Not blocking for a milestone, but please correct the wording and add a one-line migration pointer for former BertTokenizer users → WordpieceEncoder.

One note on coverage: I reviewed the parity/fixture tests for form, not by re-running them. The piece-for-piece reference assertions look right structurally, but I have not executed the suite as part of this pass.

[mawiesne]

This will be projected to OpenNLP 3.0.0 (M6) - not M5

[rzo1]

This PR removes stuff which was added in a previous milestone release of opennlp. Please add some rational so reviewers get an idea why sth was dropped.

In addition, it needs to be carefully checked, which classes belong into API and which can go into a custom submoduel.

[krickert]

@rzo1 the removals are intentional; I'll provide the full rational in a few hours. I should've provided a changelog at the top of the review. It'll go over every detail.

Moving to draft until that detailed explanation lands.

[krickert]

Since I pushed it, while enhancing it I didn't think it was as good of a contract.

There are two subword engines and they needed one contract. BertTokenizer could not be it: it was pinned to Tokenizer, which promises spans into the input, and wordpiece pieces are not substrings of the text. That is why its tokenizePos threw from the day I added it.

The new contract is SubwordTokenizer, returning SubwordPiece(piece, id, start, end), so the piece and the original-text span are separate fields. BertTokenizer folded into WordpieceEncoder under it, and SentencePieceTokenizer implements the same one.

Example:

WordpieceEncoder encoder = new WordpieceEncoder(vocabulary, lowerCase);
List<SubwordPiece> pieces = encoder.encode(text);   // piece, vocab id, and span
String[] tokens = encoder.encodeToPieces(text);     // what BertTokenizer.tokenize returned

The old pipeline is kept as ReferenceBertPipeline and the encoder is differential-tested against it, so the piece sequence is asserted identical.

AbstractDL.createTokenizer now returns Tokenizer but there's no in-tree caller. It is binary-incompatible for anyone overriding it.

opennlp-dl compiles against opennlp-api only and consumes wordpiece, so wordpiece stays in api. SentencePiece has no core consumer, so that engine sits in opennlp-extensions/opennlp-subword with only the contract in api.

[rzo1]

Nothing in your argumentation force the removal: those are separable. EncoderTokenizer already is the compat shim, it's just package-private and in opennlp-dl. Three things I'd like to change:

1. Keep BertTokenizer, reimplemented over WordpieceEncoder. @deprecated(since = "3.0.0", forRemoval = true), same constructors, tokenize() delegates to encodeToPieces(), tokenizePos() keeps throwing as it does today. The old ctor takes Set and the encoder wants ids, but ids are unused on the tokenize() path, so a synthesized index map is fine (worth a comment saying so). EncoderTokenizer then goes away and the adapter sits in opennlp-api, where downstream code can actually reach it, instead of being hidden in opennlp-dl.
2. Drop ReferenceBertPipeline and differential-test against the real class. Right now the baseline is a test-only copy of the class being deleted. If the deprecated BertTokenizer stays, point WordpieceEncoderTest at it instead : same assertion, and it additionally pins the shim and the encoder to the same sequence. One less copy of the normalization pipeline to keep in sync.
3. Revert AbstractDL.createTokenizer to protected BertTokenizer createTokenizer(...). You flag this yourself and it's the part that worries me most. Narrowing the return type from BertTokenizer to Tokenizer doesn't only break recompilation: an already-compiled subclass overriding it with descriptor ()Lopennlp/tools/tokenize/BertTokenizer; stops overriding at runtime, so the base implementation silently wins and the subclass's tokenizer is never used. A silent behavior change in a protected extension point is worse than a compile error. With (1) in place this reverts to a one-word change, since createPipelineTokenizer can hand back the shim.

If you'd rather see the class gone in this PR, the minimum I'd want is the adapter promoted to public API in opennlp-api plus a migration note, so Tokenizer t = new BertTokenizer(vocab, lowerCase) has a one-line replacement. But it's ~30 lines of delegation and it buys back both the source API and the binary compatibility, so I'd rather deprecate now and remove in 3.1, after it has been deprecated through one stable release.

The encoder itself and the span mapping look good - this is only about how we retire the old entry point.

[krickert]

No problem! On it now.

[krickert]

All three are in.

1. BertTokenizer is back in opennlp-api as a shim over WordpieceEncoder: @Deprecated(since = "3.0.0", forRemoval = true), the original three constructors, tokenize() delegating to encodeToPieces(), tokenizePos() throwing with the original message. Ids are synthesized from the set order, with the comment, since the tokenize() path never reads them. EncoderTokenizer is gone.
2. ReferenceBertPipeline is gone; the curated and randomized differential tests now run against the shim, and a new BertTokenizerTest pins the constructors, the default special token chain, and the exact tokenizePos message. The independent expected sequences stay in WordpieceEncoderReferenceSequencesTest.
3. AbstractDL.createTokenizer returns BertTokenizer again, so the old override descriptor holds, and createPipelineTokenizer hands back the shim.

The compatibility check before deleting the old pipeline surfaced a real divergence: the encoder kept U+2028 and U+2029 inside words while the old WhitespaceTokenizer split on them, so a word carrying a line or paragraph separator collapsed to [UNK]. Fixed in cleanAndIsolateCjk (Zl and Zp map to a space now, matching reference BERT's str.split()), with a span-asserting regression test. The old fuzz pool never contained those characters, which is how it survived the differential tests.

One deliberate drift to follow convention, documented in the @throws clauses: the shim rejects nulls with IllegalArgumentException rather than the old Objects.requireNonNull NPE, and it fails at construction when a special token is missing from the vocabulary instead of tokenizing toward unmappable pieces. Both follow the null-contract convention this branch was reviewed to. Say so if you want the old NPE behavior kept instead.

[krickert]

Per our DEV conversation, I have slimmed this one down a bit so the rest of the impl can be in an addon

apache/opennlp-addons#178

The API portion is here.

[rzo1]

Here are some load bearing ( ;-) ) comments:

Blocking

1. WordpieceEncoder.java:477 — case-mapping regression vs. both the reference and 3.0.0-M5.
StringUtil.toLowerCase is a simple, per-code-point mapping (Character.toLowerCase(int)). The BertTokenizer.normalize this replaces used String.toLowerCase(Locale.ROOT), and the Python reference uses str.lower() — both full mappings. Verified divergence:

"ΣΟΦΟΣ"  full → σ ο φ ο ς (U+03C2)   per-codepoint → σ ο φ ο σ (U+03C3)

So Greek text ending in Σ now tokenizes differently from HuggingFace and from what shipped in M4/M5, while the class Javadoc (WordpieceEncoder.java:36-53) claims reference-pipeline parity and links tokenization.py. Please lower case the whole run once and map offsets onto the result, or state the divergence explicitly. Note that WordpieceEncoderTest#testCodePointCaseMappingPreservesSourceRange currently pins the wrong value (vocabulary entry σοφοσ).

2. WordpieceEncoderTest#testPieceSequenceMatchesBertTokenizerOnCuratedInputs / #...OnRandomInputs — the parity tests are tautological.
BertTokenizer.tokenize() is return encoder.encodeToPieces(text), so both tests compare the encoder against itself; the 800 random rounds assert nothing. That is exactly why (1) slipped through. Please replace with expectations pinned from the reference implementation (as WordpieceEncoderReferenceSequencesTest does) or with sequences captured from the M5 BertTokenizer. The span-invariant checks embedded in the random test are the only non-vacuous part — keep those, in their own test.

3. AbstractDL.java:259 — the new API has no real caller; DL still routes through the deprecated shim and throws the ids away.
createPipelineTokenizer builds a BertTokenizer from vocab.keySet(), so ids get reassigned by arbitrary Set iteration order and are then discarded. DocumentCategorizerDL:413, NameFinderDL:826 and SentenceVectorsDL:155 each re-look-up every token via vocab.get(token) — which is precisely encodeToIds. AbstractDL already holds the real Map<String,Integer>, and WordpieceEncoder has a Map constructor for exactly this. Using it, plus encodeToIds, also removes every @SuppressWarnings("removal") in this PR.

4. AbstractDL.java:253 — protected BertTokenizer createTokenizer(...) returns a forRemoval type from protected DL API. Please return Tokenizer, or drop the overload.

5. BertTokenizer.java:76 — the "compatibility class" is not compatible.
Special tokens must now be present in the vocabulary (WordpieceEncoder.requiredId). new BertTokenizer(Set.of("the", "fox")) worked in M4/M5 — the deleted testCustomSpecialTokens did exactly that — and now throws IAE at construction. A deprecated shim must not change behavior: either keep it lenient, or make this a removal rather than a deprecation.

6. BertTokenizer.java:47 — the deprecation decision is not made.
@Deprecated(since = "3.0.0", forRemoval = true) sits on a class introduced in 3.0.0 (e7e1189, OPENNLP-1837) that has only ever shipped in M4/M5, and there is no other forRemoval in the codebase. Preference: delete it before GA — no pre-GA API deserves a shim plus suppression annotations across two modules. If it stays, name the removal version.

7. WordpieceEncoder.java:61 — inconsistent siblings in the same package.
MAX_WORD_CHARACTERS is hard-coded at 100 with no constructor, while WordpieceTokenizer exposes the same limit as a constructor parameter defaulting to 50, counted in UTF-16 chars, where this counts code points. Three divergences between two classes users have to pick between.

8. Partial validation alignment.
This PR gives WordpieceEncoder and BertTokenizer full IAE validation, but WordpieceTokenizer.java:119 still does this.vocabulary = vocabulary; with no checks at all. Either align all tokenizers in the package or drop it.

Minor

• WordpieceEncoder.java:63,164,284 — the vocabulary Set is a redundant copy of ids.keySet(), and contains() + get() is two hash lookups per candidate in the inner longest-match loop. One Integer id = ids.get(s); if (id != null) does it.
• WordpieceEncoder.java:282-284 — new String(chars, ...) plus CONTINUATION_PREFIX + substring is allocated per candidate length per word, i.e. O(n²) strings for an OOV word. Hot path.
• WordpieceEncoder.java:471-495 — Character.toChars + new String + StringUtil.toLowerCase + Normalizer.normalize + a StringBuilder, all per code point. Please do this per run, and add a Normalizer.isNormalized fast path.
• WordpieceEncoder.java:443 — the run splitting in lowerCaseAndStripAccents is dead complexity: transformRun ignores the from/to boundaries and works code point by code point. Drop the split.
• WordpieceEncoder.java:465-470 — the Javadoc says ranges fall back to "the run's full range otherwise"; there is no such branch. Javadoc must describe the code.
• WordpieceEncoder.java:149,188 — new HashMap<>(size * 2) is magic sizing. We are on Java 21: HashMap.newHashMap(size).
• WordpieceEncoder.java:213 — parameter ids shadows the field ids, which is already assigned at both call sites. Drop the parameter.
• WordpieceEncoder.java:395-403 — isLineOrParagraphSeparator is character classification and belongs in BertNormalization next to isControl/isWhitespace/isCjk/isPunctuation. Its four-line Javadoc arguing with the reference implementation should be one line plus the JIRA pointer.
• WordpieceEncoder.java:415 — second copy of BertNormalization.isolatePunctuation. Offset tracking justifies it, but please add a pointer comment that the two must stay in sync.
• SubwordPiece.java:36, SubwordTokenizer.java:31, WordpieceEncoder.java:54 — missing @since 3.0.0; we use it on new API (see Document.java:48).
• SubwordTokenizer.java:37,46,62 — "@return ... empty when no units can be encoded", but WordpieceEncoder never returns fewer than two pieces. The interface says nothing about CLS/SEP framing, so encodeToIds is not portable across implementations. Please decide whether framing is part of the contract and document it there.
• SubwordPiece.java:60 — span() allocates per call and duplicates the record's own start/end. Given Span is right there, either hold a Span or drop the accessor.
• BertTokenizer.java:9 — the license-header re-indent and the removed blank line after it are unrelated churn; please revert.
• BertTokenizer.java:88 — tokenize now throws IAE for null (was NPE). {@inheritDoc} alone is not enough, add @throws IllegalArgumentException.
• AbstractDL.java:245-258 — @return A configured {@link BertTokenizer} on a method whose return type is being removed. Also, "pipeline tokenizer" is invented vocabulary; createBertPipelineTokenizer or plain createTokenizer reads better.
• WordpieceEncoderTest#testValidationRejectsInvalidInput — nine assertThrows in one method, so a failure does not identify the input. Please use @ParameterizedTest. Also java.util.Map.of(...) is written fully qualified inline — import it.
• WordpieceEncoderTest — magic seed 42 and 400 hand-rolled rounds; and the comment "The Turkish dotted capital I: lower cases to two chars, then the dot strips away" describes String.toLowerCase, not this code — Character.toLowerCase(U+0130) yields i directly.
• tokenizer.xml:552 — the docs point users at a SentencePiece implementation in opennlp-addons, but apache/opennlp-addonsOPENNLP-1040: Add OntoNotes4 training data verification #178 is not merged. Please drop the forward reference until it lands.
• WordpieceEncoder.java:47 — @ThreadSafe plus a prose repetition of it, and the fields are mutable HashSet/HashMap. Set.copyOf/Map.copyOf if immutability is being claimed.

Process

• Finding 1 changes DL output (SentenceVectorsDL vectors, doccat, NER) for any input with context-sensitive case mappings, so an eval build is needed before merge.
• The Javadoc, validation and test-parameterization items recur across your other open PRs (OPENNLP-1886: UniNE light and minimal stemmer tiers with vocabulary parity fixtures #1166, OPENNLP-1887: Lexical expansion over the knowledge base seam: synonyms, hypernyms, optional hyponyms #1167, OPENNLP-1909: General verified installer for user-supplied third-party resources #1211, OPENNLP-547: Add a dependency parser component #1236) — please apply them there likewise.
• The PR title is fine for release notes, but it omits the BertTokenizer deprecation; worth adding once finding 6 is decided.

[krickert]

tl;dr - I double and triple checked that everything was fixed. I thought it was a good review.

The summary:

Reply to rzo1's 2026-09-04 review (issue comment)

Rebased on main and reworked; all eight blocking items and the minor list are in.

1. Case mapping: the lower-casing run now applies the word-final sigma rule, so ΣΟΦΟΣ encodes to σοφος with U+03C2, and testFinalSigmaMappingPreservesSourceRange pins it. The Turkish dotted I reaches i by either route once accents are stripped, so no other full-mapping case differs from the reference.
2. Parity tests: the shim is gone. WordpieceEncoderTest checks generated Unicode input against the reference basic tokenizer's normalization order, and WordpieceEncoderReferenceSequencesTest pins expected sequences. The span invariants have their own test.
3. and 4. AbstractDL builds a WordpieceEncoder from the vocabulary map, and the doccat, name finder, and sentence vector callers take ids from encode. No @SuppressWarnings("removal") remains, and the protected createTokenizer keeps its WordpieceTokenizer return type.
4. and 6. BertTokenizer is deleted, per your preference, and the PR title says so.
5. and 8. Both wordpiece classes use a 100 code point word limit through a constructor parameter, validate their arguments, and copy their vocabularies.

Minor items: the vocabulary set and per-candidate strings are replaced by a trie, normalization runs per word with an isNormalized fast path, the dead run split and the stale Javadoc are gone, HashMap.newHashMap, no shadowed ids, isLineOrParagraphSeparator lives in BertNormalization with a sync note on isolatePunctuation, @since on the three types, the framing statement on SubwordTokenizer, no span() accessor, validation tests parameterized with named seeds, and the manual no longer points at the addons PR.

The old inline threads on the SentencePiece fixtures and trie refer to code that moved to apache/opennlp-addons#178; resolving them here.

Resolution note for the ten outdated inline threads

Moved to apache/opennlp-addons#178 with the SentencePiece implementation; this PR carries only the API and WordPiece.

===================

The load-bearing rung of greek casing has landed. OpenNLP has earned it's keep.

[krickert]

Getting ready for eval build on this one

https://ci-builds.apache.org/job/OpenNLP/job/eval-tests-configurable/72/console

Running...

[krickert]

[INFO] Running opennlp.tools.util.normalizer.GermanUmlautCharSequenceNormalizerTest
[INFO] Running opennlp.tools.util.normalizer.UrlCharSequenceNormalizerCharacterizationTest
[ERROR] Tests run: 10, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 3.090 s <<< FAILURE! -- in opennlp.tools.namefind.RegexNameFinderFactoryTest
[ERROR] opennlp.tools.namefind.RegexNameFinderFactoryTest.testBuiltinPatternsAreNotVulnerableToReDoS -- Time elapsed: 2.372 s <<< FAILURE!
org.opentest4j.AssertionFailedError: execution timed out after 2000 ms
	at org.junit.jupiter.api.Assertions.assertTimeoutPreemptively(Assertions.java:3570)
	at opennlp.tools.namefind.RegexNameFinderFactoryTest.testBuiltinPatternsAreNotVulnerableToReDoS(RegexNameFinderFactoryTest.java:109)
Caused by: org.junit.jupiter.api.timeout.PreemptiveTimeoutUtils$ExecutionTimeoutException: Execution timed out in thread junit-timeout-thread-1
	at java.base/java.util.regex.Pattern$Branch.match(Pattern.java:4914)
	at java.base/java.util.regex.Pattern$GroupHead.match(Pattern.java:4969)
	at java.base/java.util.regex.Pattern$Branch.match(Pattern.java:4914)
	at java.base/java.util.regex.Pattern$GroupHead.match(Pattern.java:4969)
	at java.base/java.util.regex.Pattern$Bound.match(Pattern.java:5579)
	at java.base/java.util.regex.Pattern$StartS.match(Pattern.java:3820)
	at java.base/java.util.regex.Matcher.search(Matcher.java:1767)
	at java.base/java.util.regex.Matcher.find(Matcher.java:787)
	at opennlp.tools.namefind.RegexNameFinder.find(RegexNameFinder.java:93)
	at opennlp.tools.namefind.RegexNameFinderFactoryTest.lambda$testBuiltinPatternsAreNotVulnerableToReDoS$0(RegexNameFinderFactoryTest.java:113)

[INFO] Running opennlp.tools.util.normalizer.CharSequenceNormalizerContractTest
[INFO] Tests run: 81, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.215 s -- in opennlp.tools.namefind.BilouCodecTest
[INFO] Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.705 s -- in 

The RegexNameFinderFactoryTest failed. It might survive if I re-ran, but treating it as an error. I'll make it a separate ticket and keep this open in case downstream there's another test that is broken. Because this is a class we just worked on, we should speed it up rather than declare a flake. If I find something, I'll make a new ticket.

Keeping open until eval green

[krickert]

#1268

I created this for the eval test fix. I took the guilty test and separated it with better test coverage. I also found some regex in the code that we should deal with (lesson: we always see vast performance improvements whenever we remove regex). That'll be a separate set of tickets and out of scope (and not needed for 3.0, but should have a PR).

[jzonthemtn]

Is this one ready for a review? I see @rzo1 made some comments but was holding off for a bit while that discussion was going on.

[krickert]

Yes, this one is ready for review. All eight blocking items and the minor list from Richard's September 4 review are in the code: the lower casing now applies the full Unicode case mapping including the final sigma context, and I checked every pinned Greek case against Python's str.lower(), which is what the reference tokenizer uses. The parity tests compare against an independent reference pipeline and pinned sequences rather than against the encoder itself. The DL classes build a WordpieceEncoder from the vocabulary id map and take ids straight from encode, and BertTokenizer is removed rather than shimmed, per Richard's preference. Both wordpiece classes share the 100 code point word limit, validate their arguments, and copy their vocabularies.

Since then I merged current main (which includes the eval test fix from #1268), folded the three per-class encode helpers in opennlp-dl into the one AbstractDL method, added the missing Javadoc, and pinned the manual's WordPiece example output in a test. The earlier inline threads refer to the SentencePiece code that moved to opennlp-addons#178; this PR is only the SubwordTokenizer contract, SubwordPiece, and the WordPiece encoder. I will resolve those threads. The description is updated to the current head, CI is running on it, and I will re-run the eval build on it before merge.

[rzo1]

Reviewed b7c3a07 against my September 4 concerns. The eight blocking code items are addressed, and 101 focused tests passed locally. No new correctness blocker found.

[mawiesne]

Thx @krickert and @rzo1 - I'm fine with the changes