OPENNLP-1895: Add quantized vector storage and search for static embeddings - #1213

Draft

krickert wants to merge 196 commits into

apache:mainfrom

ai-pipestream:OPENNLP-1895-turboquant

Commits on Aug 6, 2026

OPENNLP-1885: Add opennlp-subword: pure-Java SentencePiece inference with exact original-text spans
krickert
committed
6a71703
OPENNLP-1885: Speed up the encode path 2.3x, parity-checked at every step
krickert
committed
fe4abdd
OPENNLP-1885: Move the subword contract into opennlp-api
krickert
committed
db1c371
OPENNLP-1885: Add WordpieceEncoder and fold the unreleased BertTokenizer into it
krickert
committed
f71db4d
OPENNLP-1885: Document the hand-rolled protobuf reader rationale and refactor trigger
krickert
committed
abbf3e6
OPENNLP-1885: Trim commentary and tighten javadoc per review conventions
krickert
committed
5c78b97
OPENNLP-1885: Tighten javadoc to contracts and document helpers and overrides
krickert
committed
dea3031
OPENNLP-1885: Declare serialVersionUID on SentencePieceTokenizer
krickert
committed
1f3883a
OPENNLP-1885: Trim residual commentary per review conventions
krickert
committed
976326a
OPENNLP-1885: Document subword tokenization in the manual
krickert
committed
bf3cacf
OPENNLP-1885: Make the tokenizer graph serializable with computed UIDs, name the format constants, document every helper
krickert
committed
81b1581
OPENNLP-1885: Guard tokenizer deserialization with an allow-listing ObjectInputFilter
krickert
committed
e430109
OPENNLP-1885: Cite the SentencePiece usage example test in the manual
krickert
committed
b843cf7
OPENNLP-1885: Align null contracts and annotations with the review conventions
krickert
committed
f55fbec
OPENNLP-1885: Address review: checked InvalidFormatException for malformed models, tag helpers, javadoc throws
krickert
committed
8ebca7e
OPENNLP-1885: Address review: validation message style, shared test fixtures, thread safety wording
krickert
committed
80029ca
OPENNLP-1885: Deprecate BertTokenizer as a shim over WordpieceEncoder per review
krickert
committed
397c9ec
OPENNLP-1885: Document fixture regeneration and Utf8Text span mapping
krickert
committed
632d996
OPENNLP-1885: Attribute the double-array literature in DoubleArrayTrie
krickert
committed
9670c34
OPENNLP-1885: Expand fixtures README into a validation tutorial and link it from the manual
krickert
committed
d8b0449

Commits on Aug 8, 2026

Commits on Aug 9, 2026

Commits on Aug 10, 2026

Commits on Aug 16, 2026

Commits on Aug 21, 2026

OPENNLP-1877: load self-contained Model2Vec Unigram tokenizers
krickert
committed
f6f1bef

Commits on Aug 22, 2026

OPENNLP-1895: merge latest embedding foundation
krickert
committed
3388e41
OPENNLP-1895: adapt Unigram loading to embedding tables
krickert
committed
05398bb

Commits on Sep 2, 2026

Commits on Sep 3, 2026

Commits on Sep 4, 2026

OPENNLP-1885: Add subword API and WordPiece encoder
krickert
committed
6491446
OPENNLP-1885: Cover WordPiece parity and model ids
krickert
committed
c7ebfce
OPENNLP-1885: Align WordPiece API and DL callers
krickert
committed
141160e

Commits on Sep 5, 2026

Commits on Sep 6, 2026

OPENNLP-1877: Validate teacher vector lengths and test saved models
krickert
committed
9b0c434
Merge tested teacher validation into OPENNLP-1895-turboquant
krickert
committed
20d6689
OPENNLP-1895: Correct quantizer midpoint precision
krickert
committed
737fb6f
OPENNLP-1877: Test model loading and benchmark operation units
krickert
committed
54c70f8
OPENNLP-1895: Merge embedding benchmark coverage
krickert
committed
be60c73

Commits on Sep 9, 2026