feat: add a Node-side NLP segmentation strategy comparable to spaCy

[Moskize91]

Background

Before the TypeScript rewrite, the Python implementation relied on spaCy-style NLP segmentation. After the rewrite, the current Node pipeline mainly uses Intl.Segmenter plus lightweight normalization.

That replacement is not capability-equivalent. It has no real linguistic analysis, and in practice it can split Chinese text in misleading ways for our digest pipeline.

Problem

The current sentence segmentation can diverge significantly from what the model sees in the fragment text:

• quoted Chinese questions can be split at awkward boundaries
• list-like / archive-like paragraphs can be broken into many short sentence records
• line-break-heavy EPUB text can be segmented structurally rather than semantically

This causes downstream evidence matching to become much harder than it should be.

Even after hardening the matcher, the core limitation remains: we are compensating for weak segmentation instead of feeding the pipeline a stronger sentence/structure analysis layer.

Why this matters

The digest pipeline depends on stable evidence alignment between:

• stored fragment sentences
• fragment text sent to the LLM
• anchors returned by the LLM

If segmentation is too weak, we get:

• low-confidence anchor matches
• anchors spanning multiple stored sentences unexpectedly
• more retries and regeneration loops
• lower EPUB smoke-test stability

Expected direction

We should evaluate and introduce a Node-side NLP strategy that is closer in spirit to the old Python+spaCy behavior.

Possible directions:

• adopt a stronger Node NLP / sentence-boundary library for Chinese and mixed-format text
• provide a pluggable segmentation strategy instead of hard-coding Intl.Segmenter
• preserve richer structural boundaries from readers/parsers so segmentation is not purely punctuation-driven
• make fragment text generation and evidence matching share the same canonical sentence representation

Acceptance criteria

• segmentation quality is meaningfully stronger than raw Intl.Segmenter
• Chinese quoted dialogue and list/archive-style paragraphs are handled more robustly
• EPUB smoke stability improves without relying only on matcher-side fallbacks
• the segmentation backend is replaceable/configurable for future provider/library changes

Notes

This issue is not asking for a one-off matcher relaxation. The main gap is architectural: the TS rewrite currently lacks a Node-side NLP layer comparable to what the Python version had.