feat(brief): the brief contract — top-8 synthesis, verified citations, enforce-mode gate, shared grounding spine (#4921)

[koala73]

Closes #4921 (Bet 3 of the strategic news-pipeline review). Do not auto-merge — held for review. ⚠️ Stacked on #4927 → #4924 — merge those first; this PR's diff then shrinks to its own commits.

The contract, delivered

Contract clause	Before	Now
Every top story gets prose	World Brief = LLM rewrite of ONE headline (pickBriefCluster)	one structured call → cited lead + one line per top-8 story; fence-tolerant JSON parser; L1 synthesis → L2 legacy single-headline → degraded (brief always ships)
Citations verified mechanically	[n] model-generated, rendered blind	verifyCitationIndexes strips out-of-range markers — World Brief lead + lines AND the country-intel brief (which had zero citation checks)
Validator in enforce mode	shadow default; detected hallucinations shipped	enforce default (env shadow escape hatch); per-line enforcement degrades an inventing line to its story's headline
One grounding implementation	checkLeadGrounding lived only in the Pro digest's lib	ported to shared/brief-llm-core.js (edge-safe, parameterized cap, no env reads), re-exported for backcompat; same-function-object test pins against drift; country-intel gains measure-only grounding telemetry
Staleness footer	nothing	payload sourceAgeRange + "Generated Nm ago · newest source Hh old" + cited story lines rendered under the lead ([n] → source links via formatIntelBrief)

Safety rails

• L2 keeps today's exact single-headline behavior (its prompts/tests untouched); callLLM gained prompt/maxTokens overrides backward-compatibly (retry tests unchanged).
• Parse contract: ≥ half the stories must have usable lines or L1 is rejected (a model that ignored the format is less trustworthy than the legacy path); missing lines fill from headlines.
• LKG preservation untouched: a degraded run still never overwrites an ok payload.
• i18n fanned out to all 26 locales (incl. fa) with {{token}} placeholders.

Verification

• New tests/brief-contract.test.mjs (17): prompt invariants, parser (fences/prose-wrap/dup+range indexes/thin-output rejection), citation verifier, grounding-port behavior incl. fabricated-lead rejection, same-function re-export, mirror byte-parity, wiring.
• Battery: brief-llm (107), brief-llm-core (47), edge-functions (216 — mirror gates), seed-insights-brief (13), seed-insights-llm-retry (4), completeness (21), both typechecks, biome.

https://claude.ai/code/session_01WmhkGjZrb9RbV7pV3muFxP

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
WorldMonitor	🟢 Ready	View Preview	Jul 5, 2026, 8:27 PM

💡 Tip: Enable Workflows to automatically generate PRs for you.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
worldmonitor	Ready	Preview, Comment	Jul 6, 2026 11:03am

[greptile-apps]

Greptile Summary

This PR completes the "brief contract" — replacing the single-headline World Brief with a structured top-8 synthesis (cited lead + one verified line per story), porting the grounding spine to a shared edge-safe module, flipping the hallucination validator to enforce-by-default, and consolidating three inconsistent "same story?" definitions into one dual-view lexical similarity engine.

• Brief synthesis (L1/L2 gate): seed-insights.mjs now makes a structured LLM call for a cited lead + per-story lines; parseBriefSynthesis strips fence markers and validates JSON shape; verifyCitationIndexes enforces in-range [n] markers on both the lead and each line; per-line hallucination failures degrade to the story's own headline. A clean cascade (L1 → L2 → degraded) means the brief always ships.
• Story identity unification (shared/story-identity.js): DUAL-VIEW feature-hashed lexical vectors replace three inconsistent matchers (Jaccard-0.5, word-overlap-0.6, exact sha256). Connected-components union-find clustering is order-independent. assignStoryIdentity anchors each cluster's canonical hash to its earliest-published member. mentionCount is now incremented once per unique hash per cycle.
• Coverage ledger (epic(news): completeness measurement — feed-health cron w/ silent-zero detection, coverage ledger + brief provenance, external recall benchmark (Bet 2) #4920): Every silent-drop gate in the digest pipeline now has a counted drop; a daily GDELT recall benchmark publishes a recall percentage.

Confidence Score: 4/5

The L1 to L2 to degraded cascade ensures the brief always ships; LKG preservation is untouched; all grounding and citation checks degrade gracefully.

The changes are architecturally sound and heavily tested (17 brief-contract + 389-line story-identity + 281-line completeness tests). The L2 fallback block indentation is a readability hazard and the parseBriefSynthesis stray-brace case is a silent failure class, but neither affects correctness or data safety.

scripts/seed-insights.mjs (L2 block indentation), scripts/_insights-brief.mjs (parseBriefSynthesis JSON extraction edge case)

Important Files Changed

Filename	Overview
scripts/seed-insights.mjs	Major rewrite adding L1 top-8 synthesis with grounding + per-line citation verification + enforce-mode gate; L2 legacy path preserved. L2 block body is unindented inside its guard. Staleness footer and provenance payload correctly wired.
scripts/_insights-brief.mjs	Adds synthesisSystemPrompt, synthesisUserPrompt, and parseBriefSynthesis. JSON extraction via lastIndexOf('}') degrades safely but silently on brace-containing trailing prose; quality gate and fence stripping are correct.
shared/story-identity.js	New DUAL-VIEW feature-hashed lexical similarity engine. Union-find clustering (order-independent), inverted-index candidate generation with hot-token bucket cap (250), containment rescue for severe RSS truncations.
server/worldmonitor/news/v1/dedup.mjs	Replaces word-overlap dedup with assignStoryIdentity using shared story-identity. Canonical hash anchored to earliest-published member. Sentinel identity for contentless titles avoids phantom-track pooling.
server/worldmonitor/news/v1/list-feed-digest.ts	Wires assignStoryIdentity before AI enrichment/scoring. Fixes mentionCount inflation by guarding HINCRBY once per unique hash per cycle. Adds coverage ledger with cardinality clamp on variant/lang.
scripts/_clustering.mjs	Delegates clustering to shared clusterTexts (connected components). selectTopStories gains stats parameter for coverage-ledger drop counts. Selected-story list is identical to old code.
shared/brief-llm-core.js	Ported checkLeadGrounding; adds verifyCitationIndexes. Edge-safe, no env reads. Byte-for-byte mirror to scripts/shared enforced by test.
server/worldmonitor/intelligence/v1/get-country-intel-brief.ts	Adds citation verification (enforce) and grounding telemetry (measure-only) to country-intel brief. Grounding logged but not enforced pending false-positive window.
src/components/InsightsPanel.ts	Adds renderBriefExtras (cited story lines + staleness footer) and renderProvenance. Correctly handles absent sourceAgeRange/generatedAt on old payloads.
tests/brief-contract.test.mjs	17 tests covering prompt invariants, parser edge cases, citation verifier, grounding-port behavior, same-function-object export pin, and byte-parity mirror check.
tests/story-identity.test.mjs	389-line suite with 13 positive + 10 negative labeled pairs (margin assertions), CJK, degenerate titles, canonical stability under batch reordering, and attribution-suffix handling.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[fetchInsights] --> B{topStories.length > 0?}
    B -- yes --> C[callLLM synthesis]
    B -- no --> G[L2 single-headline path]
    C --> D{parseBriefSynthesis valid?}
    D -- null --> G
    D -- parsed --> E{checkLeadGrounding?}
    E -- fail --> G
    E -- pass --> F[verifyCitationIndexes on lead]
    F --> F2[per-line verify + validate enforce degrade to headline]
    F2 --> F3[synthesized = true]
    G --> H{briefCluster + topHeadline?}
    H -- no --> I[degraded]
    H -- yes --> J[callLLM single headline]
    J --> K{validateNoHallucinatedProperNouns}
    K -- pass --> L[worldBrief = LLM]
    K -- enforce fail --> M[worldBrief = headline]
    F3 --> N[payload + LKG preservation]
    L --> N
    M --> N
    I --> N

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A[fetchInsights] --> B{topStories.length > 0?}
    B -- yes --> C[callLLM synthesis]
    B -- no --> G[L2 single-headline path]
    C --> D{parseBriefSynthesis valid?}
    D -- null --> G
    D -- parsed --> E{checkLeadGrounding?}
    E -- fail --> G
    E -- pass --> F[verifyCitationIndexes on lead]
    F --> F2[per-line verify + validate enforce degrade to headline]
    F2 --> F3[synthesized = true]
    G --> H{briefCluster + topHeadline?}
    H -- no --> I[degraded]
    H -- yes --> J[callLLM single headline]
    J --> K{validateNoHallucinatedProperNouns}
    K -- pass --> L[worldBrief = LLM]
    K -- enforce fail --> M[worldBrief = headline]
    F3 --> N[payload + LKG preservation]
    L --> N
    M --> N
    I --> N

Loading

Comments Outside Diff (1)

1. scripts/seed-insights.mjs, line 548-554 (link)

   L2 block body not indented inside if (!synthesized)

   The body of the if (!synthesized) guard is written at the same indentation level as the if statement itself, making it visually indistinguishable from outer-scope code. The closing } appears to "float" alone. In a 100-line block, a future editor could easily add logic outside the guard believing it is inside, or strip the guard in a refactor without realising the block is longer than it looks. Proper 2-space indentation for all lines between the opening and closing } would prevent this ambiguity.

   Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

_{Reviews (1): Last reviewed commit: "feat(brief): the brief contract — top-8 ..." | Re-trigger Greptile}

[greptile-apps]

lastIndexOf('}') JSON extraction fails silently when LLM appends brace-containing prose

The fence-strip + indexOf('{')/lastIndexOf('}') extraction handles the common case correctly, and the surrounding try/catch degrades to null (L2 fallback) on a parse failure. However, if the model appends trailing prose containing a } — e.g. "I formatted this as you asked {see above}." — lastIndexOf('}') picks up the stray brace rather than the object's closing delimiter. JSON.parse then fails and the synthesis silently falls back to the single-headline path with no warning, making this failure class invisible in logs. Adding a console.warn inside the catch block would surface these cases without changing the fallback behaviour.

[koala73]

Multi-agent code review — applied at f0ab384

Four reviewer personas (correctness, adversarial+security, testing, maintainability+standards) plus the independent Codex cross-model pass. All actionable findings applied; held for manual merge (stacked on #4927 → #4924).

Applied

Sev	Finding	Reviewers	Fix
P1	Citation indexes decoupled from worldBriefSources — filter(Boolean) shifted every later [n] to the wrong article	correctness + adversarial(75) + codex	strict index-lockstep: substitute entries, never filter; regression-tested
P1	L1 synthesis dropped the legacy corroboration bar (all-single-source days would ship a synthesized lead)	adversarial + codex + own fast-pass	editorial gate: ≥1 corroborated story or L1 rejects to L2
P1	Lead only anchor-grounded — could name one real anchor and still fabricate	adversarial + codex	validateNoHallucinatedProperNouns on the lead vs all story titles, enforce = reject to L2
P1	Grounding-port silently dropped the digest's env-tunable story cap (12→8)	maint(100)	explicit MAX_STORIES_PER_USER at the remaining call site
P1	[123] sailed through the citation verifier unverified	testing(100)	1–3-digit range; bracketed years stay prose; tested
P2	Degraded/missing lines lost their [n] (contract break for renderers)	codex	headline fallbacks carry their citation
P2	renderBriefExtras <p>-strip regex was a no-op (formatIntelBrief emits brief-para divs)	correctness(75)	correct wrapper strip
P2	L1 logic untestable inline; un-indented L2 if-wrapper brace hazard	testing + maint(100)	extracted pure composeSynthesizedBrief (7 functional tests: enforce/degrade, lockstep, editorial gate, fabricated-lead rejection) + generateLegacySingleHeadlineBrief
P2/P3	Boundary tests (lead 40/700), prompt-key pinning, dead imports, stale core header, stale corroboration comment	testing + maint	all covered/fixed

Gate catch worth noting: the pre-push unit gate caught that my refactor's end-anchor had swallowed the enrichedStories/provenance block — restored and re-verified before this push. The tiered gate earned its keep.

Verified during review

• XSS path closed by construction: formatIntelBrief escapes the full text before citation-linking and runs sanitizeUrl on hrefs (adversarial probe found no injection route from RSS titles → model output → rendered HTML).

Battery: brief-contract (26), brief-llm (107), brief-llm-core (47), edge-functions (216), completeness (21), seed-insights suites (17), both typechecks, biome.

[koala73]

External review adjudication — 8 findings: 7 validated + fixed, 1 fixed on the stack parent (52e70bb)

#	Verdict	Fix
1 (P1)	Validated — corpus-wide lead validation passed shape-valid misattribution.	Citation-SCOPED per sentence: every lead sentence must cite, and its proper nouns ground against only the stories it cites; uncited sentences reject to fallback. Misattribution + uncited-sentence regressions.
2 (P1)	Validated — a story-2 line citing [1] linked the wrong source.	Lines are rewritten to carry exactly their own [n]: all markers stripped, canonical citation appended (content is validated against story n, so [n] is its only correct citation). Wrong-[n] + no-surviving-citation regressions.
3 (P2)	Validated — 251 identical titles made every bucket hot; zero comparisons.	Exact-duplicate pre-union before the candidate scan (landed on #4924, ca80a2c35); 251-title regression.
4 (P2)	Validated — landed on #4927 (31d7a0465): a failed previous-state read now skips the publish instead of resetting streaks.
5 (P2)	Validated as a semantics mismatch — footer implied brief-source freshness while measuring the whole pool.	sourceAgeRange now computed from the brief's own top stories.
6 (P2)	Validated — DEFAULT_MAX_SOURCES=6 orphaned [7]/[8] through two layers.	Cap raised to the payload's citation index space in both collectBriefSources and the footer render.
7 (P3)	Validated — pre-rollout payloads rendered literal ?h old.	Footer omitted when sourceAgeRange is absent.
8 (P3)	Validated — lastIndexOf('}') died on stray braces in trailing prose.	Balanced, string-aware brace scan; stray-brace and brace-inside-string regressions.

brief-contract suite 32/32; both typechecks clean. Still held for manual merge.