feat(i18n): add PT-BR multilingual keyword locale support

[antoniocarlos97ss]

Summary

This PR implements multilingual keyword support for the Manifest prompt complexity scorer, starting with PT-BR (Brazilian Portuguese) as the first non-English locale.

Closes #1724

---

What changed

New files (purely additive — no existing code modified)

packages/backend/src/scoring/keywords/locales/
├── index.ts                    ← locale registry + detectLanguage() + mergeComplexityKeywords()
└── pt-BR/
    ├── complexity.ts           ← 14 scoring dimensions in PT-BR
    ├── calendar-management.ts
    ├── data-analysis.ts
    ├── email-management.ts
    ├── image-generation.ts
    ├── social-media.ts
    ├── trading.ts
    ├── video-generation.ts
    └── web-browsing.ts

Architecture

locales/index.ts exposes three functions:

Function	Purpose
detectLanguage(text)	Uses franc-min (< 1ms, local, zero API calls) to detect language. Falls back to MANIFEST_LOCALE env var.
getLocaleKeywords(lang)	Returns the LocaleKeywords object for a given BCP-47 code, or null if unsupported.
mergeComplexityKeywords(base, locale)	Merges locale keywords into base English set. Returns new object, never mutates.

franc-min is an optional peer dependency — if not installed, detection silently falls back to English-only behavior.

---

How to integrate into the scorer

The PR intentionally does not modify the scorer itself to allow maintainers to choose the integration point. The suggested minimal integration in scan-messages.ts or the complexity scorer:

import { detectLanguage, getLocaleKeywords, mergeComplexityKeywords } from './keywords/locales';
import { COMPLEXITY_KEYWORDS } from './keywords/complexity';

// Inside the scoring function, before building the trie:
const lang = detectLanguage(promptText);
const locale = getLocaleKeywords(lang);
const keywords = mergeComplexityKeywords(COMPLEXITY_KEYWORDS, locale);
// ... build trie from `keywords` as before

---

Adding more languages

To add a new locale (e.g. Spanish):

1. Create locales/es/ with the same 9 files
2. Register it in locales/index.ts under LOCALE_KEYWORDS['es']

No scorer changes needed.

---

Testing

To manually verify PT-BR detection and scoring works:

MANIFEST_LOCALE=pt-BR  # force PT-BR without needing franc-min installed

---

Notes

• ✅ Zero breaking changes — English behavior fully preserved
• ✅ franc-min graceful degradation — works without the package installed
• ✅ MANIFEST_LOCALE env override for power users / testing
• ✅ All 9 keyword categories covered for PT-BR
• ✅ Extensible: adding es, fr, de is ~50 lines each

---

Summary by cubic

Adds PT-BR multilingual keyword support to the prompt complexity scorer with auto language detection and safe fallback to English. Also fixes detection mapping, enforces strict typing across all 14 dimensions, and adds unaccented variants for better web-browsing matching.

• New Features

  • Locale registry with detectLanguage, getLocaleKeywords, and mergeComplexityKeywords (auto-detects with optional franc-min, supports MANIFEST_LOCALE; registers pt and pt-BR).
  • Full PT-BR keyword sets: 14 complexity dimensions + 8 task categories (calendar, data analysis, email, image/video, social, trading, web browsing).
  • Scorer unchanged; merge locale keywords into the base set before building the trie. Unsupported/undetected languages fall back to English.

• Bug Fixes

  • Normalize franc-min ISO-639-3 outputs to BCP-47 for locale lookup; PT-BR auto-detection now works.
  • Enforce ComplexityDimensions typing; added missing questionComplexity and domainSpecificity.
  • Added unaccented variants to web-browsing keywords since the trie isn’t accent-insensitive.

^{Written for commit 0cd0956. Summary will update on new commits.}

[cubic-dev-ai]

3 issues found across 10 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/backend/src/scoring/keywords/locales/index.ts">

<violation number="1" location="packages/backend/src/scoring/keywords/locales/index.ts:91">
P1: `detectLanguage` returns ISO-639-3 codes from `franc-min` (e.g., `por`), but locale lookup expects `pt`/`pt-BR`, so PT-BR auto-detection never matches.</violation>
</file>

<file name="packages/backend/src/scoring/keywords/locales/pt-BR/complexity.ts">

<violation number="1" location="packages/backend/src/scoring/keywords/locales/pt-BR/complexity.ts:8">
P2: PT-BR complexity locale uses a permissive map type and omits canonical dimensions (`questionComplexity`, `domainSpecificity`), allowing incomplete locale coverage to compile silently.</violation>
</file>

<file name="packages/backend/src/scoring/keywords/locales/pt-BR/web-browsing.ts">

<violation number="1" location="packages/backend/src/scoring/keywords/locales/pt-BR/web-browsing.ts:23">
P2: PT-BR web-browsing keywords use accented forms without unaccented variants, but trie matching is only case-insensitive (not accent-insensitive), so common unaccented inputs can be missed.</violation>
</file>

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.}

[antoniocarlos97ss]

Thanks for the detailed review, @cubic-dev-ai! All three issues are fixed in commit 0cd0956.

---

P1 ✅ — ISO-639-3 → BCP-47 normalisation

franc-min indeed returns ISO-639-3 codes (por, eng, spa…), not BCP-47. Added an ISO639_3_TO_BCP47 lookup table in detectLanguage() that normalises before the locale map lookup:

const ISO639_3_TO_BCP47: Record<string, string> = {
  por: 'pt',
  spa: 'es',
  fra: 'fr',
  // ... 11 more common languages pre-mapped
};
// ...
return ISO639_3_TO_BCP47[raw] ?? raw;

PT-BR auto-detection now actually resolves to the locale.

---

P2a ✅ — Strict ComplexityDimensions type with all 14 dimensions

Exported a new ComplexityDimensions type from locales/index.ts that explicitly lists all 14 canonical dimensions as required keys. The PT-BR complexity file now uses this type instead of the permissive Record<string, string[]>. TypeScript will now fail compilation if any future locale file omits a dimension.

Also added the two missing dimensions to pt-BR/complexity.ts:

• questionComplexity — 9 PT-BR phrases
• domainSpecificity — 18 PT-BR terms (lgpd, bayesiano, rede neural, blockchain, etc.)

---

P2b ✅ — Unaccented variants in web-browsing keywords

Rewrote pt-BR/web-browsing.ts to include both accented and unaccented variants for every keyword that has diacritics:

'preencha o formulario',  // unaccented — common on mobile
'preencha o formulário',  // accented
'va para',                // unaccented
'vá para',                // accented
// etc.

This covers the case where users type quickly or on devices that don't auto-correct to accented forms.

---

Let me know if there's anything else to address!

[cubic-dev-ai]

Thanks for the detailed review, @cubic-dev-ai! All three issues are fixed in commit 0cd0956.

---

P1 ✅ — ISO-639-3 → BCP-47 normalisation

...

@antoniocarlos97ss These fixes look excellent and make the multilingual support much more robust.

The ISO-639-3 normalization (P1) was a critical catch—without that mapping, franc-min would have returned por and failed to trigger the pt keyword set. The implementation in detectLanguage correctly handles this and still allows for the MANIFEST_LOCALE override if needed.

The move to the strict ComplexityDimensions type (P2a) is also a big improvement. It ensures that when someone adds Spanish or French later, the compiler will force them to provide keywords for all 14 dimensions, maintaining parity with the English scorer.

Finally, adding unaccented variants (P2b) to web-browsing.ts is the right call for real-world usage. Since the trie matching isn't accent-folded, users typing quickly on mobile (e.g., "abra a pagina" instead of "página") would have been missed. Applying this to the task-specific keywords ensures the specificity detection remains accurate across different typing styles.

I don't see any other issues—this is ready to go.

Settings | Documentation