Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
76 commits
Select commit Hold shift + click to select a range
f943d5d
Create route for lexicon page
aidanpscott Nov 11, 2024
3dffb58
Removed page.ts
EthanFennell Feb 15, 2025
a94997e
Changed use of "slot" to "render" for navbar
EthanFennell Feb 15, 2025
702d50b
List of words
EthanFennell Feb 16, 2025
e4fade1
Fix mobile layout
aidanpscott Feb 20, 2025
1df629e
Fix lexicon auto load
aidanpscott Feb 21, 2025
395f895
Fix word id mismatch
MrCars0n Feb 21, 2025
775a0c0
Add margins to alphabet strip
aidanpscott Feb 21, 2025
d033725
Add first draft of vernacular letter id handling
MrCars0n Feb 21, 2025
7a60099
Restore dictionary functionality while preserving styling
aidanpscott Feb 21, 2025
acc6751
Fixed vernacular letter id handling
MrCars0n Feb 21, 2025
ba1bf5d
Create XML element and reorganize
MrCars0n Feb 24, 2025
c5d9837
Fix loading of second letter on tab change
MrCars0n Feb 24, 2025
c4a298e
Fix alphabet bar navigation bug
MrCars0n Feb 24, 2025
65230fe
Store selectedLanguage after XMLView
MrCars0n Mar 1, 2025
ff8ade0
Add word subtexts, indexes, and alphabet strip size change
MrCars0n Mar 2, 2025
8e8e9db
Add word navigation component
aidanpscott Mar 3, 2025
00f5160
Add XML new lines
MrCars0n Mar 3, 2025
06fa8a2
Add multi-index XML call and new line formatting
MrCars0n Mar 5, 2025
b88233d
Bug fixes
EthanFennell Mar 17, 2025
0ad9475
Moved vernacular query to page.js, simplified page.svelte
EthanFennell Mar 17, 2025
17909f8
Revert to threshold
EthanFennell Mar 17, 2025
586f27d
Fixed lazy loading glitches on firefox
EthanFennell Mar 23, 2025
177eace
Fixed sqlite fetch issue
EthanFennell Mar 24, 2025
ae26814
add single entry style type
AslanRules Mar 24, 2025
4ff134a
Merge branch 'feature/display-reversals/685' of github.com:sillsdev/a…
AslanRules Mar 24, 2025
c0664e7
Add hyperlinks to headwords in XML
MrCars0n Mar 28, 2025
3b36c1b
Remove unused variable and comments
MrCars0n Mar 28, 2025
9461d04
show single entry styles
AslanRules Mar 31, 2025
d9ad301
Added stores. Refactored lexicon code. Included reversal/vernacular m…
EthanFennell Apr 2, 2025
c95120b
Merge and formatting
EthanFennell Apr 4, 2025
c616955
Applied suggestions
EthanFennell Apr 4, 2025
4d70312
Small changes
EthanFennell Apr 4, 2025
f1392b7
Small changes
EthanFennell Apr 4, 2025
0f55156
miscellaneous small refactors
AslanRules Apr 4, 2025
071ec65
Format
EthanFennell Apr 7, 2025
b88c6c3
Fix test
EthanFennell Apr 9, 2025
ef9aaf7
Fix accessibility issues
aidanpscott Apr 9, 2025
ba77bdc
Add fallback for summary without matches
MrCars0n Apr 9, 2025
4531462
Fix comma issue. Fix duplicate listeners
EthanFennell Apr 9, 2025
7a5cb54
Add proper type definition for vernacularWordsStore
MrCars0n Apr 9, 2025
1585440
Improve store definitions
MrCars0n Apr 9, 2025
fc0081c
Vernacular word hyperlink fix
MrCars0n Apr 9, 2025
9dc61e4
Fix lint issues
chrisvire Apr 10, 2025
facc036
Fix language tab to match native app
aidanpscott Apr 10, 2025
0300807
Remove padding
MrCars0n Apr 10, 2025
4bed4c7
Applied feedback for language tabs
EthanFennell Apr 11, 2025
e2fcdb6
fix lint
chrisvire Apr 11, 2025
c9b048c
Fix spacing
EthanFennell Apr 11, 2025
d0a9562
Adjust word list spacing
aidanpscott Apr 11, 2025
2dccab3
Fix navbar hiding on scroll and div organization
MrCars0n Apr 13, 2025
e848ba0
Fix more div organizaiton
MrCars0n Apr 13, 2025
a791eeb
Spacing fix
MrCars0n Apr 13, 2025
d2f7d05
Implement lexicon error handling
aidanpscott Apr 14, 2025
1d141aa
Implement fetch logic for reversal view
aidanpscott Apr 14, 2025
e82a119
Fix a potential security vulnerability
aidanpscott Apr 14, 2025
0fa7c90
fix lint
chrisvire Apr 14, 2025
40e9333
Remove hardcoded colors
aidanpscott Apr 15, 2025
bdf07c1
Split LexiconReversalListView into separate components.
chrisvire Apr 16, 2025
52859d0
Remove hardcoded color
aidanpscott Apr 16, 2025
c2f7e40
Remove alphabetStrip artifact
MrCars0n Apr 16, 2025
da80dab
Add proper reversal file fetch
aidanpscott Apr 16, 2025
23fa6e5
Remove phantom endpoint call
MrCars0n Apr 16, 2025
6ff83a3
Add grid view
MrCars0n Apr 16, 2025
843b73a
Remove HEAD request in file loading
aidanpscott Apr 16, 2025
92eedb1
Remove unnecessary comments
aidanpscott Apr 16, 2025
248cb73
Add grid view
MrCars0n Apr 16, 2025
cc26d25
Add grid view
MrCars0n Apr 16, 2025
a0c31de
Remove redundancy
aidanpscott Apr 16, 2025
4231b70
convertReversalIndex create index.json
chrisvire Apr 16, 2025
3ae4202
Use ReversalIndex to load reversal files
chrisvire Apr 16, 2025
ab03d14
Fix wide screen gap
MrCars0n Apr 17, 2025
14c5e45
Fix display on wide screen
chrisvire Apr 18, 2025
be9693a
Rename AlphabetStrip
chrisvire Apr 18, 2025
144d55e
Rename LexiconReversalView
chrisvire Apr 18, 2025
a41d30a
Rename LexiconXMLView
chrisvire Apr 18, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions config/index.d.ts
Original file line number Diff line number Diff line change
Expand Up @@ -285,4 +285,11 @@ export type DictionaryConfig = AppConfig & {
displayed: boolean;
};
};
singleEntryStyles?: {
name: string;
category?: string;
properties: {
[key: string]: string;
};
}[];
};
9 changes: 9 additions & 0 deletions convert/convertConfig.ts
Original file line number Diff line number Diff line change
Expand Up @@ -248,6 +248,15 @@ function convertConfig(dataDir: string, verbose: number) {
const mainStyles = document.querySelector('styles')!;
data.styles = parseStyles(mainStyles, verbose);

if (isDictionaryConfig(data)) {
const singleEntryStyles = document.querySelector('styles[type=single-entry]');
if (singleEntryStyles) {
data.singleEntryStyles = parseStyles(singleEntryStyles, verbose);
} else if (verbose) {
console.log('No single-entry styles found in the XML document');
}
}

if (isScriptureConfig(data)) {
data.traits = parseTraits(document, dataDir, verbose);
data.bookCollections = parseBookCollections(document, verbose);
Expand Down
13 changes: 12 additions & 1 deletion convert/convertReverseIndex.ts
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ export function convertReverseIndex(
.filter(([gloss]) => gloss?.length > 0);

const entriesByLetter: { [letter: string]: [string, string][] } = {};
const fileIndexByLetter: { [letter: string]: string[] } = {}; // New index for files by letter

let latestLetter = makeEntryLetter(alphabet[0]);
indexEntries.forEach((entry) => {
Expand All @@ -57,14 +58,17 @@ export function convertReverseIndex(

const firstLetter = getBaseLetter(gloss, alphabet);
const entryLetter = firstLetter ?? latestLetter;
const fileEntryLetter = entryLetter.toLowerCase();
if (!entriesByLetter[entryLetter]) {
entriesByLetter[entryLetter] = [];
fileIndexByLetter[fileEntryLetter] = []; // Initialize file index for the letter
}
entriesByLetter[entryLetter].push([entry[0], entry[1]]);
latestLetter = entryLetter;
});

Object.entries(entriesByLetter).forEach(([letter, entries]) => {
const fileLetter = letter.toLowerCase();
entries.sort(([a], [b]) => a.localeCompare(b, language));

let currentChunk: { [key: string]: ReversalEntry[] } = {};
Expand Down Expand Up @@ -99,14 +103,16 @@ export function convertReverseIndex(
currentCount++;

if (currentCount >= ENTRIES_PER_CHUNK || i === entries.length - 1) {
const chunkFileName = `${letter.toLowerCase()}-${String(chunkIndex + 1).padStart(3, '0')}.json`;
const chunkFileName = `${fileLetter}-${String(chunkIndex + 1).padStart(3, '0')}.json`;
const chunkPath = path.join(outputDir, chunkFileName);

files.push({
path: chunkPath,
content: JSON.stringify(currentChunk, null, 2)
});

fileIndexByLetter[fileLetter].push(chunkFileName); // Add file to index

currentChunk = {};
currentCount = 0;
chunkIndex++;
Expand All @@ -115,6 +121,11 @@ export function convertReverseIndex(
}
});

files.push({
path: path.join(outputDir, 'index.json'),
content: JSON.stringify(fileIndexByLetter, null, 2) // Write the file index to index.json
});

return files;
}

Expand Down
2 changes: 1 addition & 1 deletion convert/tests/sab/convertConfigSAB.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -196,7 +196,7 @@ if (programType === 'DAB') {

test('convertConfig: parse features', () => {
const result = parseFeatures(document, 1);
expect(Object.keys(result)).toHaveLength(137);
expect(Object.keys(result)).toHaveLength(140);
});

test('convertConfig: parse background images', () => {
Expand Down
24 changes: 24 additions & 0 deletions src/lib/components/LexiconAlphabetStrip.svelte
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
<script lang="ts">
export let alphabet: string[];
export let activeLetter: string;
export let onLetterSelect: (letter: string) => void;
</script>

<div
class="flex m-2 gap-1 md:gap-2 mb-4 justify-start overflow-x-auto whitespace-nowrap pb-2 snap-x"
>
{#each alphabet as letter}
<button
class="px-3 py-2 text-sm font-bold border rounded-md bg-gray-100 hover:bg-gray-200 cursor-pointer snap-start
sm:px-4 sm:py-3 sm:text-base
md:px-5 md:py-4 md:text-base
lg:px-6 lg:py-4 lg:text-lg"
style={activeLetter === letter
? 'background-color: var(--TitleBackgroundColor); border-color: black;'
: ''}
on:click={() => onLetterSelect(letter)}
>
{letter}
</button>
{/each}
</div>
188 changes: 188 additions & 0 deletions src/lib/components/LexiconEntryView.svelte
Original file line number Diff line number Diff line change
@@ -0,0 +1,188 @@
<script>
import { base } from '$app/paths';
import config from '$lib/data/config';
import { convertStyle } from '$lib/data/stores';
import initSqlJs from 'sql.js';
import { afterUpdate, onMount } from 'svelte';

export let selectedWord;
export let vernacularWordsList;
export let vernacularLanguage;
export let onSelectWord;
export let onSwitchLanguage;

let xmlData = '';

let singleEntryStyles = config.singleEntryStyles;

async function queryXmlByWordId(wordId) {
try {
const SQL = await initSqlJs({
locateFile: (file) => `${base}/wasm/sql-wasm.wasm`
});

const response = await fetch(`${base}/data.sqlite`);
if (!response.ok) {
throw new Error(
`Failed to fetch database: ${response.status} ${response.statusText}`
);
}
const buffer = await response.arrayBuffer();
const db = new SQL.Database(new Uint8Array(buffer));
if (!db) {
console.error('Database not initialized');
Comment on lines +18 to +33
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

⚠️ Potential issue

Avoid re‑initialising sql.js and re‑downloading the DB on every word lookup

initSqlJs and the fetch for data.sqlite are executed inside queryXmlByWordId, which means a full WASM initialisation and DB download happens for each word ID (and for every component update).
This is extremely expensive (multi‑MB download + WASM compile) and becomes noticeable as soon as a user clicks through several entries.

Create a module‑level (or store‑level) singleton that initialises SQL and keeps a Database instance open for the life‑time of the session, then reuse it:

-let SQL;
-let db;
+let SQL;              // undefined until first call
+let db;               // cache DB instance

async function queryXmlByWordId(wordId) {
     try {
-        const SQL = await initSqlJs({ locateFile: … });
-        const response = await fetch(`${base}/data.sqlite`);
-
-        const db = new SQL.Database(new Uint8Array(buffer));
+        if (!SQL) {
+            SQL = await initSqlJs({ locateFile: (f) => `${base}/wasm/${f}` });
+            const response = await fetch(`${base}/data.sqlite`);
+            const buffer = await response.arrayBuffer();
+            db = new SQL.Database(new Uint8Array(buffer));
+        }

This reduces network traffic, CPU time, and memory dramatically.

Committable suggestion skipped: line range outside the PR's diff.

return null;
}

const stmt = db.prepare('SELECT xml FROM entries WHERE id = ?');
stmt.bind([wordId]);

let result = null;
if (stmt.step()) {
result = stmt.getAsObject().xml;
}
stmt.free();
db.close();

return result;
} catch (error) {
console.error(`Error querying XML for word ID ${wordId}:`, error);
return null;
}
}

function formatXmlByClass(xmlString) {
if (!xmlString) return '';

const parser = new DOMParser();
const xmlDoc = parser.parseFromString(xmlString, 'text/xml');

// Check if parsing failed
const parseError = xmlDoc.querySelector('parsererror');
if (parseError) {
console.error('XML parsing error:', parseError.textContent);
return `<span class="text-error">Error parsing XML: Invalid format</span>`;
}

function processNode(node, parentHasSenseNumber = false) {
Comment on lines +54 to +67
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Potential XSS – XML → HTML is injected without sanitisation

formatXmlByClass blindly reconstructs HTML from the parsed XML and injects it with {@html}.
If the XML contains <script> or dangerous attributes (on*, javascript: URLs, etc.) they will be executed.

Please run the output through an HTML sanitizer (e.g. DOMPurify) or explicitly strip/escape unsafe tags and attributes before exposing them to the DOM.

let output = '';

if (node.nodeType === Node.TEXT_NODE) {
return node.nodeValue.trim() ? node.nodeValue + ' ' : '';
}

if (node.nodeType === Node.ELEMENT_NODE) {
let className = node.getAttribute('class') || '';
let isSenseNumber = className.includes('sensenumber');

let parentContainsSenseNumber =
parentHasSenseNumber ||
[...node.parentNode.children].some(
(child) =>
child.getAttribute &&
(child.getAttribute('class') || '').includes('sensenumber')
);

if (node.tagName === 'a' && node.hasAttribute('href')) {
const href = node.getAttribute('href');
const match = href.match(/E-(\d+)/); // Extract index number
if (match) {
const index = parseInt(match[1], 10); // Extracted number as integer
const wordObject = vernacularWordsList.find((item) => item.id === index);
const word = wordObject ? wordObject.name : 'Unknown'; // Fallback if not found
const homonymIndex = wordObject ? wordObject.homonym_index : 1; // Default to 1 if not found

let linkText = node.textContent.trim();

// If the text inside the link matches the homonym index, use the homonym index as the text
if (linkText === String(homonymIndex)) {
linkText = homonymIndex.toString();
}

output += `<span class="clickable cursor-pointer" data-word="${word}" data-index="${index}" data-homonym="${homonymIndex}">${linkText}</span>`;
return output;
}
} else {
output += '<' + node.tagName;
for (let attr of node.attributes) {
output += ` ${attr.name}="${attr.value}"`;
}
output += '>';

for (let child of node.childNodes) {
output += processNode(child, parentContainsSenseNumber || isSenseNumber);
}

output += `</${node.tagName}>`;
}
}

return output;
}

return processNode(xmlDoc.documentElement);
}

async function updateXmlData() {
if (
!selectedWord ||
(!selectedWord.index && (!selectedWord.indexes || selectedWord.indexes.length === 0))
) {
xmlData = '';
return;
}

Comment on lines +126 to +134
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

selectedWord.index may legally be 0 – current check treats it as “missing”

!selectedWord.index is falsy for 0, so an entry whose index is zero will never be shown.
Use an explicit null/undefined test instead:

-if (
-    !selectedWord ||
-    (!selectedWord.index && (!selectedWord.indexes || selectedWord.indexes.length === 0))
-)
+if (
+    !selectedWord ||
+    (selectedWord.index == null && (!selectedWord.indexes || selectedWord.indexes.length === 0))
+)
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
async function updateXmlData() {
if (
!selectedWord ||
(!selectedWord.index && (!selectedWord.indexes || selectedWord.indexes.length === 0))
) {
xmlData = '';
return;
}
async function updateXmlData() {
if (
!selectedWord ||
(selectedWord.index == null && (!selectedWord.indexes || selectedWord.indexes.length === 0))
) {
xmlData = '';
return;
}

let wordIds = selectedWord.indexes ? selectedWord.indexes : [selectedWord.index];
let xmlResults = await Promise.all(wordIds.map(queryXmlByWordId));

// Insert an `<hr>` tag or a visible separator between entries
xmlData =
xmlResults
.filter((xml) => xml) // Ensure no null values are included
.map(formatXmlByClass)
.join('\n<hr>\n') + '\n<hr>\n'; // `<hr>` adds a visible line between entries
}

function attachEventListeners() {
const spans = document.querySelectorAll('.clickable');

spans.forEach((span) => {
const oldSpan = span.cloneNode(true);
span.parentNode.replaceChild(oldSpan, span);
});

const freshSpans = document.querySelectorAll('.clickable');
freshSpans.forEach((span) => {
span.addEventListener('click', () => {
onSwitchLanguage(vernacularLanguage);
const word = span.getAttribute('data-word');
const index = parseInt(span.getAttribute('data-index'), 10);
const homonym_index = parseInt(span.getAttribute('data-homonym'), 10);

onSelectWord({
word,
index,
homonym_index
});
});
});
}

function applyStyles() {
for (let stl of singleEntryStyles) {
for (let elm of document.querySelectorAll(stl.name)) {
elm.style = convertStyle(stl.properties);
}
}
Comment on lines +172 to +176
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Inline‑style assignment is wrong – use style.cssText or setAttribute

elm.style is a CSSStyleDeclaration, not a string. Assigning a string to it is a no‑op in most browsers.

-elm.style = convertStyle(stl.properties);
+elm.style.cssText = convertStyle(stl.properties);   // or elm.setAttribute('style', …)

Without this fix none of the configured styles are actually applied.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
for (let stl of singleEntryStyles) {
for (let elm of document.querySelectorAll(stl.name)) {
elm.style = convertStyle(stl.properties);
}
}
for (let stl of singleEntryStyles) {
for (let elm of document.querySelectorAll(stl.name)) {
- elm.style = convertStyle(stl.properties);
+ elm.style.cssText = convertStyle(stl.properties); // or elm.setAttribute('style', …)
}
}

}

onMount(updateXmlData);

afterUpdate(() => {
updateXmlData();
applyStyles();
attachEventListeners();
});
Comment on lines +181 to +185
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

afterUpdate triggers an expensive DB fetch every render

afterUpdate runs on every component update; you call updateXmlData() (async) each time, causing needless DB queries and re‑render loops.

Prefer a reactive statement that depends only on selectedWord / vernacularWordsList:

$: if (selectedWord) {
    await updateXmlData();
    await tick();          // wait for DOM update
    applyStyles();
    attachEventListeners();
}

This fires exactly when required and avoids thrashing.

</script>

<pre class="p-4 whitespace-pre-wrap break-words min-w-[100vw]">{@html xmlData}</pre>
45 changes: 45 additions & 0 deletions src/lib/components/LexiconLanguageTabs.svelte
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
<script>
import { expoInOut } from 'svelte/easing';
import { fly } from 'svelte/transition';

export let reversalLanguage;
export let selectedLanguage;
export let onSwitchLanguage;
export let vernacularLanguage;
</script>

<div class="flex w-full" style="background-color: var(--TabBackgroundColor);">
<div
role="button"
tabindex="0"
aria-pressed={selectedLanguage === vernacularLanguage}
on:click={() => onSwitchLanguage(vernacularLanguage)}
on:keydown={(e) => e.key === 'Enter' && onSwitchLanguage(vernacularLanguage)}
class="py-2.5 px-3.5 text-sm uppercase text-center relative dy-tabs dy-tabs-bordered mb-1"
>
{vernacularLanguage}
{#if selectedLanguage === vernacularLanguage}
<div
transition:fly={{ axis: 'x', easing: expoInOut, x: 70 }}
class="absolute -bottom-1 left-0 w-full h-1 bg-black"
></div>
{/if}
</div>
<div
role="button"
tabindex="0"
aria-pressed={selectedLanguage === reversalLanguage}
on:click={() => onSwitchLanguage(reversalLanguage)}
on:keydown={(e) => e.key === 'Enter' && onSwitchLanguage(reversalLanguage)}
class="py-2.5 px-3.5 text-sm uppercase text-center relative dy-tabs dy-tabs-bordered mb-1"
>
{reversalLanguage}
{#if selectedLanguage === reversalLanguage}
<div
transition:fly={{ axis: 'x', easing: expoInOut, x: -70 }}
class="absolute -bottom-1 left-0 w-full h-1 bg-black"
></div>
{/if}
</div>
<div class="flex-1"></div>
</div>
35 changes: 35 additions & 0 deletions src/lib/components/LexiconListViewHeader.svelte
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
<script>
import LexiconAlphabetStrip from './LexiconAlphabetStrip.svelte';
import LexiconLanguageTabs from './LexiconLanguageTabs.svelte';

export let alphabet = [];
export let selectedLanguage;
export let vernacularLanguage;
export let reversalLanguage;
export let selectedLetter;
export let onSwitchLanguage;
export let onLetterChange;

let currentLetter = alphabet[0];

async function handleLetterSelect(letter) {
currentLetter = letter;
onLetterChange(letter);
}

$: if (alphabet && alphabet.length > 0) {
currentLetter = alphabet[0];
}
$: if (selectedLetter !== currentLetter) {
currentLetter = selectedLetter;
}
</script>

<LexiconLanguageTabs
{reversalLanguage}
{selectedLanguage}
{onSwitchLanguage}
{vernacularLanguage}
/>

<LexiconAlphabetStrip {alphabet} activeLetter={currentLetter} onLetterSelect={handleLetterSelect} />
Comment thread
chrisvire marked this conversation as resolved.
Loading