Skip to content

Commit a245872

Browse files
authored
fix: pass skipSpecialTokens param to Tokenizer in _TokenizerModule (#175)
## Description Currently, the native module accepts two params, namely tokens to decode and a flag whether we should skip special tokens. The _TokenizerModule wrapper's decode called Tokenizer.decode with just the tokens to decode param, resulting in wrong parameters being passed to native side. This PR solves that problem. ### Type of change - [x] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation update (improves or adds clarity to existing documentation) ### Tested on - [ ] iOS - [ ] Android ### Testing instructions <!-- Provide step-by-step instructions on how to test your changes. Include setup details if necessary. --> ### Screenshots <!-- Add screenshots here, if applicable --> ### Related issues <!-- Link related issues here using #issue-number --> ### Checklist - [ ] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have updated the documentation accordingly - [ ] My changes generate no new warnings ### Additional notes <!-- Include any additional information, assumptions, or context that reviewers might need to understand this PR. -->
1 parent 6c8a629 commit a245872

File tree

2 files changed

+3
-3
lines changed

2 files changed

+3
-3
lines changed

src/controllers/SpeechToTextController.ts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -334,7 +334,7 @@ export class SpeechToTextController {
334334

335335
private async tokenIdsToText(tokenIds: number[]): Promise<string> {
336336
try {
337-
return this.nativeTokenizer.decode(tokenIds);
337+
return this.nativeTokenizer.decode(tokenIds, true);
338338
} catch (e) {
339339
this.onErrorCallback?.(
340340
new Error(`An error has ocurred when decoding the token ids: ${e}`)

src/native/RnExecutorchModules.ts

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -279,8 +279,8 @@ class _TokenizerModule {
279279
async load(tokenizerSource: string): Promise<number> {
280280
return await Tokenizer.load(tokenizerSource);
281281
}
282-
async decode(input: number[]): Promise<string> {
283-
return await Tokenizer.decode(input);
282+
async decode(input: number[], skipSpecialTokens: boolean): Promise<string> {
283+
return await Tokenizer.decode(input, skipSpecialTokens);
284284
}
285285
async encode(input: string): Promise<number[]> {
286286
return await Tokenizer.encode(input);

0 commit comments

Comments
 (0)