Skip to content

Commit 72b2505

Browse files
chmjkbmdydek
authored andcommitted
fix: pass skipSpecialTokens param to Tokenizer in _TokenizerModule (#175)
## Description Currently, the native module accepts two params, namely tokens to decode and a flag whether we should skip special tokens. The _TokenizerModule wrapper's decode called Tokenizer.decode with just the tokens to decode param, resulting in wrong parameters being passed to native side. This PR solves that problem. ### Type of change - [x] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation update (improves or adds clarity to existing documentation) ### Tested on - [ ] iOS - [ ] Android ### Testing instructions <!-- Provide step-by-step instructions on how to test your changes. Include setup details if necessary. --> ### Screenshots <!-- Add screenshots here, if applicable --> ### Related issues <!-- Link related issues here using #issue-number --> ### Checklist - [ ] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have updated the documentation accordingly - [ ] My changes generate no new warnings ### Additional notes <!-- Include any additional information, assumptions, or context that reviewers might need to understand this PR. -->
1 parent 0f270a7 commit 72b2505

File tree

1 file changed

+2
-2
lines changed

1 file changed

+2
-2
lines changed

src/native/RnExecutorchModules.ts

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -279,8 +279,8 @@ class _TokenizerModule {
279279
async load(tokenizerSource: string): Promise<number> {
280280
return await Tokenizer.load(tokenizerSource);
281281
}
282-
async decode(input: number[]): Promise<string> {
283-
return await Tokenizer.decode(input);
282+
async decode(input: number[], skipSpecialTokens: boolean): Promise<string> {
283+
return await Tokenizer.decode(input, skipSpecialTokens);
284284
}
285285
async encode(input: string): Promise<number[]> {
286286
return await Tokenizer.encode(input);

0 commit comments

Comments
 (0)