Fix int16 TOSA.TABLE LUT zeroed when output range uses <16 bits (#20401)#20401
Draft
ryan-monroe wants to merge 1 commit into
Draft
Fix int16 TOSA.TABLE LUT zeroed when output range uses <16 bits (#20401)#20401ryan-monroe wants to merge 1 commit into
ryan-monroe wants to merge 1 commit into
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/20401
Note: Links to docs will display an error until the docs builds have been completed. ❌ 2 Pending, 3 Unrelated Failures, 2 Unclassified FailuresAs of commit 74ebdfc with merge base 6f8a889 ( UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:
FLAKY - The following jobs failed but were likely due to flakiness present on trunk:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
|
|
meta-cla
Bot
added
the
CLA Signed
Contributor
|
@ryan-monroe has exported this pull request. If you are a Meta employee, you can view the originating Diff in D107331163. |
This PR needs a
|
meta-codesync
Bot
changed the title
ryan-monroe
added a commit
to ryan-monroe/executorch
that referenced
this pull request
Jun 18, 2026
…rch#20401) Summary: `InsertTableOpsPass.generate_16_bit_table_values` builds the int16 `TOSA.TABLE` lookup for unary ops (sigmoid, tanh, ...). It computes `rshift = ceil(log2(max_table_value)) + 1 - 16` to fit the table into 16 signed bits, then does `lut_values >> rshift`, assuming the table fills ~16 bits (its own comment notes "for int16, rshift == 0"). When the op's output range uses fewer than 16 bits this breaks. A sigmoid output is in `[0, 1]`; quantized with the scale the observer picks (here `1/4096`), the largest table value is `4096` (13 bits), so `rshift = 13 - 16 = -3`. `lut_values >> -3` is an undefined negative right-shift; on the host the shift count is masked and the entire table is zeroed, so the on-device activation returns 0 for every input. On the Auth (ECAPA-TDNN) U85 model this zeroed the `Sigmoid` in every Squeeze-and-Excitation block, collapsing the SE channel-attention scale to 0 and dropping PTQ<->FVP SQNR from ~87 dB to 0 dB. The ideal table is a perfectly well-conditioned sigmoid ramp (`0 -> 2048 -> 4096`, 159 distinct levels) -- the shift, not the qparams, was the problem. Fix: clamp `rshift` to >= 0. When it would be negative the values already fit in int16, so no shift is needed; this restores the documented `rshift == 0` / `rescale_lshift == -7` case. The fix is general -- it covers any int16 `TABLE` op whose output range is small. This patches both copies of the pass (`xplat/executorch/...` and `fbcode/executorch/...`); the fbcode copy is the one fbcode test targets build, so both must stay in sync. A full writeup (localization, the LUT before/after, FVP verification, and two secondary findings -- a runner-memory HardFault and the graph-sim e2e path) is in this Google Doc: https://docs.google.com/document/d/1WGAR01mdnwOLKcQBtC4qzYLPWB1Lh4fp_8fHF4tXORE/edit Differential Revision: D107331163
ryan-monroe
force-pushed
the
export-D107331163
branch
from
894a5ec to
813a584
Compare
ryan-monroe
added a commit
to ryan-monroe/executorch
that referenced
this pull request
Jun 19, 2026
…rch#20401) Summary: `InsertTableOpsPass.generate_16_bit_table_values` builds the int16 `TOSA.TABLE` lookup for unary ops (sigmoid, tanh, ...). It computes `rshift = ceil(log2(max_table_value)) + 1 - 16` to fit the table into 16 signed bits, then does `lut_values >> rshift`, assuming the table fills ~16 bits (its own comment notes "for int16, rshift == 0"). When the op's output range uses fewer than 16 bits this breaks. A sigmoid output is in `[0, 1]`; quantized with the scale the observer picks (here `1/4096`), the largest table value is `4096` (13 bits), so `rshift = 13 - 16 = -3`. `lut_values >> -3` is an undefined negative right-shift; on the host the shift count is masked and the entire table is zeroed, so the on-device activation returns 0 for every input. On the Auth (ECAPA-TDNN) U85 model this zeroed the `Sigmoid` in every Squeeze-and-Excitation block, collapsing the SE channel-attention scale to 0 and dropping PTQ<->FVP SQNR from ~87 dB to 0 dB. The ideal table is a perfectly well-conditioned sigmoid ramp (`0 -> 2048 -> 4096`, 159 distinct levels) -- the shift, not the qparams, was the problem. Fix: clamp `rshift` to >= 0. When it would be negative the values already fit in int16, so no shift is needed; this restores the documented `rshift == 0` / `rescale_lshift == -7` case. The fix is general -- it covers any int16 `TABLE` op whose output range is small. This patches both copies of the pass (`xplat/executorch/...` and `fbcode/executorch/...`); the fbcode copy is the one fbcode test targets build, so both must stay in sync. A full writeup (localization, the LUT before/after, FVP verification, and two secondary findings -- a runner-memory HardFault and the graph-sim e2e path) is in this Google Doc: https://docs.google.com/document/d/1WGAR01mdnwOLKcQBtC4qzYLPWB1Lh4fp_8fHF4tXORE/edit Differential Revision: D107331163
ryan-monroe
force-pushed
the
export-D107331163
branch
2 times, most recently
from
2cda5ae to
b6383b4
Compare
ryan-monroe
added a commit
to ryan-monroe/executorch
that referenced
this pull request
Jun 22, 2026
…rch#20401) Summary: `InsertTableOpsPass.generate_16_bit_table_values` builds the int16 `TOSA.TABLE` lookup for unary ops (sigmoid, tanh, ...). It computes `rshift = ceil(log2(max_table_value)) + 1 - 16` to fit the table into 16 signed bits, then does `lut_values >> rshift`, assuming the table fills ~16 bits (its own comment notes "for int16, rshift == 0"). When the op's output range uses fewer than 16 bits this breaks. A sigmoid output is in `[0, 1]`; quantized with the scale the observer picks (here `1/4096`), the largest table value is `4096` (13 bits), so `rshift = 13 - 16 = -3`. `lut_values >> -3` is an undefined negative right-shift; on the host the shift count is masked and the entire table is zeroed, so the on-device activation returns 0 for every input. On the Auth (ECAPA-TDNN) U85 model this zeroed the `Sigmoid` in every Squeeze-and-Excitation block, collapsing the SE channel-attention scale to 0 and dropping PTQ<->FVP SQNR from ~87 dB to 0 dB. The ideal table is a perfectly well-conditioned sigmoid ramp (`0 -> 2048 -> 4096`, 159 distinct levels) -- the shift, not the qparams, was the problem. Fix: clamp `rshift` to >= 0. When it would be negative the values already fit in int16, so no shift is needed; this restores the documented `rshift == 0` / `rescale_lshift == -7` case. The fix is general -- it covers any int16 `TABLE` op whose output range is small. This patches both copies of the pass (`xplat/executorch/...` and `fbcode/executorch/...`); the fbcode copy is the one fbcode test targets build, so both must stay in sync. A full writeup (localization, the LUT before/after, FVP verification, and two secondary findings -- a runner-memory HardFault and the graph-sim e2e path) is in this Google Doc: https://docs.google.com/document/d/1WGAR01mdnwOLKcQBtC4qzYLPWB1Lh4fp_8fHF4tXORE/edit Differential Revision: D107331163
ryan-monroe
force-pushed
the
export-D107331163
branch
from
b6383b4 to
7a6db1f
Compare
ryan-monroe
pushed a commit
to ryan-monroe/executorch
that referenced
this pull request
Jun 30, 2026
…rch#20401) Summary: `InsertTableOpsPass.generate_16_bit_table_values` builds the int16 `TOSA.TABLE` lookup for unary ops (sigmoid, tanh, ...). It computes `rshift = ceil(log2(max_table_value)) + 1 - 16` to fit the table into 16 signed bits, then does `lut_values >> rshift`, assuming the table fills ~16 bits (its own comment notes "for int16, rshift == 0"). When the op's output range uses fewer than 16 bits this breaks. A sigmoid output is in `[0, 1]`; quantized with the scale the observer picks (here `1/4096`), the largest table value is `4096` (13 bits), so `rshift = 13 - 16 = -3`. `lut_values >> -3` is an undefined negative right-shift; on the host the shift count is masked and the entire table is zeroed, so the on-device activation returns 0 for every input. On the model this zeroed the `Sigmoid` in every Squeeze-and-Excitation block, collapsing the channel-attention scale to 0 and dropping PTQ<->FVP SQNR from ~87 dB to 0 dB. The ideal table is a perfectly well-conditioned sigmoid ramp (`0 -> 2048 -> 4096`, 159 distinct levels) -- the shift, not the qparams, was the problem. Fix: clamp `rshift` to >= 0. When it would be negative the values already fit in int16, so no shift is needed; this restores the documented `rshift == 0` / `rescale_lshift == -7` case. The fix is general -- it covers any int16 `TABLE` op whose output range is small. This patches both copies of the pass (`xplat/executorch/...` and `fbcode/executorch/...`); the fbcode copy is the one fbcode test targets build, so both must stay in sync. Differential Revision: D107331163
ryan-monroe
pushed a commit
to ryan-monroe/executorch
that referenced
this pull request
Jun 30, 2026
…rch#20401) Summary: `InsertTableOpsPass.generate_16_bit_table_values` builds the int16 `TOSA.TABLE` lookup for unary ops (sigmoid, tanh, ...). It computes `rshift = ceil(log2(max_table_value)) + 1 - 16` to fit the table into 16 signed bits, then does `lut_values >> rshift`, assuming the table fills ~16 bits (its own comment notes "for int16, rshift == 0"). When the op's output range uses fewer than 16 bits this breaks. A sigmoid output is in `[0, 1]`; quantized with a small scale (e.g. `1/4096`), the largest table value is `4096` (13 bits), so `rshift = 13 - 16 = -3`. `lut_values >> -3` is an undefined negative right-shift; on the host the shift count is masked and the entire table is zeroed, so the activation returns 0 for every input. This makes any int16 `TABLE` op with a small output range (e.g. a sigmoid in a Squeeze-and-Excitation block) degenerate. Fix: clamp `rshift` to >= 0. When it would be negative the values already fit in int16, so no shift is needed; this restores the documented `rshift == 0` / `rescale_lshift == -7` case. The fix is general -- it covers any int16 `TABLE` op whose output range is small. Differential Revision: D107331163
ryan-monroe
force-pushed
the
export-D107331163
branch
from
398e060 to
4d52808
Compare
…rch#20401) Summary: `InsertTableOpsPass.generate_16_bit_table_values` builds the int16 `TOSA.TABLE` lookup for unary ops (sigmoid, tanh, ...). It computes `rshift = ceil(log2(max_table_value)) + 1 - 16` to fit the table into 16 signed bits, then does `lut_values >> rshift`, assuming the table fills ~16 bits (its own comment notes "for int16, rshift == 0"). When the op's output range uses fewer than 16 bits this breaks. A sigmoid output is in `[0, 1]`; quantized with a small scale (e.g. `1/4096`), the largest table value is `4096` (13 bits), so `rshift = 13 - 16 = -3`. `lut_values >> -3` is an undefined negative right-shift; on the host the shift count is masked and the entire table is zeroed, so the activation returns 0 for every input. This makes any int16 `TABLE` op with a small output range (e.g. a sigmoid in a Squeeze-and-Excitation block) degenerate. Fix: clamp `rshift` to >= 0. When it would be negative the values already fit in int16, so no shift is needed; this restores the documented `rshift == 0` / `rescale_lshift == -7` case. The fix is general -- it covers any int16 `TABLE` op whose output range is small. Differential Revision: D107331163
ryan-monroe
force-pushed
the
export-D107331163
branch
from
4d52808 to
74ebdfc
Compare
Contributor
|
Converting the draft as I cannot re-export the commandeered diff to this PR due to owner is on PTO. Will create new PR to get around this problem |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
InsertTableOpsPass.generate_16_bit_table_valuesbuilds the int16TOSA.TABLElookup for unary ops (sigmoid, tanh, ...). It computesrshift = ceil(log2(max_table_value)) + 1 - 16to fit the table into 16 signed bits, then doeslut_values >> rshift, assuming the table fills ~16 bits (its own comment notes "for int16, rshift == 0").When the op's output range uses fewer than 16 bits this breaks. A sigmoid output is in
[0, 1]; quantized with a small scale (e.g.1/4096), the largest table value is4096(13 bits), sorshift = 13 - 16 = -3.lut_values >> -3is an undefined negative right-shift; on the host the shift count is masked and the entire table is zeroed, so the activation returns 0 for every input. This makes any int16TABLEop with a small output range (e.g. a sigmoid in a Squeeze-and-Excitation block) degenerate.Fix: clamp
rshiftto >= 0. When it would be negative the values already fit in int16, so no shift is needed; this restores the documentedrshift == 0/rescale_lshift == -7case. The fix is general -- it covers any int16TABLEop whose output range is small.Differential Revision: D107331163