Skip to content

[Cherry-Pick][CI][BugFix] Fix rotary_embs initialization(#8012)#8017

Merged
EmmonsCurse merged 1 commit into
PaddlePaddle:release/2.6from
EmmonsCurse:cherry-pick/8012/release/2.6
Jun 8, 2026
Merged

[Cherry-Pick][CI][BugFix] Fix rotary_embs initialization(#8012)#8017
EmmonsCurse merged 1 commit into
PaddlePaddle:release/2.6from
EmmonsCurse:cherry-pick/8012/release/2.6

Conversation

@EmmonsCurse

@EmmonsCurse EmmonsCurse commented Jun 8, 2026

Copy link
Copy Markdown
Collaborator

Cherry-pick of #8012 (authored by @EmmonsCurse) to release/2.6.

devPR:#8012


Motivation

A CUDA kernel out-of-bounds issue was observed when running in environments with the 580 driver stack. The problem may lead to test failures, runtime instability, or unexpected kernel execution behavior under specific workloads.

This change addresses the compatibility issue to ensure stable execution in 580 driver environments.
tests/layers/test_plas_attention.py fails with CUDA illegal memory access errors during execution:

CUDA error(700): an illegal memory access was encountered

Investigation using compute-sanitizer --tool memcheck revealed that the failures originate from out-of-bounds memory accesses inside fused_block_mean_and_rope_kernel.

The test initializes rotary_embs using:

paddle.ones([2, seq_len, head_dim // 2])

However, the kernel computes the sin buffer offset using plas_max_seq_length rather than the actual test seq_len. As a result, the kernel accesses memory beyond the allocated buffer, causing CUDA illegal memory access and corrupting the CUDA context. Once the illegal access occurs, subsequent GPU operations also fail, which explains why test_server reports the same error during initialization.

In addition, the test validation expects RoPE to behave as an identity transformation, which requires:

  • cos = 1
  • sin = 0

The previous initialization incorrectly set both cosine and sine values to 1.

Modifications

  • Updated rotary_embs allocation to use plas_max_seq_length instead of seq_len.
  • Initialized RoPE buffers explicitly:
  • cosine values set to 1
  • sine values set to 0
  • Ensured the buffer layout matches kernel offset calculations and prevents out-of-bounds memory access.
  • Restored identity RoPE behavior required by the test validation logic.
  • Fixed CUDA illegal memory access failures in:
  • test_plas_attention
  • test_server

Usage or Command

N/A

Accuracy Tests

N/A

Checklist

  • Add at least a tag in the PR title.
  • Tag list: [[FDConfig],[APIServer],[Engine], [Scheduler], [PD Disaggregation], [Executor], [Graph Optimization], [Speculative Decoding], [RL], [Models], [Quantization], [Loader], [OP], [KVCache], [DataProcessor], [BugFix], [Docs], [CI], [Optimization], [Feature], [Benchmark], [Others], [XPU], [HPU], [GCU], [DCU], [Iluvatar], [Metax]]
  • You can add new tags based on the PR content, but the semantics must be clear.
  • Format your code, run pre-commit before commit.
  • Add unit tests. Please write the reason in this PR if no unit tests.
  • Provide accuracy results.
  • If the current PR is submitting to the release branch, make sure the PR has been submitted to the develop branch, then cherry-pick it to the release branch with the [Cherry-Pick] PR tag.

@EmmonsCurse

Copy link
Copy Markdown
Collaborator Author

/skip-ci ci_iluvatar
/skip-ci ci_hpu
/skip-ci build_xpu
/skip-ci gpu_4cards_test
/skip-ci pre_ce_test
/skip-ci stable_test
/skip-ci base_test
/skip-ci logprob_test

@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (release/2.6@1c5ec61). Learn more about missing BASE report.

Additional details and impacted files
@@              Coverage Diff               @@
##             release/2.6    #8017   +/-   ##
==============================================
  Coverage               ?   72.12%           
==============================================
  Files                  ?      386           
  Lines                  ?    55715           
  Branches               ?     8748           
==============================================
  Hits                   ?    40186           
  Misses                 ?    12691           
  Partials               ?     2838           
Flag Coverage Δ
GPU 72.12% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@EmmonsCurse
EmmonsCurse merged commit 3a639b2 into PaddlePaddle:release/2.6 Jun 8, 2026
36 checks passed
@EmmonsCurse
EmmonsCurse deleted the cherry-pick/8012/release/2.6 branch June 8, 2026 09:02
@paddle-bot

paddle-bot Bot commented Jul 13, 2026

Copy link
Copy Markdown

Thanks for your contribution!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants