Kernel dies executing sample 26 (layernorm + ReLU bitmask) after cudnn.pygraph IR rewrite (fd2cdd18)

[liujane-dev]

Describe the bug
The Python sample samples/python/26_layernorm_forward_training_and_backward_with_relu_bitmask.ipynb crashes the Python/CUDA process on Blackwell — nbclient reports DeadKernelError: Kernel died (a native abort, not a Python exception or a CUDNN_STATUS_* error). The CI job py_samples:ga: [Blackwell|Hopper|Ampere] runs the sample notebooks sequentially under set -e; notebooks 00–25, 27–28, 50–53 execute and pass, then the kernel dies while executing notebook 26, failing the job.

Bisect isolates this to a pure frontend regression (cuDNN backend is pinned at 9.24.0.43 across good and bad):

• Good: cudnn_frontend 305de065 → notebook 26 passes,
• Good: cudnn_frontend 07679e28 → py_samples passes,
• Bad: cudnn_frontend fd2cdd18 → kernel dies

The culprit is fd2cdd18 — "Python-native cudnn.pygraph: graph IR + pluggable execution backends (#336)". It renames the C++ graph binding (python/pygraph/pygraph.cpp: py::class_<PyGraph>(m, "pygraph") → "backend_graph") and makes the public cudnn.pygraph a new Python-native IR class (python/cudnn/_pygraph.py) that lazily lowers to the C++ builder via _lower_to_cpp() at build()/execute() time. Every sample constructs its graph through cudnn.pygraph(...), so all of them now route through this new lowering layer. The layer is not exercise-complete: it handles the other notebooks but crashes at native level when lowering notebook 26's layernorm + ReLU bitmask fusion graph. Notebook 26 itself is unchanged in the 305de065..fd2cdd18 range, which rules out a test-side cause.

Expected behavior
Notebook 26 (layernorm forward-training + backward with ReLU bitmask fusion) builds and executes successfully on Blackwell through the new cudnn.pygraph IR / _lower_to_cpp() lowering path, exactly as it did prior to fd2cdd18, and the py_samples:ga: xxx job passes.

System Environment (please complete the following information):

• cudnn_frontend version: fd2cdd1 (wheel nvidia_cudnn_frontend-1.26.0), regressed from good 305de06
• cudnn_backend version: 9.24.0.43 (baked into container image cudnn_frontend:cudnn_9.24.0.43_13.3.0)
• GPU arch: Blackwell (SM 10.0)
• cuda runtime version: 13.3
• cuda driver version: 610.43.02 (forward-compat mode; host kernel driver 580.100)
• host compiler: container default (NVIDIA PyTorch 26.06, Python 3.12) — not separately captured
• OS: Ubuntu (NVIDIA PyTorch container, Release 26.06)

API logs
FE/BE API logs were NOT captured

To Reproduce
Steps to reproduce the behavior:

1. Check out cudnn_frontend at fd2cdd1 (or develop at/after it) with cuDNN backend 9.24.0.43, CUDA 13.3, on a Blackwell (SM 10.0) GPU.
2. pip install -v .
3. Run the single failing notebook:
   jupyter execute samples/python/26_layernorm_forward_training_and_backward_with_relu_bitmask.ipynb
4. Kernel dies mid-execution → nbclient.exceptions.DeadKernelError: Kernel died.

Additional context

• First bad commit: fd2cdd1 — "Python-native cudnn.pygraph: graph IR + pluggable execution backends (Python-native cudnn.pygraph: graph IR + pluggable execution backends #336)". Last good: 305de06.
• Suspected fix area: the lowering path in python/cudnn/_pygraph.py (_lower_to_cpp() and the pointwise/bitmask + norm node builders) — likely a missing or mis-emitted virtual-tensor / bitmask-output attribute in the ReLU-bitmask fusion path, so the emitted C++ graph differs from what the old direct binding produced.
• Failing CI job: py_samples:ga: [Blackwell|Hopper|Ampher], py_samples:rel: [Blackwell|Hopper], pipeline 57107110 (job 356628792).

[YangXu1990uiuc]

there is an internal bug (6422108) tracking this issue, this is an issue in the cudnn backend
tldr when you compile 2 graphs of the same everything but with different uids, the second time cache will hit and the engine will be looking for the wrong pointers

[Anerudhan]

Hi @liujane-dev This should be fixed 9.25.0 and later.