The agent's persistent memory. Survives context compaction. Update it after every experiment. Build your own dead-end log here as you go, it stops you (and future runs) from re-grinding known failures.
- exp_000 baseline: Wan 2.2 VAE, fp32, eager, no compile. Latency: TBD (run it). 1.00x. Gate compares against the frozen fp32 reference (it IS the reference path → max_abs_diff 0).
- reference_fp32.pt generated (
python harness/make_reference.py). - exp_000: measure the fp32-eager baseline.
- climb toward the hardware floor (gate must stay green).
Follow program.md. One change at a time to optimize.py. After every KEEP the driver
auto-runs the profiler and appends the breakdown below, read it (especially the GPU-idle
%) and let it pick your next move. Use your own knowledge of GPU inference optimization
plus what the profiler shows. Do not consult any external/prior optimized
implementation of this model, discover the wins here.
GPU util 395.4% | wall 15270.48ms | gpu 60375.39ms | idle -45104.91ms per iter top ops: Command Buffer Full 31626.374ms, aten::cudnn_convolution 8672.763ms, sm90_xmma_fprop_implicit_gemm_f32f32_tf32f32_f32_nhwckrsc_nhwc_tilesize256x128x32_warpgroupsize2x1x1_g1_execute_segment_k_off_kernel__5x_cudnn 3682.868ms, sm90_xmma_fprop_implicit_gemm_f32f32_tf32f32_f32_nhwckrsc_nhwc_tilesize128x128x32_warpgroupsize1x1x1_g1_execute_segment_k_off_kernel__5x_cudnn 3054.379ms, void at::native::elementwise_kernel<128, 2, at::native::gpu_kernel_impl_nocast<at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#7}::operator()() const::{lambda(float)#1}>(at::TensorIteratorBase&, at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#7}::operator()() const::{lambda(float)#1} const&)::{lambda(int)#1}>(int, at::native::gpu_kernel_impl_nocast<at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#7}::operator()() const::{lambda(float)#1}>(at::TensorIteratorBase&, at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#7}::operator()() const::{lambda(float)#1} const&)::{lambda(int)#1}) 1944.861ms full report: runs/profile_exp0.txt | trace: runs/profile_exp0.json next: COMPUTE-BOUND, attack the top op (dtype, layout, fusion)
GPU util 100.0% | wall 9160.47ms | gpu 16841.15ms | idle 0.0ms per iter top ops: Command Buffer Full 13487.781ms, aten::cudnn_convolution 4462.358ms, sm90_xmma_fprop_implicit_gemm_bf16bf16_bf16f32_f32_nhwckrsc_nhwc_tilesize256x128x64_warpgroupsize2x1x1_g1_execute_segment_k_off_kernel__5x_cudnn 2378.953ms, void at::native::elementwise_kernel<128, 4, at::native::gpu_kernel_impl_nocast<at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#12}::operator()() const::{lambda(c10::BFloat16)#1}>(at::TensorIteratorBase&, at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#12}::operator()() const::{lambda(c10::BFloat16)#1} const&)::{lambda(int)#1}>(int, at::native::gpu_kernel_impl_nocast<at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#12}::operator()() const::{lambda(c10::BFloat16)#1}>(at::TensorIteratorBase&, at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#12}::operator()() const::{lambda(c10::BFloat16)#1} const&)::{lambda(int)#1}) 1478.733ms, aten::copy_ 1190.569ms full report: runs/profile_exp1.txt | trace: runs/profile_exp1.json next: COMPUTE-BOUND, attack the top op (dtype, layout, fusion)
GPU util 100.0% | wall 8534.74ms | gpu 16496.69ms | idle 0.0ms per iter top ops: Command Buffer Full 19846.785ms, aten::cudnn_convolution 4464.678ms, sm90_xmma_fprop_implicit_gemm_bf16bf16_bf16f32_f32_nhwckrsc_nhwc_tilesize256x128x64_warpgroupsize2x1x1_g1_execute_segment_k_off_kernel__5x_cudnn 2717.127ms, void at::native::elementwise_kernel<128, 4, at::native::gpu_kernel_impl_nocast<at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#12}::operator()() const::{lambda(c10::BFloat16)#1}>(at::TensorIteratorBase&, at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#12}::operator()() const::{lambda(c10::BFloat16)#1} const&)::{lambda(int)#1}>(int, at::native::gpu_kernel_impl_nocast<at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#12}::operator()() const::{lambda(c10::BFloat16)#1}>(at::TensorIteratorBase&, at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#12}::operator()() const::{lambda(c10::BFloat16)#1} const&)::{lambda(int)#1}) 1123.665ms, aten::copy_ 764.565ms full report: runs/profile_exp2.txt | trace: runs/profile_exp2.json next: COMPUTE-BOUND, attack the top op (dtype, layout, fusion)
GPU util 100.0% | wall 8688.07ms | gpu 15895.59ms | idle 0.0ms per iter top ops: Command Buffer Full 8580.693ms, aten::cudnn_convolution 3709.179ms, sm90_xmma_fprop_implicit_gemm_bf16bf16_bf16f32_f32_nhwckrsc_nhwc_tilesize256x128x64_warpgroupsize2x1x1_g1_execute_segment_k_off_kernel__5x_cudnn 2499.037ms, aten::copy_ 1369.526ms, void at::native::elementwise_kernel<128, 4, at::native::gpu_kernel_impl_nocast<at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#12}::operator()() const::{lambda(c10::BFloat16)#1}>(at::TensorIteratorBase&, at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#12}::operator()() const::{lambda(c10::BFloat16)#1} const&)::{lambda(int)#1}>(int, at::native::gpu_kernel_impl_nocast<at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#12}::operator()() const::{lambda(c10::BFloat16)#1}>(at::TensorIteratorBase&, at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#12}::operator()() const::{lambda(c10::BFloat16)#1} const&)::{lambda(int)#1}) 1197.786ms full report: runs/profile_exp3.txt | trace: runs/profile_exp3.json next: COMPUTE-BOUND, attack the top op (dtype, layout, fusion)
GPU util 45.8% | wall 8412.75ms | gpu 16263.87ms | idle 0.0ms per iter top ops: Command Buffer Full 19163.986ms, aten::cudnn_convolution 4456.311ms, sm90_xmma_fprop_implicit_gemm_bf16bf16_bf16f32_f32_nhwckrsc_nhwc_tilesize256x128x64_warpgroupsize2x1x1_g1_execute_segment_k_off_kernel__5x_cudnn 2725.627ms, void at::native::elementwise_kernel<128, 4, at::native::gpu_kernel_impl_nocast<at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#12}::operator()() const::{lambda(c10::BFloat16)#1}>(at::TensorIteratorBase&, at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#12}::operator()() const::{lambda(c10::BFloat16)#1} const&)::{lambda(int)#1}>(int, at::native::gpu_kernel_impl_nocast<at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#12}::operator()() const::{lambda(c10::BFloat16)#1}>(at::TensorIteratorBase&, at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#12}::operator()() const::{lambda(c10::BFloat16)#1} const&)::{lambda(int)#1}) 1120.275ms, aten::copy_ 760.936ms full report: runs/profile_exp4.txt | trace: runs/profile_exp4.json next: HOST-BOUND, recover idle time (fewer/larger kernels, overlap, CUDA graphs, compile a bigger region)
GPU util 65.3% | wall 8597.85ms | gpu 15696.35ms | idle 0.0ms per iter top ops: Command Buffer Full 7903.259ms, aten::cudnn_convolution 3704.683ms, sm90_xmma_fprop_implicit_gemm_bf16bf16_bf16f32_f32_nhwckrsc_nhwc_tilesize256x128x64_warpgroupsize2x1x1_g1_execute_segment_k_off_kernel__5x_cudnn 2358.547ms, aten::copy_ 1279.323ms, void at::native::elementwise_kernel<128, 4, at::native::gpu_kernel_impl_nocast<at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#12}::operator()() const::{lambda(c10::BFloat16)#1}>(at::TensorIteratorBase&, at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#12}::operator()() const::{lambda(c10::BFloat16)#1} const&)::{lambda(int)#1}>(int, at::native::gpu_kernel_impl_nocast<at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#12}::operator()() const::{lambda(c10::BFloat16)#1}>(at::TensorIteratorBase&, at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#12}::operator()() const::{lambda(c10::BFloat16)#1} const&)::{lambda(int)#1}) 1207.754ms full report: runs/profile_exp6.txt | trace: runs/profile_exp6.json next: HOST-BOUND, recover idle time (fewer/larger kernels, overlap, CUDA graphs, compile a bigger region)
GPU util 94.7% | wall 8539.93ms | gpu 7650.35ms | idle 889.58ms per iter top ops: sm90_xmma_fprop_implicit_gemm_bf16bf16_bf16f32_f32_nhwckrsc_nhwc_tilesize256x128x64_warpgroupsize2x1x1_g1_execute_segment_k_off_kernel__5x_cudnn 2712.097ms, void at::native::elementwise_kernel<128, 4, at::native::gpu_kernel_impl_nocast<at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#12}::operator()() const::{lambda(c10::BFloat16)#1}>(at::TensorIteratorBase&, at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#12}::operator()() const::{lambda(c10::BFloat16)#1} const&)::{lambda(int)#1}>(int, at::native::gpu_kernel_impl_nocast<at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#12}::operator()() const::{lambda(c10::BFloat16)#1}>(at::TensorIteratorBase&, at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#12}::operator()() const::{lambda(c10::BFloat16)#1} const&)::{lambda(int)#1}) 1210.034ms, void at::native::elementwise_kernel<128, 4, at::native::gpu_kernel_impl_nocast<at::native::CUDAFunctor_addc10::BFloat16 >(at::TensorIteratorBase&, at::native::CUDAFunctor_addc10::BFloat16 const&)::{lambda(int)#1}>(int, at::native::gpu_kernel_impl_nocast<at::native::CUDAFunctor_addc10::BFloat16 >(at::TensorIteratorBase&, at::native::CUDAFunctor_addc10::BFloat16 const&)::{lambda(int)#1}) 914.929ms, sm90_xmma_fprop_implicit_gemm_bf16bf16_bf16f32_f32_nhwckrsc_nhwc_tilesize256x128x64_warpgroupsize2x1x1_g1_execute_segment_k_on_kernel__5x_cudnn 530.615ms, memcpy128 420.634ms full report: runs/profile_exp7.txt | trace: runs/profile_exp7.json next: COMPUTE-BOUND, attack the top op (dtype, layout, fusion)
GPU util 94.7% | wall 7856.14ms | gpu 7027.24ms | idle 828.89ms per iter top ops: sm90_xmma_fprop_implicit_gemm_bf16bf16_bf16f32_f32_nhwckrsc_nhwc_tilesize256x128x64_warpgroupsize2x1x1_g1_execute_segment_k_off_kernel__5x_cudnn 2519.304ms, void at::native::elementwise_kernel<128, 4, at::native::gpu_kernel_impl_nocast<at::native::CUDAFunctor_addc10::BFloat16 >(at::TensorIteratorBase&, at::native::CUDAFunctor_addc10::BFloat16 const&)::{lambda(int)#1}>(int, at::native::gpu_kernel_impl_nocast<at::native::CUDAFunctor_addc10::BFloat16 >(at::TensorIteratorBase&, at::native::CUDAFunctor_addc10::BFloat16 const&)::{lambda(int)#1}) 906.178ms, void at::native::elementwise_kernel<128, 4, at::native::gpu_kernel_impl_nocast<at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#12}::operator()() const::{lambda(c10::BFloat16)#1}>(at::TensorIteratorBase&, at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#12}::operator()() const::{lambda(c10::BFloat16)#1} const&)::{lambda(int)#1}>(int, at::native::gpu_kernel_impl_nocast<at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#12}::operator()() const::{lambda(c10::BFloat16)#1}>(at::TensorIteratorBase&, at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#12}::operator()() const::{lambda(c10::BFloat16)#1} const&)::{lambda(int)#1}) 680.596ms, sm90_xmma_fprop_implicit_gemm_bf16bf16_bf16f32_f32_nhwckrsc_nhwc_tilesize256x128x64_warpgroupsize2x1x1_g1_execute_segment_k_on_kernel__5x_cudnn 531.43ms, sm90_xmma_fprop_implicit_gemm_bf16bf16_bf16f32_f32_nhwckrsc_nhwc_tilesize128x256x64_warpgroupsize2x1x1_g1_execute_segment_k_off_kernel__5x_cudnn 490.327ms full report: runs/profile_exp8.txt | trace: runs/profile_exp8.json next: COMPUTE-BOUND, attack the top op (dtype, layout, fusion)
GPU util 94.6% | wall 6720.29ms | gpu 6311.27ms | idle 409.02ms per iter top ops: sm90_xmma_fprop_implicit_gemm_bf16bf16_bf16f32_f32_nhwckrsc_nhwc_tilesize256x128x64_warpgroupsize2x1x1_g1_execute_segment_k_off_kernel__5x_cudnn 2562.836ms, void at::native::elementwise_kernel<128, 4, at::native::gpu_kernel_impl_nocast<at::native::CUDAFunctor_addc10::BFloat16 >(at::TensorIteratorBase&, at::native::CUDAFunctor_addc10::BFloat16 const&)::{lambda(int)#1}>(int, at::native::gpu_kernel_impl_nocast<at::native::CUDAFunctor_addc10::BFloat16 >(at::TensorIteratorBase&, at::native::CUDAFunctor_addc10::BFloat16 const&)::{lambda(int)#1}) 832.979ms, sm90_xmma_fprop_implicit_gemm_bf16bf16_bf16f32_f32_nhwckrsc_nhwc_tilesize256x128x64_warpgroupsize2x1x1_g1_execute_segment_k_on_kernel__5x_cudnn 546.707ms, sm90_xmma_fprop_implicit_gemm_bf16bf16_bf16f32_f32_nhwckrsc_nhwc_tilesize128x256x64_warpgroupsize2x1x1_g1_execute_segment_k_off_kernel__5x_cudnn 525.77ms, void at::native::elementwise_kernel<128, 4, at::native::gpu_kernel_impl_nocast<at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#12}::operator()() const::{lambda(c10::BFloat16)#1}>(at::TensorIteratorBase&, at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#12}::operator()() const::{lambda(c10::BFloat16)#1} const&)::{lambda(int)#1}>(int, at::native::gpu_kernel_impl_nocast<at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#12}::operator()() const::{lambda(c10::BFloat16)#1}>(at::TensorIteratorBase&, at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#12}::operator()() const::{lambda(c10::BFloat16)#1} const&)::{lambda(int)#1}) 395.852ms full report: runs/profile_exp9.txt | trace: runs/profile_exp9.json next: COMPUTE-BOUND, attack the top op (dtype, layout, fusion)
GPU util 94.5% | wall 6960.92ms | gpu 6539.99ms | idle 420.93ms per iter top ops: sm90_xmma_fprop_implicit_gemm_bf16bf16_bf16f32_f32_nhwckrsc_nhwc_tilesize256x128x64_warpgroupsize2x1x1_g1_execute_segment_k_off_kernel__5x_cudnn 2672.218ms, void at::native::elementwise_kernel<128, 4, at::native::gpu_kernel_impl_nocast<at::native::CUDAFunctor_addc10::BFloat16 >(at::TensorIteratorBase&, at::native::CUDAFunctor_addc10::BFloat16 const&)::{lambda(int)#1}>(int, at::native::gpu_kernel_impl_nocast<at::native::CUDAFunctor_addc10::BFloat16 >(at::TensorIteratorBase&, at::native::CUDAFunctor_addc10::BFloat16 const&)::{lambda(int)#1}) 851.867ms, sm90_xmma_fprop_implicit_gemm_bf16bf16_bf16f32_f32_nhwckrsc_nhwc_tilesize256x128x64_warpgroupsize2x1x1_g1_execute_segment_k_on_kernel__5x_cudnn 579.978ms, sm90_xmma_fprop_implicit_gemm_bf16bf16_bf16f32_f32_nhwckrsc_nhwc_tilesize128x256x64_warpgroupsize2x1x1_g1_execute_segment_k_off_kernel__5x_cudnn 554.034ms, void at::native::elementwise_kernel<128, 4, at::native::gpu_kernel_impl_nocast<at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#12}::operator()() const::{lambda(c10::BFloat16)#1}>(at::TensorIteratorBase&, at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#12}::operator()() const::{lambda(c10::BFloat16)#1} const&)::{lambda(int)#1}>(int, at::native::gpu_kernel_impl_nocast<at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#12}::operator()() const::{lambda(c10::BFloat16)#1}>(at::TensorIteratorBase&, at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#12}::operator()() const::{lambda(c10::BFloat16)#1} const&)::{lambda(int)#1}) 407.197ms full report: runs/profile_exp10.txt | trace: runs/profile_exp10.json next: COMPUTE-BOUND, attack the top op (dtype, layout, fusion)
exp1 bf16 -> 8.42s (1.72x). The one big lever. diagnosis_correct: yes (fp32->bf16 tensor cores). max_abs_diff 0.071 (<0.10). lesson: bf16 is the dominant win; everything after is incremental. exp2 +torch.compile(default) -> 8.26s. Marginal. fix_effective: barely, graph breaks from the stateful per-frame feat_cache leave nothing to fuse. lesson: compile is useless UNTIL the per-conv F.pad/cache churn is removed. exp3 +channels_last_3d -> 8.15s. Small real win (NHWC tensor-core convs); convs 4.46->3.71s but adds layout copies. exp4 +cudnn.benchmark -> DISCARD (8.13, slower). convs already on a good algo. exp6 +bf16-native upsample (skip fp32 round-trip in nearest-exact Upsample) -> 8.02s. Zero quality change (max_abs_diff unchanged 0.0667). Flipped profiler to "host-bound". exp7 +full-decode CUDA graph -> 8.00s. The "host-bound" was a profiler artifact; real launch overhead ~0 (CPU runs ahead, GPU is the wall). BUT the graph made the profiler honest: 94.7% util, clean per-op breakdown. KEY enabler. exp8 +native-spatial-pad conv -> 7.42s (-7%). CausalConv3d.forward F.pad'd the FULL activation (spatial+temporal); let cuDNN pad spatial natively, only F.pad the causal-temporal dim. Killed the #2 cost. Bit-identical. exp9 +torch.compile(default) AGAIN -> 6.66s (-10%!). Now that F.pad churn is gone and convs are functional F.conv3d, inductor fuses norm->silu and RMS_norm; graph captures the fused kernels. lesson: ORDERING matters, compile only pays off after the eager hot path is clean. exp10 +max-autotune-no-cudagraphs -> 6.64s. Flat vs default; convs stay cuDNN (autotune confirms cuDNN optimal), max_abs_diff crept to 0.089 (closer to gate). Compile-mode axis exhausted.
- cudnn.benchmark: no help, convs already optimal algo. Don't retry.
- torch.compile BEFORE removing F.pad/cache churn: ~0 gain (graph breaks). Only works after exp8.
- max-autotune vs default compile: flat (cuDNN convs not replaced by Triton; no epilogue fusion of the residual add).
- 2-chunk / N-frame chunked decode (tested in scratch, NOT logged as a run): decoding [frame0] then [frames 1..N] as big causal passes is BITWISE-equivalent to the 20 single-frame streaming passes (verified: max_abs_diff 0.0667, identical). BUT it is NOT faster than streaming+compile+graph: eager chunk=4 = 7.599s @ 61GB; chunk=2+compile(no graph) = 6.696s @ 35.7GB, both >= best 6.643. Reasons: (a) compile hits inductor XBLOCK>4096 on big chunks, (b) CUDA graph private pool doubles memory -> OOM at chunk>=2 with graph, (c) the launch-count + op-size benefits of chunking were ALREADY captured by compile+graph on the streaming path. Chunking trades 1.5-3x memory for break-even speed. Abandoned. Don't retry unless a way is found to compile big chunks (avoid XBLOCK) AND fit the graph pool.
- convs ~45% of GPU time, cuDNN bf16 NHWC, optimal, irreducible without arch change.
- residual add
x + h~2.5s/window, EAGER (graph-break boundary after cuDNN conv); inductor can't epilogue-fuse a cuDNN extern conv. Biggest remaining fusable cost. - feat_cache clones + memcpy (rearrange/permute.contiguous in DupUp3D/Resample) ~intrinsic to the streaming-cache design.
GPU util 94.6% | wall 6704.22ms | gpu 6297.9ms | idle 406.32ms per iter top ops: sm90_xmma_fprop_implicit_gemm_bf16bf16_bf16f32_f32_nhwckrsc_nhwc_tilesize256x128x64_warpgroupsize2x1x1_g1_execute_segment_k_off_kernel__5x_cudnn 2545.174ms, void at::native::elementwise_kernel<128, 4, at::native::gpu_kernel_impl_nocast<at::native::CUDAFunctor_addc10::BFloat16 >(at::TensorIteratorBase&, at::native::CUDAFunctor_addc10::BFloat16 const&)::{lambda(int)#1}>(int, at::native::gpu_kernel_impl_nocast<at::native::CUDAFunctor_addc10::BFloat16 >(at::TensorIteratorBase&, at::native::CUDAFunctor_addc10::BFloat16 const&)::{lambda(int)#1}) 825.937ms, sm90_xmma_fprop_implicit_gemm_bf16bf16_bf16f32_f32_nhwckrsc_nhwc_tilesize256x128x64_warpgroupsize2x1x1_g1_execute_segment_k_on_kernel__5x_cudnn 561.794ms, sm90_xmma_fprop_implicit_gemm_bf16bf16_bf16f32_f32_nhwckrsc_nhwc_tilesize128x256x64_warpgroupsize2x1x1_g1_execute_segment_k_off_kernel__5x_cudnn 549.836ms, void at::native::elementwise_kernel<128, 4, at::native::gpu_kernel_impl_nocast<at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#12}::operator()() const::{lambda(c10::BFloat16)#1}>(at::TensorIteratorBase&, at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#12}::operator()() const::{lambda(c10::BFloat16)#1} const&)::{lambda(int)#1}>(int, at::native::gpu_kernel_impl_nocast<at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#12}::operator()() const::{lambda(c10::BFloat16)#1}>(at::TensorIteratorBase&, at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#12}::operator()() const::{lambda(c10::BFloat16)#1} const&)::{lambda(int)#1}) 397.365ms full report: runs/profile_exp13.txt | trace: runs/profile_exp13.json next: COMPUTE-BOUND, attack the top op (dtype, layout, fusion)
GPU util 99.9% | wall 4820.79ms | gpu 4788.05ms | idle 32.74ms per iter top ops: sm90_xmma_fprop_implicit_gemm_bf16bf16_bf16f32_f32_nhwckrsc_nhwc_tilesize256x128x64_warpgroupsize2x1x1_g1_execute_segment_k_off_kernel__5x_cudnn 1857.533ms, sm90_xmma_fprop_implicit_gemm_bf16bf16_bf16f32_f32_nhwckrsc_nhwc_tilesize128x256x64_warpgroupsize2x1x1_g1_execute_segment_k_off_kernel__5x_cudnn 1487.682ms, sm90_xmma_fprop_implicit_gemm_bf16bf16_bf16f32_f32_nhwckrsc_nhwc_tilesize256x128x64_warpgroupsize2x1x1_g1_execute_segment_k_on_kernel__5x_cudnn 597.201ms, sm80_xmma_fprop_implicit_gemm_bf16bf16_bf16f32_f32_nhwckrsc_nhwc_tilesize256x64x32_stage3_warpsize4x1x1_g1_tensor16x8x16_t3r3s3_execute_kernel__5x_cudnn 91.247ms, triton_per_fused_add_convolution_div_linalg_vector_norm_mul_silu_45 73.019ms full report: runs/profile_exp14.txt | trace: runs/profile_exp14.json next: COMPUTE-BOUND, attack the top op (dtype, layout, fusion)
GPU util 99.8% | wall 4822.69ms | gpu 4791.32ms | idle 31.38ms per iter top ops: sm90_xmma_fprop_implicit_gemm_bf16bf16_bf16f32_f32_nhwckrsc_nhwc_tilesize256x128x64_warpgroupsize2x1x1_g1_execute_segment_k_off_kernel__5x_cudnn 2703.852ms, sm90_xmma_fprop_implicit_gemm_bf16bf16_bf16f32_f32_nhwckrsc_nhwc_tilesize128x256x64_warpgroupsize2x1x1_g1_execute_segment_k_off_kernel__5x_cudnn 644.176ms, sm90_xmma_fprop_implicit_gemm_bf16bf16_bf16f32_f32_nhwckrsc_nhwc_tilesize256x128x64_warpgroupsize2x1x1_g1_execute_segment_k_on_kernel__5x_cudnn 596.511ms, sm80_xmma_fprop_implicit_gemm_bf16bf16_bf16f32_f32_nhwckrsc_nhwc_tilesize256x64x32_stage3_warpsize4x1x1_g1_tensor16x8x16_t3r3s3_execute_kernel__5x_cudnn 91.283ms, triton_per_fused_add_convolution_div_linalg_vector_norm_mul_silu_45 72.96ms full report: runs/profile_exp16.txt | trace: runs/profile_exp16.json next: COMPUTE-BOUND, attack the top op (dtype, layout, fusion)
GPU util 99.9% | wall 4822.02ms | gpu 4789.72ms | idle 32.3ms per iter top ops: sm90_xmma_fprop_implicit_gemm_bf16bf16_bf16f32_f32_nhwckrsc_nhwc_tilesize256x128x64_warpgroupsize2x1x1_g1_execute_segment_k_off_kernel__5x_cudnn 2680.899ms, sm90_xmma_fprop_implicit_gemm_bf16bf16_bf16f32_f32_nhwckrsc_nhwc_tilesize128x256x64_warpgroupsize2x1x1_g1_execute_segment_k_off_kernel__5x_cudnn 666.954ms, sm90_xmma_fprop_implicit_gemm_bf16bf16_bf16f32_f32_nhwckrsc_nhwc_tilesize256x128x64_warpgroupsize2x1x1_g1_execute_segment_k_on_kernel__5x_cudnn 596.791ms, sm80_xmma_fprop_implicit_gemm_bf16bf16_bf16f32_f32_nhwckrsc_nhwc_tilesize256x64x32_stage3_warpsize4x1x1_g1_tensor16x8x16_t3r3s3_execute_kernel__5x_cudnn 91.289ms, triton_per_fused_add_convolution_div_linalg_vector_norm_mul_silu_45 73.017ms full report: runs/profile_exp18.txt | trace: runs/profile_exp18.json next: COMPUTE-BOUND, attack the top op (dtype, layout, fusion)
The remaining wall after exp10 (6.64s) was: convs ~45% (cuDNN floor) + a large EAGER residual
add (x+h) + intrinsic streaming copies. The add stayed eager because the model's decoder
forwards GRAPH-BREAK at the external feat_cache list ops and the "Rep" STRING sentinel
(dynamo: "Unsupported method call" / "non-Tensor"), so inductor could only fuse tiny isolated
regions.
Feasibility-checked first (scratch): dynamo CAN fullgraph-compile nn.Module tensor-attribute
mutation. So I reimplemented the decoder forwards in optimize.py with a PER-MODULE cache
(tensor attr _cx + int state _fc_state, no list, no string), bitwise-replicating the
streaming cache math, and reimplemented the decode loop (decode_fn) to drive it. The decoder
then compiles BREAK-FREE -> inductor fuses add->norm->silu chains across whole blocks and
reuses buffers, collapsing the elementwise/copy overhead.
Milestone ladder (gate-passing, vs 14.467s fp32 baseline):
- exp0 14.467s 1.00x fp32 eager
- exp1 8.423s 1.72x bf16
- exp3 8.154s 1.77x + channels_last_3d
- exp6 8.020s 1.80x + bf16-native upsample (skip fp32 round-trip)
- exp8 7.419s 1.95x + native-spatial-pad conv (cuDNN pads spatially, not F.pad)
- exp9 6.658s 2.17x + torch.compile (works ONLY after F.pad churn removed)
- exp10 6.643s 2.18x + max-autotune
- exp14 4.816s 3.00x + BREAK-FREE per-module-cache decoder (fuses the eager residual add)
- exp16 4.796s 3.02x + unpatchify fused into the compiled region
- exp18 4.791s 3.02x + drop coordinate_descent_tuning (was a slight overfit)
BEST = 4.791s, 3.02x, max_abs_diff 0.0610, psnr 61.9, peak 32.7GB. Quality margin IMPROVED (0.061 vs 0.089 in the pre-break-free max-autotune) because fullgraph fusion has cleaner numerics.
exp14 reflexion: diagnosis_correct: yes, graph breaks (not the convs) were blocking elementwise fusion. fix_effective: hugely, 6.64 -> 4.82s (-27%), far more than the add alone (inductor also fused norm/silu/copies and reused buffers across the whole block). lesson: when compile "does nothing," the cause may be graph breaks from host-side bookkeeping (lists, string sentinels), not the math. Removing them unlocks fusion. Verify dynamo can compile the replacement (module-attr mutation) BEFORE the big rewrite.
- drop channels_last (break-free): 5.65s, CL is ESSENTIAL; inductor picks slow NCDHW convs without it.
- default vs max-autotune compile (break-free): parity (~noise).
- coordinate_descent_tuning: slight overfit, removing it was marginally faster + simpler.
- chunked decode (frames>1/pass) on break-free: inductor int32 index overflow at chunk>=2; earlier XBLOCK>4096 too. Chunking blocked by compile limits AND wasn't faster than per-frame.
GPU util 99.8%, idle 31ms. ~84% of GPU time is cuDNN bf16 implicit-gemm conv3d (the highest-res convs at 9ms/call). Elementwise is fully fused into triton conv+norm kernels. Attention (fmha) ~1.2%. There is NO remaining fusable/host bottleneck, the decode is the cuDNN conv compute floor. Beating 4.79s would require reducing conv FLOPs (architecture change, out of scope) or faster-than-cuDNN conv3d kernels.