gpu-project: precise projection shaders do not finish compiling on Windows (D3D12) #3425

Summary

On Windows, Chrome cannot create the compute pipeline for the precise projection program. createComputePipelineAsync for the benchmark's projection shader (53.5 K characters of WGSL) had not resolved after 3 minutes in headed stable Chrome, or after 15 minutes in the headless test runner, on an RTX 4090. The same WGSL compiles in about 0.35-0.45 s in Chrome on an M1 Max (Metal). As a result, projection-performance.spec.ts cannot run on Windows hardware at any row count, including 32 rows: on current master all three cases time out at 180 s.

Using classic double-single arithmetic with its uniforms bound, and calling each heavy helper from one site, brings pipeline creation to about 1.7-1.9 s per path. The benchmark then passes its own accuracy checks, with the same maximum errors as the integer path on Metal (see Fix, measured end to end). I will open a draft PR with this fix.

Environment

• luma.gl master at 5da4efd (Add projected ray depth for raster composition #3419) for the end-to-end benchmark runs. The size and compile-time analysis below was done at f87695a (feat(gpu-project): align math.gl engines, CRS metadata and CPU benchmarks #3401) and has not been repeated on current master; the benchmark still times out there.
• Windows: NVIDIA GeForce RTX 4090, driver 610.88. Chrome 155 stable, headed, and the Playwright Chromium 151 used by yarn test-headless. chrome://gpu reports WebGPU "Hardware accelerated" on the D3D12 backend, with the use_dxc and d3d12_use_hlsl_2021 toggles enabled. The adapter reports vendor: nvidia, architecture: lovelace, fallback: false.
• macOS: Apple M1 Max, Chrome 154 stable headed, and Playwright Chromium 151 headless.
• Linux: NVIDIA Orin (Ampere), Vulkan, Chrome 156 headed (hardware adapter, fallback: false).

Cause

D3D shader bytecode has no general function calls, so every WGSL function call is inlined in full before DXC (or FXC) compiles it. The integer-controlled fp64 functions are large and call each other, so the fully inlined size grows quickly. The sizes below are measured in WGSL source characters, adding each call site's callee recursively:

Compile time grows faster than inlined size. Each row below is a minimal shader containing only the fp64 arithmetic module plus N chained sum_fp64 calls. Times are for pipeline creation in Chrome on Windows:

Headless Playwright Chromium in the test runner and headed stable Chrome produced the same DXC numbers to within 0.1 s.

In headless Chromium on macOS we also saw pipeline creation fail with "A valid external Instance reference no longer exists". That is the headless device loss described in #2874, not a compile failure: headed Chrome on the same Mac compiles the shader normally. All Windows numbers come from a hardware adapter (fallback: false) and were confirmed in headed Chrome.

Where the time goes

Chrome's dump_shaders toggle shows that Tint emits normal HLSL functions: about 530 lines for every variant. The inlining happens in DXC, because DXIL has no function calls. Compiling the dumped HLSL offline with DXC 1.9.2609, using Dawn's flags (-HV 2021 -Gis -Zpr):

• DXIL size is linear in the number of call sites, at about 66 KB per sum_fp64.
• Turning optimization off does not help. -Od is barely faster and produces twice the DXIL, because inlining is mandatory.
• DXC accounts for only 20-45 % of pipeline creation time. The rest grows with DXIL size too, which points to the driver's DXIL-to-native compile.

No compiler setting avoids the cost. Only reducing the inlined size does.

Pipeline cache

Chrome's disk cache makes a cached pipeline almost free: 5-70 ms after a browser restart. But it is unreliable for shaders this large. After caching the 16-call variant, the 4-call variant had been evicted and took 5.8 s again. The cache also never helps a first visit. Metal keeps real function calls, which is why the Mac compiles the full shader quickly.

CI cannot see this. As gpu-project/README.md notes, these suites need a hardware adapter, and CI runs SwiftShader.

Arithmetic mode

ShaderAssembler enables LUMA_FP64_INTEGER_ARITHMETIC only on Apple WebGPU (shader-assembler.ts:339, with the comment about Metal reassociating double-single transforms). Precise geospatial kernels force it on every platform (geospatial-utils.ts:342), and they use a module that drops the classic-path uniforms: "Precise kernels use only the integer-controlled functions, so omit the classic-path uniforms" (geospatial-utils.ts:31-35).

Because of that, the other arithmetic modes cannot work as wired today. Turning the define off makes every row invalid: 41 of 43 rows in the 32-row benchmark, with all-zero output. The cause is the classic-path uniforms, not precision:

• switching the precise kernels to the full fp64arithmetic module did not change the result;
• substituting the uniform values (ONE = 1.0, SPLIT = 4097.0) into the shader text made the benchmark pass.

Classic arithmetic needs the code-elimination workaround on Vulkan

We measured the classic double-single sum_fp64, mul_fp64, div_fp64 and sub_fp64 in Chrome 156 on Linux/Vulkan (NVIDIA Orin, Ampere, hardware adapter). The test used 65,536 random double-single pairs per operation and compared each result with the exact double result computed on the CPU:

Without the workaround, this driver optimizes the error-free transforms away and the low limb is lost silently. The WGSL path does not currently apply the per-vendor defines that GLSL gets (platform-defines.ts; the call is commented out in assemble-shaders.ts). Any switch to classic arithmetic therefore has to turn the workaround on. On D3D12 (Chrome 155, RTX 4090) the same test is precise with or without the workaround: 3.5e-16 to 1.9e-15 median and 0 rows over 1e-9 in both cases. Dawn compiles HLSL with IEEE strictness (-Gis), which is the likely reason. The workaround costs nothing measurable there: pipeline creation takes 59 ms without it and 90 ms with it.

Fix, measured end to end

Prototype: shader substitution (at f87695a)

We first ran the benchmark (projection-performance.spec.ts, 32 rows, 4 pipelines) on the Windows machine, replacing each projection shader at createShaderModule with modified WGSL. The validation and error numbers come from the benchmark's own oracle checks.

GPU time per path was about 3 ms in both classic-arithmetic runs.

Supplying the uniforms as literal constants lets a compiler fold the error-free transforms the uniform exists to protect, so the library fix binds the uniforms instead.

Library fix (on 5da4efd)

The fix implements this in the library: projection programs choose classic double-single on D3D12 and Vulkan and keep integer arithmetic on Metal, the fp64arithmetic uniforms are bound and uploaded, and each heavy helper in the generated project() has one call site. Same benchmark, 32 rows, one consumer, LUMA_TEST_BROWSER_BENCHMARKS=true, headless Playwright Chromium 151:

The fixtures are utm-local, utm-regional and albers-regional, each with degree-2 and degree-3 adaptive paths. Every path validated 41 of 43 rows on every run. On Windows the classic-arithmetic errors match the integer-arithmetic errors on Metal to within 2e-7 mm. The albers-regional paths compile in about 10 ms on Windows and 7 ms on Mac, because the shader is the same as an earlier fixture's and comes from Chrome's in-process pipeline cache.

The single-call-site change only alters how the projection calls its helpers, in project():

• Patch containment: checked through one call site. It covers the assigned patch, or a search over all patches, instead of findProjectionPatch() followed by a second projectionPatchContains().
• Raw binary64 conversions: the two source offsets and two destination origins go through one sub_fp64u32_to_fp64 call in a loop of four. This integer conversion is 95 K inlined, and it was called four times.
• Divisions and polynomial evaluations: each runs in a loop over the two axes.

Loop bounds are constants. DXC did not unroll loops with bodies this large: the DXIL contains one copy of each helper.

If integer arithmetic stays

Where integer-controlled arithmetic remains in use, such as on Apple, the library can apply the same principle internally. This does not change its results:

• sum_fp64 calls fp64_sum_integer_unfused from two early returns. That fallback is 254 K of the function's 281 K inlined, so merging the two returns into one call matters.
• The unfused fallback's six fp64_two_sum_integer calls run as a scheduled loop.
• mul_fp64 computes its three two-products in one loop, then accumulates them with one sum_fp64 call site.
• fp64_div_fp64_normalized runs its two correction steps as a loop.
• div_fp64 merges its special-value path and its normal path into one call of the normalized division.

On 65,536 inputs the restructured sum_fp64, sub_fp64, mul_fp64 and div_fp64 were bit-identical to the current functions. The inputs covered random pairs over a ±60 binade range, cancellation, ±0, ±Inf, NaN, subnormals, unnormalized pairs and raw bit patterns. We checked that the comparison detects real changes: each mutant that changed the restructured schedule, loop count or exponent-gap threshold produced thousands of differences. Compile time with integer arithmetic stays high on D3D12, though: 14.5 s with the projection loops. In our runs the restructured library also made several Metal tests time out, so it is not part of the proposed fix. Classic arithmetic is the larger win.

Suggestions

1. On non-Metal backends, use classic double-single for precise kernels, bind its uniforms, and enable the code-elimination workaround (required on Vulkan, see above). Keep integer arithmetic on Metal, whatever the vendor, and wherever the backend is unknown on an Apple host. Today the integer path is forced (geospatial-utils.ts:342) and the classic path's uniforms are dropped (:31-35). With them supplied, the benchmark passes with the same errors as Metal.
2. Give each heavy helper one call site in generated projection programs, as above. This is about a 2.7× reduction in compile time on top of suggestion 1, with no change to results.
3. Where integer arithmetic stays, restructure the library as above. The results are bit-identical, and the functions are 7-19× smaller.
4. Add a compile-time budget to the hardware benchmark, so a regression shows as a failed check rather than a timeout.

Tested: D3D12 (NVIDIA RTX 4090), Vulkan arithmetic (NVIDIA Orin) and Metal (Apple M1 Max, integer path). Not yet tested: AMD and Intel GPUs on any backend.

Repro material

We can attach the following:

• the captured WGSL of the benchmark's projection shader;
• the variant shaders (1-64 sum_fp64 calls, the loop version, and multiply-add chains);
• the prototypes: projection call-site restructuring for integer and classic arithmetic, and the restructured integer library;
• the shader-substitution harness for the benchmark;
• the equivalence kernels;
• a small page that times createComputePipelineAsync for each file;
• the script that computes inlined sizes.