Skip to content

ENH: One online evaluation for both metrics, feature subsets drawn once per evaluation - #6

Merged
vboussot merged 27 commits into
mainfrom
subset-once-per-evaluation
Nov 11, 2026
Merged

vboussot merged 27 commits into
mainfrom
subset-once-per-evaluation

Conversation

@vboussot

@vboussot vboussot commented Oct 18, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes how the v4 metric and the ConvexAdam stages draw their random numbers, and makes the v4 metric faster: Static mode about 6x, the online (Jacobian) mode 4x to 50x on a GPU.

Fixes

  • The v4 metric drew its feature subset per point (Static) or per batch (Jacobian), from a generator per work unit. NCC sums its statistics channel by channel over the points, so those sums mixed channels, and the result depended on the number of threads and on the batch size. With 2 of 4 channels, the Jacobian derivative moved by 7 to 15% between batch sizes. The subset is now drawn once per evaluation.
  • The coarse and fine stages called torch::manual_seed(Seed), which reseeds the whole process, a Python caller's generator included. They now use a generator of their own.
  • The losses summed a batch's values in float32 (2e-8 to 6e-8 relative error over 20000 points). They now sum in double.

Speed, Static mode (lung CT test data, 20000 points, a 32-channel model)

main this PR
32 channels, 24 threads 120 ms 20 ms
32 channels, 1 thread 1281 ms 219 ms
8 of 32 channels, 24 threads 54 ms 6-9 ms
8 of 32 channels, 1 thread 514 ms 33 ms
  • Static now evaluates a work unit's points in batches instead of point by point.
  • The vector interpolator computes the B-spline weights once per point for all channels, instead of once per channel. It gives the same result to the bit as BSplineInterpolateImageFunction, which the new interpolator test checks.
  • The moving gradient image, which IMPACT never reads, is no longer built in Initialize().

Speed, online mode (same data and model, 2000 points unless noted, best of 3)

main this PR
L2, GPU 103 ms 13 ms
L2, GPU, 20000 points 430 ms 110 ms
NCC, GPU 782 ms 15 ms
NCC, CPU, 24 threads 901 ms 121 ms
L2, CPU, 24 threads, 8 of 32 channels (best of 10) 83 ms 85 ms
  • On a GPU, the models ran each work unit's points in batches of 64, one work unit after the other. The work units now only collect their points; all the threads then cut the patches, and the models run on batches as large as the device holds. BatchSize now defaults to 0: that measured bound on a GPU, 64 on the CPU.
  • The trial batch that measures the device budget is no larger than the largest batch an evaluation runs. Sized to the whole free memory, it took that memory from the other processes on a shared GPU.
  • NCC only knows its derivative with respect to one point's features once it has seen every point, so it took every channel's derivative instead, with one backward pass per channel. It now takes two passes: the first, without gradient, gives the value and that derivative at every point; the second needs one backward pass per batch, as L2 does.

Each fix comes with a test that fails without it. A new test checks that the GPU gives the CPU's value and derivative, for L2 and NCC; it is skipped without a CUDA device.

Added to this PR: one online evaluation for both metrics

The commits that built on this branch are folded in here rather than opened as a second PR.

  • Impact::EvaluateOnline (itkImpactOnlineInference.h) now runs the online (Jacobian) mode of the v4 metric, and is what the elastix metric calls too: batches as large as the device holds, a batch that runs out of memory halved and replayed after the losses are rewound, a global loss (NCC, OrganDice) taken in one pass when one batch holds every point and in two otherwise.
  • API change. GenerateOutputs and GenerateOutputsAndJacobian, the inference per work unit, are removed. A metric that still calls them, as elastix does while it pins ITKIMPACT 0.1.6, keeps building against 0.1.6 and moves to EvaluateOnline when it raises its pin.
  • The fixed features of a sampled point set are kept from one evaluation to the next, and dropped when anything they depend on changes (the points, the fixed image or transform, the models, the feature subsets).
  • Each thread of an evaluation runs on one OpenMP thread, so that a long derivative does not start a team of threads per operation.
  • OrganDice: the Dice of each organ over the image, for feature maps that are organ probabilities.
  • benchmarks/ab.sh builds two revisions and times them side by side (ImpactPerfBench), on the v4 metric and on the elastix online path.

Fixed while reviewing this PR

  • Static mode gathered every point of a work unit before evaluating them in batches: a dense domain of 512^3 voxels held about 12 GiB of points. They are now evaluated as they are gathered.
  • The Static batch size divided by the transform's local parameters, which an identity transform has none of.
  • The online backward pass took a global loss's derivative as its seed in the model's precision: in half precision it rounded to zero from a few thousand points on. The seed is scaled by a power of two and the result divided by it.
  • The OrganDice derivative squared a denominator that could underflow to zero on a faint channel.

The v4 metric drew its feature subset (SubsetFeatures) from a generator
per work unit, reseeded at every evaluation: per point in Static mode,
per batch in Jacobian mode. A point's channels therefore depended on its
rank in its work unit, on the batch size, and on whether a batch was
replayed after running out of device memory. NCC accumulates its
statistics channel by channel over the points, so its sums mixed
channels. With 2 of 4 channels, the Jacobian derivative moved by 7 to
15% between batch sizes.

The subsets are now drawn once per evaluation, before the threads start,
from the seed the planes use, and every point compares the same channels.
Two evaluations of the same parameters still compare the same channels.

The batch-size test now keeps 2 of the 4 channels instead of all of
them, and a new test compares 1 and 4 work units in Static mode. Both
fail without this change.
Evaluate and EvaluateDerivative took both by value, so every call copied
the subset, two or three times per point and per layer in Static mode.
Both stages called torch::manual_seed(Seed), which reseeds every torch
generator of the process: a Python caller's random numbers restarted
from the stage's seed after each run. The fine stage also drew its
sampled points on the device, from the device's global generator, which
the batch-budget probe draws from too, so a probe could shift them.

Each stage now draws from a CPU generator of its own, seeded with Seed,
and moves the draws to the device. A seed gives the same channels and
points on the CPU and on a GPU. The new test checks that running either
stage leaves torch's global generator where it was; it fails without
this change.
The point-wise losses (L1, L2, Cosine, L1Cosine, Dice) summed a batch's
values in float32 before adding the total to their double accumulator,
so the value lost digits with the size of the batch: 2e-8 to 6e-8 of
relative error over 20000 points. The online mode of the v4 metric and
elastix's IMPACT metric both hand the losses whole batches, and finite
differences of such a value are mostly rounding. The batch is now summed
in double. The new test compares a batch of 20000 points with the same
points fed one by one, and fails without this change.
In Static mode the v4 threader let ITK walk the points one by one. Each
point went through ITK's ProcessVirtualPoint, which also computed an
image gradient IMPACT never reads, then through ProcessPoint, which
built a handful of one-row tensors to update the losses.

Static now gathers a work unit's points, as the online mode already
did, and evaluates them in batches: the feature vectors and the
transform Jacobians are written into contiguous tensors, and each loss
is updated once per batch. A batch holds at most about 4 MB of
[points, channels, parameters] Jacobian, so that a dense domain under a
B-spline transform stays bounded. The vector interpolator gains the
buffer variants this needs, and a value-and-derivative evaluation that
shares the B-spline weights between the two.

On the lung CT test data, with 20000 points and 8 of a model's 32
channels, an evaluation takes 11 ms instead of 40 ms on 24 threads, and
136 ms instead of 468 ms on one.
ImageToImageMetricv4 smooths the whole moving image into a gradient
image in Initialize(), for the metrics that read image gradients at
their points. IMPACT compares features: the online mode takes its patch
gradients from the moving interpolator, and Static mode differentiates
the feature maps. The filter is now off by default. On the lung CT,
Initialize() of an online metric takes 0.01 s instead of 0.08 s.
InterpolateVectorImageFunction held one BSplineInterpolateImageFunction
per channel. Every channel of every point converted the point to a
continuous index, computed the support, the weights and the mirrored
indices, and allocated two or three vnl matrices to hold them, although
all of these are the same for every channel.

It now computes each channel's coefficients as that class does, and
evaluates all the selected channels with one support and one set of
weights, in the same order of operations, so the result is the same to
the bit. SetActiveChannels() keeps a copy of the channels
an evaluation compares, the channels of a voxel side by side, so that a
point reads the fewest cache lines whether it compares all of a model's
channels or a few; the v4 threader sets it once per evaluation. The
per-channel images are gone, and a second SetInputImage() no longer
keeps the first image's channels.

The interpolator test now compares values and derivatives with
BSplineInterpolateImageFunction channel by channel: 70 channels, a
region that does not start at zero, anisotropic spacing, a rotated
direction, through the active channels and through another subset. On
32 channels, a value takes 1.2 us instead of 8.4 us, a value and its
derivative 2.5 us instead of 19.5 us.
On a GPU the online mode ran each work unit's points as batches of at
most BatchSize (64 by default), one work unit after the other under the
device lock. The model takes 20000 patches in about 0.08 s, but they
went through it as some 300 batches.

On a GPU the work units now only collect their points. Once they are
done, all the threads cut the patches of a batch, and the models run on
batches as large as the device holds. BatchSize now defaults to 0, which
means that measured bound on a GPU, and 64 on the CPU, where the work
units remain the parallelism and keep evaluating their own points. The
measurement tries no batch larger than the largest an evaluation can
run: a trial batch sized to the whole free memory took it from the other
processes on a shared GPU.

On the lung CT test data, with the 32-channel model, an evaluation of
20000 points takes 112 ms instead of 430 ms on a GPU; with 2000 points,
13 ms instead of 103 ms (L2) and 94 ms instead of 782 ms (NCC). The new
test compares the GPU's value and derivative with the CPU's, and is
skipped without a CUDA device.
NCC compares statistics taken over all the points, so its derivative
with respect to one point's features is only known once every point has
been seen. The online mode took every channel's d(feature)/d(moving
coordinate) instead, from one backward pass per channel: 32 passes for a
32-channel layer.

A global loss now takes two passes. The first, without gradient,
computes the features of every point, from which the loss takes its
value and its derivative with respect to each point's features. The
second seeds one backward pass per batch with that derivative, as for
L2. To see every point at once, the work units now only collect their
points on the CPU too; the threads then evaluate them in shares, 64
points at a time as before.

On the lung CT test data, with 2000 points and the 32-channel model, an
NCC evaluation takes 15 ms instead of 94 ms on a GPU, and 121 ms
instead of 1006 ms on the CPU. L2 is unchanged.
@vboussot vboussot changed the title BUG: Draw the feature subsets once per evaluation, and speed up Static mode BUG: Draw the feature subsets once per evaluation, and speed up the v4 metric Oct 18, 2026
JacobianOnTheGpuMatchesTheCpu and StaticResultIsIndependentOfTheNumberOfWorkUnits
each re-typed the metric set-up of JacobianResultIsIndependentOfBatchSize: the toy
model configuration, distance, subset, PCA, seed, interpolator and the ramp images.
One MakeMetric(loss, mode, device, transform) now builds it, keyed on the mode for
the patch size, overlap, moving interpolator and refresh interval each test set, and
each test keeps only what differs: the batch size, the device, the work units and
the transform. The gradient-filter flags the two CPU tests cleared are gone: the
metric's constructor already clears the moving one, and the fixed one is never read
with the default gradient source.
SampleBatch and EvaluateStaticBatch each typed the same fill: the Jacobian with
respect to the parameters from the cached temporaries, its copy to the float rows,
the displacement-field offset and the index row. One FillTransformJacobian does it
for both. ProcessPoint, which only throws, is defined inline in the header instead
of a sixteen-line out-of-line definition.
A time measured on a shared machine moves with everything else the
machine runs, and a CI runner is slower still. How often the online mode
runs the models does not: it is its cost on any machine, and a test can
hold it exactly.

The new test counts the convolutions the toy model runs forward and
backward, which are its passes, through a RecordFunction callback. A
batch takes one forward pass of each model and one backward pass; NCC
takes one more forward pass of the moving model; a value alone takes no
backward pass; on a GPU the whole evaluation is one batch. Against the
code before the two previous commits it fails: NCC made 24 backward
passes instead of 6, one per channel.

The comment on JacobianResultIsIndependentOfBatchSize described the
closed form NCC used before; it now describes the two passes.
A time alone means little on a machine that also runs other jobs, as
ours often does: the same evaluation moved by up to 3x within the same
hour. Two revisions measured in alternation, under the same load, do
compare.

benchmarks/ab.sh builds two revisions (a reference and HEAD by default),
each with benchmarks/ImpactPerfBench.cxx appended to its GTest driver so
that both are measured by the same code, and runs them in turns. The
bench times the v4 metric on the lung CT pair and the 32-channel layer of
M258, Static and online, CPU and CUDA. The script prints the median of
each revision per case, their ratio and how many rounds the second won,
and exits 1 when it is slower by more than 10 % in the median and in
three rounds out of four. IMPACT_PERF_LOCK holds a lock file around every
run, for a GPU shared with other jobs.

Measured with it, against the code before the online commits: NCC on the
CPU takes 413 ms instead of 2051 ms, and L2 is unchanged (256 vs 282 ms).
Over a sampled point set, nothing the fixed features of a point depend on
moves from one evaluation to the next: the fixed image and transform,
the point, the plane its patch is cut on, the channels compared. The
online mode still cut the fixed patches and ran the fixed model at every
evaluation.

They are now kept per point, from the first evaluation that computes
them until Initialize(), a new plane seed or a modified fixed transform.
A batch whose points all have theirs cuts no fixed patch and runs only
the moving model. The second pass of a global loss never needed the
fixed patches, and no longer cuts them either.

The new test checks that the fixed model runs once per point, that a
later evaluation gives the value and derivative of a metric evaluating
afresh, and that a modified fixed transform has the features computed
again.

Measured with benchmarks/ab.sh on the CPU, 2000 points, five rounds on
a loaded machine, all five won: an L2 evaluation takes 239 ms instead of
299 ms, NCC 299 ms instead of 534 ms.
A global loss (NCC) took two passes because d(loss)/d(feature) at one
point is only known once every point has been seen. A batch that holds
every point has seen them all: its own statistics are the global ones,
and the single pass of a per-point loss is exact for it too. On a GPU,
where a batch is as large as the device holds, that is the usual case,
and NCC now runs the moving model once instead of twice, as L2 does.

The single batch is tried first; when the device budget is smaller than
the points, or the batch runs out of memory, the losses are rewound and
the two passes run as before. The pass-count test now checks both on a
GPU: one batch, and several with a BatchSize of 100.
LibTorch spreads an operation over as many OpenMP threads as the calling
thread allows, which by default is every core. The work units, and the
threads that share the online points out, each ran their own team as
large as the machine: 24 threads of 24. The small operations of the main
thread, between the two passes of NCC and in the final reductions, also
waited for a whole team, which on a busy machine is long.

When several threads share the work out, each now runs its operations on
one OpenMP thread, and has its previous count back when it is done. A
single work unit keeps LibTorch's count: nothing else uses the cores
then. LibTorch has no other way to set it than at::set_num_threads,
which acts on the calling thread; with its native backend, one pool for
the process, nothing changes. Running the models from one thread
instead, on batches LibTorch spreads itself, measured slower still:
537 ms for L2 instead of 181 ms.

The guard, Impact::OneOpenMPThread, is in itkImpactOnlineInference.h,
which the elastix metric includes too.

Measured with benchmarks/ab.sh on the CPU, five rounds on a loaded
machine, all five won:

| case | before | after |
|---|---|---|
| online L2, 2000 points | 210 ms | 68 ms |
| online NCC, 2000 points | 320 ms | 120 ms |
| Static L2, 20000 points | 37.8 ms | 25.5 ms |

A single work unit is unchanged (689 ms and 744 ms, within the noise).
Dice compares the two feature vectors at each point: on a segmentation's
probabilities it is close to the share of points that agree, which the
background dominates. OrganDice takes each channel, an organ, over all
the points: 1 - the mean over the channels of
2 sum(f m) / (sum(f^2) + sum(m^2)), the soft Dice with a squared
denominator. A channel absent from both images counts as a match.

It is a global statistic, like NCC. The v4 metric accumulates its sums
in Static mode and combines them in closed form, and takes its
derivative from every point at once online. The coarse stage reads it
over each cell window from the window's moments, through a new
Loss::SimilarityFromMoments, which NCC and LNCC now answer too, in place
of the branch on their names. The Adam stage takes it over the image.

It is in every per-distance test: the finite-difference derivative in
both modes of the metric, the coarse cost against a brute force with
and without masks, the masked map loss, and the recovery of a
translation by the metric and by the coarse stage.
The v4 metric's online evaluation moves into itkImpactOnlineInference.h
as Impact::EvaluateOnline, so that the elastix metric can run the same
code: the batches (the threads on the CPU, one batch as large as the
device holds on a GPU, halved on running out of memory), the features,
the exact derivative of a global loss and the backward passes. The
metric only cuts the patches (Impact::OnlineHost::Sample), lends its
threads, and keeps its fixed features over a sampled point set.

The online features are now normalized as the configuration says, as
the Static mode and the elastix metric already did; the default, none,
leaves them as they were.
GenerateOutputs and GenerateOutputsAndJacobian ran the models on one
work unit's share of the points, and took the derivative of a global
loss from that share alone. The elastix metric now hands every point to
Impact::EvaluateOnline, as the ITKv4 metric does, and nothing calls
them any more.

Their two tests now test the shared evaluation: the loss and its
derivative do not depend on the batches or the parts the points are
split into, for NCC as for L2, and an anisotropic patch reaches the
model with its extents reversed.
The bench gains the online path of the elastix metric, on the same
pair, model and points as the ITKv4 metric's cases: every point handed
to Impact::EvaluateOnline through an Impact::OnlineHost that cuts the
patches with elastix's interpolators, an order-3 B-spline for the fixed
image and order 1 for the moving one. ab.sh now runs every ImpactPerf
test.
A work unit gathered every point of its share of the domain before
evaluating them in batches: a dense domain of 512^3 voxels held some
12 GiB of points whatever the batch. The points are now evaluated every
time a batch of them is gathered. Online the work unit still gathers
them all, for the evaluation that follows.

The batch divided by the transform's local parameters, which an identity
transform has none of: the division is by one at least.
A global loss's derivative with respect to a feature shrinks with the
number of points. Given as the seed of the backward pass through a
half-precision model, it rounded to zero from a few thousand points on,
and the metric returned a zero derivative away from an optimum.

The backward pass is linear in its seed: the seed is scaled by a power
of two to a largest magnitude near one, and the result divided by it. In
single precision the result is unchanged.
The fixed features kept over a sampled point set were reused as long as
the number of points, the plane seed and the fixed transform's
modification time were the same. Another point set of the same size
read the features of the previous one, and other feature subsets read
tensors of the previous channels, or failed on their shape.

The key now also holds the modification times of the metric, which
covers the models and which image, transform and point set it is given,
of the points container and of the fixed image, and the channels drawn.
Both closed forms of the derivative divided by the square of the
channel's denominator. In single precision the square reaches zero
while the denominator has not, so a channel that passed the empty test
gave 0/0: 64 points with fixed = moving = 1e-13 returned a NaN
derivative beside a value of zero.

The derivative is now written 2 (f - dice m) / D, with the Dice the
value reports, and the denominator is never squared. Elsewhere the
result is the same to single-precision rounding.
GetModelOutputsExample built the centre index of every layer and stored
it in each model configuration. Their two readers are gone, and the
elastix metric does not read them either: the field, its accessors and
the loop that filled it are removed.
NCC::SimilarityFromMoments only forwarded to the static
NCC::Correlation, and LNCC::SimilarityFromMoments to the static
LNCC::SquaredCorrelation, which nothing else calls outside LNCC itself.
Each pair is now one function.
@vboussot vboussot changed the title BUG: Draw the feature subsets once per evaluation, and speed up the v4 metric ENH: One online evaluation for both metrics, feature subsets drawn once per evaluation Nov 11, 2026
@vboussot
vboussot merged commit 139b293 into main Nov 11, 2026
5 checks passed
@vboussot
vboussot deleted the subset-once-per-evaluation branch November 19, 2026 20:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant