Repository navigation
ENH: One online evaluation for both metrics, feature subsets drawn once per evaluation - #6
Merged
Merged
Conversation
The v4 metric drew its feature subset (SubsetFeatures) from a generator per work unit, reseeded at every evaluation: per point in Static mode, per batch in Jacobian mode. A point's channels therefore depended on its rank in its work unit, on the batch size, and on whether a batch was replayed after running out of device memory. NCC accumulates its statistics channel by channel over the points, so its sums mixed channels. With 2 of 4 channels, the Jacobian derivative moved by 7 to 15% between batch sizes. The subsets are now drawn once per evaluation, before the threads start, from the seed the planes use, and every point compares the same channels. Two evaluations of the same parameters still compare the same channels. The batch-size test now keeps 2 of the 4 channels instead of all of them, and a new test compares 1 and 4 work units in Static mode. Both fail without this change.
Evaluate and EvaluateDerivative took both by value, so every call copied the subset, two or three times per point and per layer in Static mode.
Both stages called torch::manual_seed(Seed), which reseeds every torch generator of the process: a Python caller's random numbers restarted from the stage's seed after each run. The fine stage also drew its sampled points on the device, from the device's global generator, which the batch-budget probe draws from too, so a probe could shift them. Each stage now draws from a CPU generator of its own, seeded with Seed, and moves the draws to the device. A seed gives the same channels and points on the CPU and on a GPU. The new test checks that running either stage leaves torch's global generator where it was; it fails without this change.
The point-wise losses (L1, L2, Cosine, L1Cosine, Dice) summed a batch's values in float32 before adding the total to their double accumulator, so the value lost digits with the size of the batch: 2e-8 to 6e-8 of relative error over 20000 points. The online mode of the v4 metric and elastix's IMPACT metric both hand the losses whole batches, and finite differences of such a value are mostly rounding. The batch is now summed in double. The new test compares a batch of 20000 points with the same points fed one by one, and fails without this change.
In Static mode the v4 threader let ITK walk the points one by one. Each point went through ITK's ProcessVirtualPoint, which also computed an image gradient IMPACT never reads, then through ProcessPoint, which built a handful of one-row tensors to update the losses. Static now gathers a work unit's points, as the online mode already did, and evaluates them in batches: the feature vectors and the transform Jacobians are written into contiguous tensors, and each loss is updated once per batch. A batch holds at most about 4 MB of [points, channels, parameters] Jacobian, so that a dense domain under a B-spline transform stays bounded. The vector interpolator gains the buffer variants this needs, and a value-and-derivative evaluation that shares the B-spline weights between the two. On the lung CT test data, with 20000 points and 8 of a model's 32 channels, an evaluation takes 11 ms instead of 40 ms on 24 threads, and 136 ms instead of 468 ms on one.
ImageToImageMetricv4 smooths the whole moving image into a gradient image in Initialize(), for the metrics that read image gradients at their points. IMPACT compares features: the online mode takes its patch gradients from the moving interpolator, and Static mode differentiates the feature maps. The filter is now off by default. On the lung CT, Initialize() of an online metric takes 0.01 s instead of 0.08 s.
InterpolateVectorImageFunction held one BSplineInterpolateImageFunction per channel. Every channel of every point converted the point to a continuous index, computed the support, the weights and the mirrored indices, and allocated two or three vnl matrices to hold them, although all of these are the same for every channel. It now computes each channel's coefficients as that class does, and evaluates all the selected channels with one support and one set of weights, in the same order of operations, so the result is the same to the bit. SetActiveChannels() keeps a copy of the channels an evaluation compares, the channels of a voxel side by side, so that a point reads the fewest cache lines whether it compares all of a model's channels or a few; the v4 threader sets it once per evaluation. The per-channel images are gone, and a second SetInputImage() no longer keeps the first image's channels. The interpolator test now compares values and derivatives with BSplineInterpolateImageFunction channel by channel: 70 channels, a region that does not start at zero, anisotropic spacing, a rotated direction, through the active channels and through another subset. On 32 channels, a value takes 1.2 us instead of 8.4 us, a value and its derivative 2.5 us instead of 19.5 us.
On a GPU the online mode ran each work unit's points as batches of at most BatchSize (64 by default), one work unit after the other under the device lock. The model takes 20000 patches in about 0.08 s, but they went through it as some 300 batches. On a GPU the work units now only collect their points. Once they are done, all the threads cut the patches of a batch, and the models run on batches as large as the device holds. BatchSize now defaults to 0, which means that measured bound on a GPU, and 64 on the CPU, where the work units remain the parallelism and keep evaluating their own points. The measurement tries no batch larger than the largest an evaluation can run: a trial batch sized to the whole free memory took it from the other processes on a shared GPU. On the lung CT test data, with the 32-channel model, an evaluation of 20000 points takes 112 ms instead of 430 ms on a GPU; with 2000 points, 13 ms instead of 103 ms (L2) and 94 ms instead of 782 ms (NCC). The new test compares the GPU's value and derivative with the CPU's, and is skipped without a CUDA device.
NCC compares statistics taken over all the points, so its derivative with respect to one point's features is only known once every point has been seen. The online mode took every channel's d(feature)/d(moving coordinate) instead, from one backward pass per channel: 32 passes for a 32-channel layer. A global loss now takes two passes. The first, without gradient, computes the features of every point, from which the loss takes its value and its derivative with respect to each point's features. The second seeds one backward pass per batch with that derivative, as for L2. To see every point at once, the work units now only collect their points on the CPU too; the threads then evaluate them in shares, 64 points at a time as before. On the lung CT test data, with 2000 points and the 32-channel model, an NCC evaluation takes 15 ms instead of 94 ms on a GPU, and 121 ms instead of 1006 ms on the CPU. L2 is unchanged.
JacobianOnTheGpuMatchesTheCpu and StaticResultIsIndependentOfTheNumberOfWorkUnits each re-typed the metric set-up of JacobianResultIsIndependentOfBatchSize: the toy model configuration, distance, subset, PCA, seed, interpolator and the ramp images. One MakeMetric(loss, mode, device, transform) now builds it, keyed on the mode for the patch size, overlap, moving interpolator and refresh interval each test set, and each test keeps only what differs: the batch size, the device, the work units and the transform. The gradient-filter flags the two CPU tests cleared are gone: the metric's constructor already clears the moving one, and the fixed one is never read with the default gradient source.
SampleBatch and EvaluateStaticBatch each typed the same fill: the Jacobian with respect to the parameters from the cached temporaries, its copy to the float rows, the displacement-field offset and the index row. One FillTransformJacobian does it for both. ProcessPoint, which only throws, is defined inline in the header instead of a sixteen-line out-of-line definition.
A time measured on a shared machine moves with everything else the machine runs, and a CI runner is slower still. How often the online mode runs the models does not: it is its cost on any machine, and a test can hold it exactly. The new test counts the convolutions the toy model runs forward and backward, which are its passes, through a RecordFunction callback. A batch takes one forward pass of each model and one backward pass; NCC takes one more forward pass of the moving model; a value alone takes no backward pass; on a GPU the whole evaluation is one batch. Against the code before the two previous commits it fails: NCC made 24 backward passes instead of 6, one per channel. The comment on JacobianResultIsIndependentOfBatchSize described the closed form NCC used before; it now describes the two passes.
A time alone means little on a machine that also runs other jobs, as ours often does: the same evaluation moved by up to 3x within the same hour. Two revisions measured in alternation, under the same load, do compare. benchmarks/ab.sh builds two revisions (a reference and HEAD by default), each with benchmarks/ImpactPerfBench.cxx appended to its GTest driver so that both are measured by the same code, and runs them in turns. The bench times the v4 metric on the lung CT pair and the 32-channel layer of M258, Static and online, CPU and CUDA. The script prints the median of each revision per case, their ratio and how many rounds the second won, and exits 1 when it is slower by more than 10 % in the median and in three rounds out of four. IMPACT_PERF_LOCK holds a lock file around every run, for a GPU shared with other jobs. Measured with it, against the code before the online commits: NCC on the CPU takes 413 ms instead of 2051 ms, and L2 is unchanged (256 vs 282 ms).
Over a sampled point set, nothing the fixed features of a point depend on moves from one evaluation to the next: the fixed image and transform, the point, the plane its patch is cut on, the channels compared. The online mode still cut the fixed patches and ran the fixed model at every evaluation. They are now kept per point, from the first evaluation that computes them until Initialize(), a new plane seed or a modified fixed transform. A batch whose points all have theirs cuts no fixed patch and runs only the moving model. The second pass of a global loss never needed the fixed patches, and no longer cuts them either. The new test checks that the fixed model runs once per point, that a later evaluation gives the value and derivative of a metric evaluating afresh, and that a modified fixed transform has the features computed again. Measured with benchmarks/ab.sh on the CPU, 2000 points, five rounds on a loaded machine, all five won: an L2 evaluation takes 239 ms instead of 299 ms, NCC 299 ms instead of 534 ms.
A global loss (NCC) took two passes because d(loss)/d(feature) at one point is only known once every point has been seen. A batch that holds every point has seen them all: its own statistics are the global ones, and the single pass of a per-point loss is exact for it too. On a GPU, where a batch is as large as the device holds, that is the usual case, and NCC now runs the moving model once instead of twice, as L2 does. The single batch is tried first; when the device budget is smaller than the points, or the batch runs out of memory, the losses are rewound and the two passes run as before. The pass-count test now checks both on a GPU: one batch, and several with a BatchSize of 100.
LibTorch spreads an operation over as many OpenMP threads as the calling thread allows, which by default is every core. The work units, and the threads that share the online points out, each ran their own team as large as the machine: 24 threads of 24. The small operations of the main thread, between the two passes of NCC and in the final reductions, also waited for a whole team, which on a busy machine is long. When several threads share the work out, each now runs its operations on one OpenMP thread, and has its previous count back when it is done. A single work unit keeps LibTorch's count: nothing else uses the cores then. LibTorch has no other way to set it than at::set_num_threads, which acts on the calling thread; with its native backend, one pool for the process, nothing changes. Running the models from one thread instead, on batches LibTorch spreads itself, measured slower still: 537 ms for L2 instead of 181 ms. The guard, Impact::OneOpenMPThread, is in itkImpactOnlineInference.h, which the elastix metric includes too. Measured with benchmarks/ab.sh on the CPU, five rounds on a loaded machine, all five won: | case | before | after | |---|---|---| | online L2, 2000 points | 210 ms | 68 ms | | online NCC, 2000 points | 320 ms | 120 ms | | Static L2, 20000 points | 37.8 ms | 25.5 ms | A single work unit is unchanged (689 ms and 744 ms, within the noise).
Dice compares the two feature vectors at each point: on a segmentation's probabilities it is close to the share of points that agree, which the background dominates. OrganDice takes each channel, an organ, over all the points: 1 - the mean over the channels of 2 sum(f m) / (sum(f^2) + sum(m^2)), the soft Dice with a squared denominator. A channel absent from both images counts as a match. It is a global statistic, like NCC. The v4 metric accumulates its sums in Static mode and combines them in closed form, and takes its derivative from every point at once online. The coarse stage reads it over each cell window from the window's moments, through a new Loss::SimilarityFromMoments, which NCC and LNCC now answer too, in place of the branch on their names. The Adam stage takes it over the image. It is in every per-distance test: the finite-difference derivative in both modes of the metric, the coarse cost against a brute force with and without masks, the masked map loss, and the recovery of a translation by the metric and by the coarse stage.
The v4 metric's online evaluation moves into itkImpactOnlineInference.h as Impact::EvaluateOnline, so that the elastix metric can run the same code: the batches (the threads on the CPU, one batch as large as the device holds on a GPU, halved on running out of memory), the features, the exact derivative of a global loss and the backward passes. The metric only cuts the patches (Impact::OnlineHost::Sample), lends its threads, and keeps its fixed features over a sampled point set. The online features are now normalized as the configuration says, as the Static mode and the elastix metric already did; the default, none, leaves them as they were.
GenerateOutputs and GenerateOutputsAndJacobian ran the models on one work unit's share of the points, and took the derivative of a global loss from that share alone. The elastix metric now hands every point to Impact::EvaluateOnline, as the ITKv4 metric does, and nothing calls them any more. Their two tests now test the shared evaluation: the loss and its derivative do not depend on the batches or the parts the points are split into, for NCC as for L2, and an anisotropic patch reaches the model with its extents reversed.
The bench gains the online path of the elastix metric, on the same pair, model and points as the ITKv4 metric's cases: every point handed to Impact::EvaluateOnline through an Impact::OnlineHost that cuts the patches with elastix's interpolators, an order-3 B-spline for the fixed image and order 1 for the moving one. ab.sh now runs every ImpactPerf test.
A work unit gathered every point of its share of the domain before evaluating them in batches: a dense domain of 512^3 voxels held some 12 GiB of points whatever the batch. The points are now evaluated every time a batch of them is gathered. Online the work unit still gathers them all, for the evaluation that follows. The batch divided by the transform's local parameters, which an identity transform has none of: the division is by one at least.
A global loss's derivative with respect to a feature shrinks with the number of points. Given as the seed of the backward pass through a half-precision model, it rounded to zero from a few thousand points on, and the metric returned a zero derivative away from an optimum. The backward pass is linear in its seed: the seed is scaled by a power of two to a largest magnitude near one, and the result divided by it. In single precision the result is unchanged.
The fixed features kept over a sampled point set were reused as long as the number of points, the plane seed and the fixed transform's modification time were the same. Another point set of the same size read the features of the previous one, and other feature subsets read tensors of the previous channels, or failed on their shape. The key now also holds the modification times of the metric, which covers the models and which image, transform and point set it is given, of the points container and of the fixed image, and the channels drawn.
Both closed forms of the derivative divided by the square of the channel's denominator. In single precision the square reaches zero while the denominator has not, so a channel that passed the empty test gave 0/0: 64 points with fixed = moving = 1e-13 returned a NaN derivative beside a value of zero. The derivative is now written 2 (f - dice m) / D, with the Dice the value reports, and the denominator is never squared. Elsewhere the result is the same to single-precision rounding.
GetModelOutputsExample built the centre index of every layer and stored it in each model configuration. Their two readers are gone, and the elastix metric does not read them either: the field, its accessors and the loop that filled it are removed.
NCC::SimilarityFromMoments only forwarded to the static NCC::Correlation, and LNCC::SimilarityFromMoments to the static LNCC::SquaredCorrelation, which nothing else calls outside LNCC itself. Each pair is now one function.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes how the v4 metric and the ConvexAdam stages draw their random numbers, and makes the v4 metric faster: Static mode about 6x, the online (Jacobian) mode 4x to 50x on a GPU.
Fixes
torch::manual_seed(Seed), which reseeds the whole process, a Python caller's generator included. They now use a generator of their own.Speed, Static mode (lung CT test data, 20000 points, a 32-channel model)
BSplineInterpolateImageFunction, which the new interpolator test checks.Initialize().Speed, online mode (same data and model, 2000 points unless noted, best of 3)
BatchSizenow defaults to 0: that measured bound on a GPU, 64 on the CPU.Each fix comes with a test that fails without it. A new test checks that the GPU gives the CPU's value and derivative, for L2 and NCC; it is skipped without a CUDA device.
Added to this PR: one online evaluation for both metrics
The commits that built on this branch are folded in here rather than opened as a second PR.
Impact::EvaluateOnline(itkImpactOnlineInference.h) now runs the online (Jacobian) mode of the v4 metric, and is what the elastix metric calls too: batches as large as the device holds, a batch that runs out of memory halved and replayed after the losses are rewound, a global loss (NCC, OrganDice) taken in one pass when one batch holds every point and in two otherwise.GenerateOutputsandGenerateOutputsAndJacobian, the inference per work unit, are removed. A metric that still calls them, as elastix does while it pins ITKIMPACT 0.1.6, keeps building against 0.1.6 and moves toEvaluateOnlinewhen it raises its pin.OrganDice: the Dice of each organ over the image, for feature maps that are organ probabilities.benchmarks/ab.shbuilds two revisions and times them side by side (ImpactPerfBench), on the v4 metric and on the elastix online path.Fixed while reviewing this PR