[VPlan] Connect (MemRuntime|SCEV)Check blocks as VPlan transform (NFC). #143879
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Connect SCEV and memory runtime check block directly in VPlan as
VPIRBasicBlocks, removing ILV::emitSCEVChecks and ILV::emitMemRuntimeChecks.
The new logic is currently split across LoopVectorizationPlanner::addRuntimeChecks
which collects a list of {Condition, CheckBlock} pairs and performs some
checks and emits remarks if needed. The list of checks is then added to
VPlan in VPlanTransforms;:connectCheckBlocks.
|
@llvm/pr-subscribers-llvm-transforms @llvm/pr-subscribers-backend-risc-v Author: Florian Hahn (fhahn) ChangesConnect SCEV and memory runtime check block directly in VPlan as VPIRBasicBlocks, removing ILV::emitSCEVChecks and ILV::emitMemRuntimeChecks. The new logic is currently split across LoopVectorizationPlanner::addRuntimeChecks which collects a list of {Condition, CheckBlock} pairs and performs some checks and emits remarks if needed. The list of checks is then added to VPlan in VPlanTransforms::connectCheckBlocks. Patch is 22.20 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/143879.diff 6 Files Affected:
diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h b/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h
index 70f541d64b305..aae7c9a0075d1 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h
@@ -28,6 +28,10 @@
#include "llvm/ADT/SmallSet.h"
#include "llvm/Support/InstructionCost.h"
+namespace {
+class GeneratedRTChecks;
+}
+
namespace llvm {
class LoopInfo;
@@ -548,6 +552,9 @@ class LoopVectorizationPlanner {
VPRecipeBuilder &RecipeBuilder,
ElementCount MinVF);
+ /// Add the runtime checks from \p RTChecks to \p VPlan.
+ void addRuntimeChecks(VPlan &Plan, GeneratedRTChecks &RTChecks) const;
+
#ifndef NDEBUG
/// \return The most profitable vectorization factor for the available VPlans
/// and the cost of that VF.
diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
index 474f856d20461..50ee102b92b53 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
@@ -399,12 +399,6 @@ static cl::opt<bool> EnableEarlyExitVectorization(
cl::desc(
"Enable vectorization of early exit loops with uncountable exits."));
-// Likelyhood of bypassing the vectorized loop because assumptions about SCEV
-// variables not overflowing do not hold. See `emitSCEVChecks`.
-static constexpr uint32_t SCEVCheckBypassWeights[] = {1, 127};
-// Likelyhood of bypassing the vectorized loop because pointers overlap. See
-// `emitMemRuntimeChecks`.
-static constexpr uint32_t MemCheckBypassWeights[] = {1, 127};
// Likelyhood of bypassing the vectorized loop because there are zero trips left
// after prolog. See `emitIterationCountCheck`.
static constexpr uint32_t MinItersBypassWeights[] = {1, 127};
@@ -534,16 +528,6 @@ class InnerLoopVectorizer {
/// it overflows.
void emitIterationCountCheck(BasicBlock *Bypass);
- /// Emit a bypass check to see if all of the SCEV assumptions we've
- /// had to make are correct. Returns the block containing the checks or
- /// nullptr if no checks have been added.
- BasicBlock *emitSCEVChecks(BasicBlock *Bypass);
-
- /// Emit bypass checks to check any memory assumptions we may have made.
- /// Returns the block containing the checks or nullptr if no checks have been
- /// added.
- BasicBlock *emitMemRuntimeChecks(BasicBlock *Bypass);
-
/// Emit basic blocks (prefixed with \p Prefix) for the iteration check,
/// vector loop preheader, middle block and scalar preheader.
void createVectorLoopSkeleton(StringRef Prefix);
@@ -1782,7 +1766,6 @@ class GeneratedRTChecks {
SCEVExpander MemCheckExp;
bool CostTooHigh = false;
- const bool AddBranchWeights;
Loop *OuterLoop = nullptr;
@@ -1794,11 +1777,10 @@ class GeneratedRTChecks {
public:
GeneratedRTChecks(PredicatedScalarEvolution &PSE, DominatorTree *DT,
LoopInfo *LI, TargetTransformInfo *TTI,
- const DataLayout &DL, bool AddBranchWeights,
- TTI::TargetCostKind CostKind)
+ const DataLayout &DL, TTI::TargetCostKind CostKind)
: DT(DT), LI(LI), TTI(TTI), SCEVExp(*PSE.getSE(), DL, "scev.check"),
- MemCheckExp(*PSE.getSE(), DL, "scev.check"),
- AddBranchWeights(AddBranchWeights), PSE(PSE), CostKind(CostKind) {}
+ MemCheckExp(*PSE.getSE(), DL, "scev.check"), PSE(PSE),
+ CostKind(CostKind) {}
/// Generate runtime checks in SCEVCheckBlock and MemCheckBlock, so we can
/// accurately estimate the cost of the runtime checks. The blocks are
@@ -2016,61 +1998,35 @@ class GeneratedRTChecks {
/// Adds the generated SCEVCheckBlock before \p LoopVectorPreHeader and
/// adjusts the branches to branch to the vector preheader or \p Bypass,
/// depending on the generated condition.
- BasicBlock *emitSCEVChecks(BasicBlock *Bypass,
- BasicBlock *LoopVectorPreHeader) {
+ std::pair<Value *, BasicBlock *> emitSCEVChecks() {
using namespace llvm::PatternMatch;
if (!SCEVCheckCond || match(SCEVCheckCond, m_ZeroInt()))
- return nullptr;
+ return {nullptr, nullptr};
- auto *Pred = LoopVectorPreHeader->getSinglePredecessor();
- BranchInst::Create(LoopVectorPreHeader, SCEVCheckBlock);
-
- SCEVCheckBlock->getTerminator()->eraseFromParent();
- SCEVCheckBlock->moveBefore(LoopVectorPreHeader);
- Pred->getTerminator()->replaceSuccessorWith(LoopVectorPreHeader,
- SCEVCheckBlock);
-
- BranchInst &BI =
- *BranchInst::Create(Bypass, LoopVectorPreHeader, SCEVCheckCond);
- if (AddBranchWeights)
- setBranchWeights(BI, SCEVCheckBypassWeights, /*IsExpected=*/false);
- ReplaceInstWithInst(SCEVCheckBlock->getTerminator(), &BI);
- // Mark the check as used, to prevent it from being removed during cleanup.
+ Value *Cond = SCEVCheckCond;
SCEVCheckCond = nullptr;
AddedAnyChecks = true;
- return SCEVCheckBlock;
+ return {Cond, SCEVCheckBlock};
}
/// Adds the generated MemCheckBlock before \p LoopVectorPreHeader and adjusts
/// the branches to branch to the vector preheader or \p Bypass, depending on
/// the generated condition.
- BasicBlock *emitMemRuntimeChecks(BasicBlock *Bypass,
- BasicBlock *LoopVectorPreHeader) {
+ std::pair<Value *, BasicBlock *> emitMemRuntimeChecks() {
// Check if we generated code that checks in runtime if arrays overlap.
if (!MemRuntimeCheckCond)
- return nullptr;
-
- auto *Pred = LoopVectorPreHeader->getSinglePredecessor();
- Pred->getTerminator()->replaceSuccessorWith(LoopVectorPreHeader,
- MemCheckBlock);
-
- MemCheckBlock->moveBefore(LoopVectorPreHeader);
-
- BranchInst &BI =
- *BranchInst::Create(Bypass, LoopVectorPreHeader, MemRuntimeCheckCond);
- if (AddBranchWeights) {
- setBranchWeights(BI, MemCheckBypassWeights, /*IsExpected=*/false);
- }
- ReplaceInstWithInst(MemCheckBlock->getTerminator(), &BI);
- MemCheckBlock->getTerminator()->setDebugLoc(
- Pred->getTerminator()->getDebugLoc());
+ return {nullptr, nullptr};
// Mark the check as used, to prevent it from being removed during cleanup.
+ Value *Cond = MemRuntimeCheckCond;
MemRuntimeCheckCond = nullptr;
AddedAnyChecks = true;
- return MemCheckBlock;
+ return {Cond, MemCheckBlock};
}
+ BasicBlock *getSCEVCheckBlock() const { return SCEVCheckBlock; }
+ BasicBlock *getMemCheckBlock() const { return MemCheckBlock; }
+
/// Return true if any runtime checks have been added
bool hasChecks() const { return AddedAnyChecks; }
};
@@ -2451,53 +2407,6 @@ void InnerLoopVectorizer::emitIterationCountCheck(BasicBlock *Bypass) {
"Plan's entry must be TCCCheckBlock");
}
-BasicBlock *InnerLoopVectorizer::emitSCEVChecks(BasicBlock *Bypass) {
- BasicBlock *const SCEVCheckBlock =
- RTChecks.emitSCEVChecks(Bypass, LoopVectorPreHeader);
- if (!SCEVCheckBlock)
- return nullptr;
-
- assert((!Cost->OptForSize ||
- Cost->Hints->getForce() == LoopVectorizeHints::FK_Enabled) &&
- "Cannot SCEV check stride or overflow when optimizing for size");
-
- introduceCheckBlockInVPlan(SCEVCheckBlock);
- return SCEVCheckBlock;
-}
-
-BasicBlock *InnerLoopVectorizer::emitMemRuntimeChecks(BasicBlock *Bypass) {
- BasicBlock *const MemCheckBlock =
- RTChecks.emitMemRuntimeChecks(Bypass, LoopVectorPreHeader);
-
- // Check if we generated code that checks in runtime if arrays overlap. We put
- // the checks into a separate block to make the more common case of few
- // elements faster.
- if (!MemCheckBlock)
- return nullptr;
-
- // VPlan-native path does not do any analysis for runtime checks currently.
- assert((!EnableVPlanNativePath || OrigLoop->begin() == OrigLoop->end()) &&
- "Runtime checks are not supported for outer loops yet");
-
- if (Cost->OptForSize) {
- assert(Cost->Hints->getForce() == LoopVectorizeHints::FK_Enabled &&
- "Cannot emit memory checks when optimizing for size, unless forced "
- "to vectorize.");
- ORE->emit([&]() {
- return OptimizationRemarkAnalysis(DEBUG_TYPE, "VectorizationCodeSize",
- OrigLoop->getStartLoc(),
- OrigLoop->getHeader())
- << "Code-size may be reduced by not forcing "
- "vectorization, or by source-code modifications "
- "eliminating the need for runtime checks "
- "(e.g., adding 'restrict').";
- });
- }
-
- introduceCheckBlockInVPlan(MemCheckBlock);
- return MemCheckBlock;
-}
-
/// Replace \p VPBB with a VPIRBasicBlock wrapping \p IRBB. All recipes from \p
/// VPBB are moved to the end of the newly created VPIRBasicBlock. VPBB must
/// have a single predecessor, which is rewired to the new VPIRBasicBlock. All
@@ -2614,15 +2523,6 @@ BasicBlock *InnerLoopVectorizer::createVectorizedLoopSkeleton() {
// to the scalar loop.
emitIterationCountCheck(LoopScalarPreHeader);
- // Generate the code to check any assumptions that we've made for SCEV
- // expressions.
- emitSCEVChecks(LoopScalarPreHeader);
-
- // Generate the code that checks in runtime if arrays overlap. We put the
- // checks into a separate block to make the more common case of few elements
- // faster.
- emitMemRuntimeChecks(LoopScalarPreHeader);
-
replaceVPBBWithIRVPBB(Plan.getScalarPreheader(), LoopScalarPreHeader);
return LoopVectorPreHeader;
}
@@ -7312,6 +7212,11 @@ DenseMap<const SCEV *, Value *> LoopVectorizationPlanner::executePlan(
VPlanTransforms::runPass(VPlanTransforms::unrollByUF, BestVPlan, BestUF,
OrigLoop->getHeader()->getContext());
VPlanTransforms::runPass(VPlanTransforms::materializeBroadcasts, BestVPlan);
+
+ if (!VectorizingEpilogue)
+ addRuntimeChecks(BestVPlan, ILV.RTChecks);
+
+ VPBasicBlock *VectorPH = cast<VPBasicBlock>(BestVPlan.getVectorPreheader());
VPlanTransforms::optimizeForVFAndUF(BestVPlan, BestVF, BestUF, PSE);
VPlanTransforms::simplifyRecipes(BestVPlan, *Legal->getWidestInductionType());
VPlanTransforms::narrowInterleaveGroups(
@@ -7359,7 +7264,8 @@ DenseMap<const SCEV *, Value *> LoopVectorizationPlanner::executePlan(
// 1. Set up the skeleton for vectorization, including vector pre-header and
// middle block. The vector loop is created during VPlan execution.
- VPBasicBlock *VectorPH = cast<VPBasicBlock>(Entry->getSuccessors()[1]);
+ BasicBlock *EntryBB =
+ cast<VPIRBasicBlock>(BestVPlan.getEntry())->getIRBasicBlock();
State.CFG.PrevBB = ILV.createVectorizedLoopSkeleton();
if (VectorizingEpilogue)
VPlanTransforms::removeDeadRecipes(BestVPlan);
@@ -7383,6 +7289,12 @@ DenseMap<const SCEV *, Value *> LoopVectorizationPlanner::executePlan(
ILV.getOrCreateVectorTripCount(ILV.LoopVectorPreHeader), State);
replaceVPBBWithIRVPBB(VectorPH, State.CFG.PrevBB);
+ // Move check blocks to their final position.
+ if (BasicBlock *MemCheckBlock = ILV.RTChecks.getMemCheckBlock())
+ MemCheckBlock->moveAfter(EntryBB);
+ if (BasicBlock *SCEVCheckBlock = ILV.RTChecks.getSCEVCheckBlock())
+ SCEVCheckBlock->moveAfter(EntryBB);
+
BestVPlan.execute(&State);
// 2.5 When vectorizing the epilogue, fix reduction resume values from the
@@ -7483,15 +7395,6 @@ BasicBlock *EpilogueVectorizerMainLoop::createEpilogueVectorizedLoopSkeleton() {
emitIterationCountCheck(LoopScalarPreHeader, true);
EPI.EpilogueIterationCountCheck->setName("iter.check");
- // Generate the code to check any assumptions that we've made for SCEV
- // expressions.
- EPI.SCEVSafetyCheck = emitSCEVChecks(LoopScalarPreHeader);
-
- // Generate the code that checks at runtime if arrays overlap. We put the
- // checks into a separate block to make the more common case of few elements
- // faster.
- EPI.MemSafetyCheck = emitMemRuntimeChecks(LoopScalarPreHeader);
-
// Generate the iteration count check for the main loop, *after* the check
// for the epilogue loop, so that the path-length is shorter for the case
// that goes directly through the vector epilogue. The longer-path length for
@@ -7608,6 +7511,13 @@ EpilogueVectorizerEpilogueLoop::createEpilogueVectorizedLoopSkeleton() {
EPI.EpilogueIterationCountCheck->getTerminator()->replaceUsesOfWith(
VecEpilogueIterationCountCheck, LoopScalarPreHeader);
+ BasicBlock *SCEVCheckBlock = RTChecks.getSCEVCheckBlock();
+ if (SCEVCheckBlock && SCEVCheckBlock->hasNPredecessorsOrMore(1))
+ EPI.SCEVSafetyCheck = SCEVCheckBlock;
+
+ BasicBlock *MemCheckBlock = RTChecks.getMemCheckBlock();
+ if (MemCheckBlock && MemCheckBlock->hasNPredecessorsOrMore(1))
+ EPI.MemSafetyCheck = MemCheckBlock;
if (EPI.SCEVSafetyCheck)
EPI.SCEVSafetyCheck->getTerminator()->replaceUsesOfWith(
VecEpilogueIterationCountCheck, LoopScalarPreHeader);
@@ -9325,6 +9235,47 @@ void LoopVectorizationPlanner::adjustRecipesForReductions(
VPlanTransforms::runPass(VPlanTransforms::clearReductionWrapFlags, *Plan);
}
+void LoopVectorizationPlanner::addRuntimeChecks(
+ VPlan &Plan, GeneratedRTChecks &RTChecks) const {
+ SmallVector<std::pair<VPValue *, VPIRBasicBlock *>> Checks;
+ const auto &[SCEVCheckCond, SCEVCheckBlock] = RTChecks.emitSCEVChecks();
+ if (SCEVCheckBlock) {
+ assert((!CM.OptForSize ||
+ CM.Hints->getForce() == LoopVectorizeHints::FK_Enabled) &&
+ "Cannot SCEV check stride or overflow when optimizing for size");
+ Checks.emplace_back(Plan.getOrAddLiveIn(SCEVCheckCond),
+ Plan.createVPIRBasicBlock(SCEVCheckBlock));
+ }
+ const auto &[MemCheckCond, MemCheckBlock] = RTChecks.emitMemRuntimeChecks();
+ if (MemCheckBlock) {
+ // VPlan-native path does not do any analysis for runtime checks
+ // currently.
+ assert((!EnableVPlanNativePath || OrigLoop->begin() == OrigLoop->end()) &&
+ "Runtime checks are not supported for outer loops yet");
+
+ if (CM.OptForSize) {
+ assert(
+ CM.Hints->getForce() == LoopVectorizeHints::FK_Enabled &&
+ "Cannot emit memory checks when optimizing for size, unless forced "
+ "to vectorize.");
+ ORE->emit([&]() {
+ return OptimizationRemarkAnalysis(DEBUG_TYPE, "VectorizationCodeSize",
+ OrigLoop->getStartLoc(),
+ OrigLoop->getHeader())
+ << "Code-size may be reduced by not forcing "
+ "vectorization, or by source-code modifications "
+ "eliminating the need for runtime checks "
+ "(e.g., adding 'restrict').";
+ });
+ }
+ Checks.emplace_back(Plan.getOrAddLiveIn(MemCheckCond),
+ Plan.createVPIRBasicBlock(MemCheckBlock));
+ }
+ VPlanTransforms::connectCheckBlocks(
+ Plan, Checks,
+ hasBranchWeightMD(*OrigLoop->getLoopLatch()->getTerminator()));
+}
+
void VPDerivedIVRecipe::execute(VPTransformState &State) {
assert(!State.Lane && "VPDerivedIVRecipe being replicated.");
@@ -9446,10 +9397,7 @@ static bool processLoopInVPlanNativePath(
VPlan &BestPlan = LVP.getPlanFor(VF.Width);
{
- bool AddBranchWeights =
- hasBranchWeightMD(*L->getLoopLatch()->getTerminator());
- GeneratedRTChecks Checks(PSE, DT, LI, TTI, F->getDataLayout(),
- AddBranchWeights, CM.CostKind);
+ GeneratedRTChecks Checks(PSE, DT, LI, TTI, F->getDataLayout(), CM.CostKind);
InnerLoopVectorizer LB(L, PSE, LI, DT, TLI, TTI, AC, ORE, VF.Width,
VF.Width, 1, &CM, BFI, PSI, Checks, BestPlan);
LLVM_DEBUG(dbgs() << "Vectorizing outer loop in \""
@@ -10085,10 +10033,7 @@ bool LoopVectorizePass::processLoop(Loop *L) {
if (ORE->allowExtraAnalysis(LV_NAME))
LVP.emitInvalidCostRemarks(ORE);
- bool AddBranchWeights =
- hasBranchWeightMD(*L->getLoopLatch()->getTerminator());
- GeneratedRTChecks Checks(PSE, DT, LI, TTI, F->getDataLayout(),
- AddBranchWeights, CM.CostKind);
+ GeneratedRTChecks Checks(PSE, DT, LI, TTI, F->getDataLayout(), CM.CostKind);
if (LVP.hasPlanWithVF(VF.Width)) {
// Select the interleave count.
IC = CM.selectInterleaveCount(LVP.getPlanFor(VF.Width), VF.Width, VF.Cost);
diff --git a/llvm/lib/Transforms/Vectorize/VPlan.cpp b/llvm/lib/Transforms/Vectorize/VPlan.cpp
index 1838562f26b82..8068a6b3b968f 100644
--- a/llvm/lib/Transforms/Vectorize/VPlan.cpp
+++ b/llvm/lib/Transforms/Vectorize/VPlan.cpp
@@ -484,7 +484,8 @@ void VPBasicBlock::connectToPredecessors(VPTransformState &State) {
unsigned idx = PredVPSuccessors.front() == this ? 0 : 1;
assert((TermBr && (!TermBr->getSuccessor(idx) ||
(isa<VPIRBasicBlock>(this) &&
- TermBr->getSuccessor(idx) == NewBB))) &&
+ (TermBr->getSuccessor(idx) == NewBB ||
+ PredVPBlock == getPlan()->getEntry())))) &&
"Trying to reset an existing successor block.");
TermBr->setSuccessor(idx, NewBB);
}
diff --git a/llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp b/llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp
index 593e5063802ba..a55e95ef274b7 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp
+++ b/llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp
@@ -20,6 +20,7 @@
#include "llvm/Analysis/LoopInfo.h"
#include "llvm/Analysis/LoopIterator.h"
#include "llvm/Analysis/ScalarEvolution.h"
+#include "llvm/IR/MDBuilder.h"
#define DEBUG_TYPE "vplan"
@@ -589,3 +590,41 @@ void VPlanTransforms::createLoopRegions(VPlan &Plan) {
TopRegion->setName("vector loop");
TopRegion->getEntryBasicBlock()->setName("vector.body");
}
+
+// Likelyhood of bypassing the vectorized loop because SCEV assumptions or
+// memory runtime checks.
+static constexpr uint32_t CheckBypassWeights[] = {1, 127};
+
+void VPlanTransforms::connectCheckBlocks(
+ VPlan &Plan, ArrayRef<std::pair<VPValue *, VPIRBasicBlock *>> Checks,
+ bool AddBranchWeights) {
+ VPBlockBase *VectorPH = Plan.getVectorPreheader();
+ VPBlockBase *ScalarPH = Plan.getScalarPreheader();
+ for (const auto &[Cond, CheckBlock] : Checks) {
+ VPBlockBase *PreVectorPH = VectorPH->getSinglePredecessor();
+ VPBlockUtils::insertOnEdge(PreVectorPH, VectorPH, CheckBlock);
+ VPBlockUtils::connectBlocks(CheckBlock, ScalarPH);
+ CheckBlock->swapSuccessors();
+
+ // We just connected a new block to the scalar preheader. Update all
+ // VPPhis by adding an incoming value for it, replicating the last value.
+ unsigned NumPredecessors = ScalarPH->getNumPredecessors();
+ for (VPRecipeBase &R : cast<VPBasicBlock>(ScalarPH)->phis()) {
+ assert(isa<VPPhi>(&R) && "Phi expected to be VPPhi");
+ assert(cast<VPPhi>(&R)->getNumIncoming() == NumPredecessors - 1 &&
+ "must have incoming values for all operands");
+ R.addOperand(R.getOperand(NumPredecessors - 2));
+ }
+
+ VPIRMetadata VPBranchWeights;
+ auto *Term = VPBuilder(CheckBlock)
+ .createNaryOp(VPInstruction::BranchOnCond, {Cond},
+ Plan.getCanonicalIV()->getDebugLoc());
+ if (AddBranchWeights) {
+ MDBuilder MDB(Plan.getScalarHeader()->getIRBasicBlock()->getContext());
+ MDNode *BranchWeights =
+ MDB.createBranchWeights(CheckBypassWeights, /*IsExpected=*/false);
+ Term->addMetadata(LLVMContext::MD_prof, BranchWeights);
+ }
+ }
+}
diff --git a/llvm/lib/Transforms/Vectorize/VPlanTransforms.h b/llvm/lib/Transforms/Vectorize/VPlanTransforms.h
index 34e2de4eb3b74..c13cabc87ce31 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanTransforms.h
+++ b/llvm/lib/Transforms/Vectorize/VPlanTransforms.h
@@ -74,6 +74,13 @@ struct VPlanTransforms {
/// flat CFG into a hierarchical CFG.
static void createLoopRegions(VPlan &Plan);
+ /// Connect the blocks in \p Checks to \p Plan, using the corresponding
+ /// VPValue as branch condition.
+ static void
+ connectCheckBlocks(VPlan &Plan,
+ ArrayRef<std::pair<VPValue *, VPIRBasicBlock *>> ...
[truncated]
|
|
@llvm/pr-subscribers-vectorizers Author: Florian Hahn (fhahn) ChangesConnect SCEV and memory runtime check block directly in VPlan as VPIRBasicBlocks, removing ILV::emitSCEVChecks and ILV::emitMemRuntimeChecks. The new logic is currently split across LoopVectorizationPlanner::addRuntimeChecks which collects a list of {Condition, CheckBlock} pairs and performs some checks and emits remarks if needed. The list of checks is then added to VPlan in VPlanTransforms::connectCheckBlocks. Patch is 22.20 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/143879.diff 6 Files Affected:
diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h b/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h
index 70f541d64b305..aae7c9a0075d1 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h
@@ -28,6 +28,10 @@
#include "llvm/ADT/SmallSet.h"
#include "llvm/Support/InstructionCost.h"
+namespace {
+class GeneratedRTChecks;
+}
+
namespace llvm {
class LoopInfo;
@@ -548,6 +552,9 @@ class LoopVectorizationPlanner {
VPRecipeBuilder &RecipeBuilder,
ElementCount MinVF);
+ /// Add the runtime checks from \p RTChecks to \p VPlan.
+ void addRuntimeChecks(VPlan &Plan, GeneratedRTChecks &RTChecks) const;
+
#ifndef NDEBUG
/// \return The most profitable vectorization factor for the available VPlans
/// and the cost of that VF.
diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
index 474f856d20461..50ee102b92b53 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
@@ -399,12 +399,6 @@ static cl::opt<bool> EnableEarlyExitVectorization(
cl::desc(
"Enable vectorization of early exit loops with uncountable exits."));
-// Likelyhood of bypassing the vectorized loop because assumptions about SCEV
-// variables not overflowing do not hold. See `emitSCEVChecks`.
-static constexpr uint32_t SCEVCheckBypassWeights[] = {1, 127};
-// Likelyhood of bypassing the vectorized loop because pointers overlap. See
-// `emitMemRuntimeChecks`.
-static constexpr uint32_t MemCheckBypassWeights[] = {1, 127};
// Likelyhood of bypassing the vectorized loop because there are zero trips left
// after prolog. See `emitIterationCountCheck`.
static constexpr uint32_t MinItersBypassWeights[] = {1, 127};
@@ -534,16 +528,6 @@ class InnerLoopVectorizer {
/// it overflows.
void emitIterationCountCheck(BasicBlock *Bypass);
- /// Emit a bypass check to see if all of the SCEV assumptions we've
- /// had to make are correct. Returns the block containing the checks or
- /// nullptr if no checks have been added.
- BasicBlock *emitSCEVChecks(BasicBlock *Bypass);
-
- /// Emit bypass checks to check any memory assumptions we may have made.
- /// Returns the block containing the checks or nullptr if no checks have been
- /// added.
- BasicBlock *emitMemRuntimeChecks(BasicBlock *Bypass);
-
/// Emit basic blocks (prefixed with \p Prefix) for the iteration check,
/// vector loop preheader, middle block and scalar preheader.
void createVectorLoopSkeleton(StringRef Prefix);
@@ -1782,7 +1766,6 @@ class GeneratedRTChecks {
SCEVExpander MemCheckExp;
bool CostTooHigh = false;
- const bool AddBranchWeights;
Loop *OuterLoop = nullptr;
@@ -1794,11 +1777,10 @@ class GeneratedRTChecks {
public:
GeneratedRTChecks(PredicatedScalarEvolution &PSE, DominatorTree *DT,
LoopInfo *LI, TargetTransformInfo *TTI,
- const DataLayout &DL, bool AddBranchWeights,
- TTI::TargetCostKind CostKind)
+ const DataLayout &DL, TTI::TargetCostKind CostKind)
: DT(DT), LI(LI), TTI(TTI), SCEVExp(*PSE.getSE(), DL, "scev.check"),
- MemCheckExp(*PSE.getSE(), DL, "scev.check"),
- AddBranchWeights(AddBranchWeights), PSE(PSE), CostKind(CostKind) {}
+ MemCheckExp(*PSE.getSE(), DL, "scev.check"), PSE(PSE),
+ CostKind(CostKind) {}
/// Generate runtime checks in SCEVCheckBlock and MemCheckBlock, so we can
/// accurately estimate the cost of the runtime checks. The blocks are
@@ -2016,61 +1998,35 @@ class GeneratedRTChecks {
/// Adds the generated SCEVCheckBlock before \p LoopVectorPreHeader and
/// adjusts the branches to branch to the vector preheader or \p Bypass,
/// depending on the generated condition.
- BasicBlock *emitSCEVChecks(BasicBlock *Bypass,
- BasicBlock *LoopVectorPreHeader) {
+ std::pair<Value *, BasicBlock *> emitSCEVChecks() {
using namespace llvm::PatternMatch;
if (!SCEVCheckCond || match(SCEVCheckCond, m_ZeroInt()))
- return nullptr;
+ return {nullptr, nullptr};
- auto *Pred = LoopVectorPreHeader->getSinglePredecessor();
- BranchInst::Create(LoopVectorPreHeader, SCEVCheckBlock);
-
- SCEVCheckBlock->getTerminator()->eraseFromParent();
- SCEVCheckBlock->moveBefore(LoopVectorPreHeader);
- Pred->getTerminator()->replaceSuccessorWith(LoopVectorPreHeader,
- SCEVCheckBlock);
-
- BranchInst &BI =
- *BranchInst::Create(Bypass, LoopVectorPreHeader, SCEVCheckCond);
- if (AddBranchWeights)
- setBranchWeights(BI, SCEVCheckBypassWeights, /*IsExpected=*/false);
- ReplaceInstWithInst(SCEVCheckBlock->getTerminator(), &BI);
- // Mark the check as used, to prevent it from being removed during cleanup.
+ Value *Cond = SCEVCheckCond;
SCEVCheckCond = nullptr;
AddedAnyChecks = true;
- return SCEVCheckBlock;
+ return {Cond, SCEVCheckBlock};
}
/// Adds the generated MemCheckBlock before \p LoopVectorPreHeader and adjusts
/// the branches to branch to the vector preheader or \p Bypass, depending on
/// the generated condition.
- BasicBlock *emitMemRuntimeChecks(BasicBlock *Bypass,
- BasicBlock *LoopVectorPreHeader) {
+ std::pair<Value *, BasicBlock *> emitMemRuntimeChecks() {
// Check if we generated code that checks in runtime if arrays overlap.
if (!MemRuntimeCheckCond)
- return nullptr;
-
- auto *Pred = LoopVectorPreHeader->getSinglePredecessor();
- Pred->getTerminator()->replaceSuccessorWith(LoopVectorPreHeader,
- MemCheckBlock);
-
- MemCheckBlock->moveBefore(LoopVectorPreHeader);
-
- BranchInst &BI =
- *BranchInst::Create(Bypass, LoopVectorPreHeader, MemRuntimeCheckCond);
- if (AddBranchWeights) {
- setBranchWeights(BI, MemCheckBypassWeights, /*IsExpected=*/false);
- }
- ReplaceInstWithInst(MemCheckBlock->getTerminator(), &BI);
- MemCheckBlock->getTerminator()->setDebugLoc(
- Pred->getTerminator()->getDebugLoc());
+ return {nullptr, nullptr};
// Mark the check as used, to prevent it from being removed during cleanup.
+ Value *Cond = MemRuntimeCheckCond;
MemRuntimeCheckCond = nullptr;
AddedAnyChecks = true;
- return MemCheckBlock;
+ return {Cond, MemCheckBlock};
}
+ BasicBlock *getSCEVCheckBlock() const { return SCEVCheckBlock; }
+ BasicBlock *getMemCheckBlock() const { return MemCheckBlock; }
+
/// Return true if any runtime checks have been added
bool hasChecks() const { return AddedAnyChecks; }
};
@@ -2451,53 +2407,6 @@ void InnerLoopVectorizer::emitIterationCountCheck(BasicBlock *Bypass) {
"Plan's entry must be TCCCheckBlock");
}
-BasicBlock *InnerLoopVectorizer::emitSCEVChecks(BasicBlock *Bypass) {
- BasicBlock *const SCEVCheckBlock =
- RTChecks.emitSCEVChecks(Bypass, LoopVectorPreHeader);
- if (!SCEVCheckBlock)
- return nullptr;
-
- assert((!Cost->OptForSize ||
- Cost->Hints->getForce() == LoopVectorizeHints::FK_Enabled) &&
- "Cannot SCEV check stride or overflow when optimizing for size");
-
- introduceCheckBlockInVPlan(SCEVCheckBlock);
- return SCEVCheckBlock;
-}
-
-BasicBlock *InnerLoopVectorizer::emitMemRuntimeChecks(BasicBlock *Bypass) {
- BasicBlock *const MemCheckBlock =
- RTChecks.emitMemRuntimeChecks(Bypass, LoopVectorPreHeader);
-
- // Check if we generated code that checks in runtime if arrays overlap. We put
- // the checks into a separate block to make the more common case of few
- // elements faster.
- if (!MemCheckBlock)
- return nullptr;
-
- // VPlan-native path does not do any analysis for runtime checks currently.
- assert((!EnableVPlanNativePath || OrigLoop->begin() == OrigLoop->end()) &&
- "Runtime checks are not supported for outer loops yet");
-
- if (Cost->OptForSize) {
- assert(Cost->Hints->getForce() == LoopVectorizeHints::FK_Enabled &&
- "Cannot emit memory checks when optimizing for size, unless forced "
- "to vectorize.");
- ORE->emit([&]() {
- return OptimizationRemarkAnalysis(DEBUG_TYPE, "VectorizationCodeSize",
- OrigLoop->getStartLoc(),
- OrigLoop->getHeader())
- << "Code-size may be reduced by not forcing "
- "vectorization, or by source-code modifications "
- "eliminating the need for runtime checks "
- "(e.g., adding 'restrict').";
- });
- }
-
- introduceCheckBlockInVPlan(MemCheckBlock);
- return MemCheckBlock;
-}
-
/// Replace \p VPBB with a VPIRBasicBlock wrapping \p IRBB. All recipes from \p
/// VPBB are moved to the end of the newly created VPIRBasicBlock. VPBB must
/// have a single predecessor, which is rewired to the new VPIRBasicBlock. All
@@ -2614,15 +2523,6 @@ BasicBlock *InnerLoopVectorizer::createVectorizedLoopSkeleton() {
// to the scalar loop.
emitIterationCountCheck(LoopScalarPreHeader);
- // Generate the code to check any assumptions that we've made for SCEV
- // expressions.
- emitSCEVChecks(LoopScalarPreHeader);
-
- // Generate the code that checks in runtime if arrays overlap. We put the
- // checks into a separate block to make the more common case of few elements
- // faster.
- emitMemRuntimeChecks(LoopScalarPreHeader);
-
replaceVPBBWithIRVPBB(Plan.getScalarPreheader(), LoopScalarPreHeader);
return LoopVectorPreHeader;
}
@@ -7312,6 +7212,11 @@ DenseMap<const SCEV *, Value *> LoopVectorizationPlanner::executePlan(
VPlanTransforms::runPass(VPlanTransforms::unrollByUF, BestVPlan, BestUF,
OrigLoop->getHeader()->getContext());
VPlanTransforms::runPass(VPlanTransforms::materializeBroadcasts, BestVPlan);
+
+ if (!VectorizingEpilogue)
+ addRuntimeChecks(BestVPlan, ILV.RTChecks);
+
+ VPBasicBlock *VectorPH = cast<VPBasicBlock>(BestVPlan.getVectorPreheader());
VPlanTransforms::optimizeForVFAndUF(BestVPlan, BestVF, BestUF, PSE);
VPlanTransforms::simplifyRecipes(BestVPlan, *Legal->getWidestInductionType());
VPlanTransforms::narrowInterleaveGroups(
@@ -7359,7 +7264,8 @@ DenseMap<const SCEV *, Value *> LoopVectorizationPlanner::executePlan(
// 1. Set up the skeleton for vectorization, including vector pre-header and
// middle block. The vector loop is created during VPlan execution.
- VPBasicBlock *VectorPH = cast<VPBasicBlock>(Entry->getSuccessors()[1]);
+ BasicBlock *EntryBB =
+ cast<VPIRBasicBlock>(BestVPlan.getEntry())->getIRBasicBlock();
State.CFG.PrevBB = ILV.createVectorizedLoopSkeleton();
if (VectorizingEpilogue)
VPlanTransforms::removeDeadRecipes(BestVPlan);
@@ -7383,6 +7289,12 @@ DenseMap<const SCEV *, Value *> LoopVectorizationPlanner::executePlan(
ILV.getOrCreateVectorTripCount(ILV.LoopVectorPreHeader), State);
replaceVPBBWithIRVPBB(VectorPH, State.CFG.PrevBB);
+ // Move check blocks to their final position.
+ if (BasicBlock *MemCheckBlock = ILV.RTChecks.getMemCheckBlock())
+ MemCheckBlock->moveAfter(EntryBB);
+ if (BasicBlock *SCEVCheckBlock = ILV.RTChecks.getSCEVCheckBlock())
+ SCEVCheckBlock->moveAfter(EntryBB);
+
BestVPlan.execute(&State);
// 2.5 When vectorizing the epilogue, fix reduction resume values from the
@@ -7483,15 +7395,6 @@ BasicBlock *EpilogueVectorizerMainLoop::createEpilogueVectorizedLoopSkeleton() {
emitIterationCountCheck(LoopScalarPreHeader, true);
EPI.EpilogueIterationCountCheck->setName("iter.check");
- // Generate the code to check any assumptions that we've made for SCEV
- // expressions.
- EPI.SCEVSafetyCheck = emitSCEVChecks(LoopScalarPreHeader);
-
- // Generate the code that checks at runtime if arrays overlap. We put the
- // checks into a separate block to make the more common case of few elements
- // faster.
- EPI.MemSafetyCheck = emitMemRuntimeChecks(LoopScalarPreHeader);
-
// Generate the iteration count check for the main loop, *after* the check
// for the epilogue loop, so that the path-length is shorter for the case
// that goes directly through the vector epilogue. The longer-path length for
@@ -7608,6 +7511,13 @@ EpilogueVectorizerEpilogueLoop::createEpilogueVectorizedLoopSkeleton() {
EPI.EpilogueIterationCountCheck->getTerminator()->replaceUsesOfWith(
VecEpilogueIterationCountCheck, LoopScalarPreHeader);
+ BasicBlock *SCEVCheckBlock = RTChecks.getSCEVCheckBlock();
+ if (SCEVCheckBlock && SCEVCheckBlock->hasNPredecessorsOrMore(1))
+ EPI.SCEVSafetyCheck = SCEVCheckBlock;
+
+ BasicBlock *MemCheckBlock = RTChecks.getMemCheckBlock();
+ if (MemCheckBlock && MemCheckBlock->hasNPredecessorsOrMore(1))
+ EPI.MemSafetyCheck = MemCheckBlock;
if (EPI.SCEVSafetyCheck)
EPI.SCEVSafetyCheck->getTerminator()->replaceUsesOfWith(
VecEpilogueIterationCountCheck, LoopScalarPreHeader);
@@ -9325,6 +9235,47 @@ void LoopVectorizationPlanner::adjustRecipesForReductions(
VPlanTransforms::runPass(VPlanTransforms::clearReductionWrapFlags, *Plan);
}
+void LoopVectorizationPlanner::addRuntimeChecks(
+ VPlan &Plan, GeneratedRTChecks &RTChecks) const {
+ SmallVector<std::pair<VPValue *, VPIRBasicBlock *>> Checks;
+ const auto &[SCEVCheckCond, SCEVCheckBlock] = RTChecks.emitSCEVChecks();
+ if (SCEVCheckBlock) {
+ assert((!CM.OptForSize ||
+ CM.Hints->getForce() == LoopVectorizeHints::FK_Enabled) &&
+ "Cannot SCEV check stride or overflow when optimizing for size");
+ Checks.emplace_back(Plan.getOrAddLiveIn(SCEVCheckCond),
+ Plan.createVPIRBasicBlock(SCEVCheckBlock));
+ }
+ const auto &[MemCheckCond, MemCheckBlock] = RTChecks.emitMemRuntimeChecks();
+ if (MemCheckBlock) {
+ // VPlan-native path does not do any analysis for runtime checks
+ // currently.
+ assert((!EnableVPlanNativePath || OrigLoop->begin() == OrigLoop->end()) &&
+ "Runtime checks are not supported for outer loops yet");
+
+ if (CM.OptForSize) {
+ assert(
+ CM.Hints->getForce() == LoopVectorizeHints::FK_Enabled &&
+ "Cannot emit memory checks when optimizing for size, unless forced "
+ "to vectorize.");
+ ORE->emit([&]() {
+ return OptimizationRemarkAnalysis(DEBUG_TYPE, "VectorizationCodeSize",
+ OrigLoop->getStartLoc(),
+ OrigLoop->getHeader())
+ << "Code-size may be reduced by not forcing "
+ "vectorization, or by source-code modifications "
+ "eliminating the need for runtime checks "
+ "(e.g., adding 'restrict').";
+ });
+ }
+ Checks.emplace_back(Plan.getOrAddLiveIn(MemCheckCond),
+ Plan.createVPIRBasicBlock(MemCheckBlock));
+ }
+ VPlanTransforms::connectCheckBlocks(
+ Plan, Checks,
+ hasBranchWeightMD(*OrigLoop->getLoopLatch()->getTerminator()));
+}
+
void VPDerivedIVRecipe::execute(VPTransformState &State) {
assert(!State.Lane && "VPDerivedIVRecipe being replicated.");
@@ -9446,10 +9397,7 @@ static bool processLoopInVPlanNativePath(
VPlan &BestPlan = LVP.getPlanFor(VF.Width);
{
- bool AddBranchWeights =
- hasBranchWeightMD(*L->getLoopLatch()->getTerminator());
- GeneratedRTChecks Checks(PSE, DT, LI, TTI, F->getDataLayout(),
- AddBranchWeights, CM.CostKind);
+ GeneratedRTChecks Checks(PSE, DT, LI, TTI, F->getDataLayout(), CM.CostKind);
InnerLoopVectorizer LB(L, PSE, LI, DT, TLI, TTI, AC, ORE, VF.Width,
VF.Width, 1, &CM, BFI, PSI, Checks, BestPlan);
LLVM_DEBUG(dbgs() << "Vectorizing outer loop in \""
@@ -10085,10 +10033,7 @@ bool LoopVectorizePass::processLoop(Loop *L) {
if (ORE->allowExtraAnalysis(LV_NAME))
LVP.emitInvalidCostRemarks(ORE);
- bool AddBranchWeights =
- hasBranchWeightMD(*L->getLoopLatch()->getTerminator());
- GeneratedRTChecks Checks(PSE, DT, LI, TTI, F->getDataLayout(),
- AddBranchWeights, CM.CostKind);
+ GeneratedRTChecks Checks(PSE, DT, LI, TTI, F->getDataLayout(), CM.CostKind);
if (LVP.hasPlanWithVF(VF.Width)) {
// Select the interleave count.
IC = CM.selectInterleaveCount(LVP.getPlanFor(VF.Width), VF.Width, VF.Cost);
diff --git a/llvm/lib/Transforms/Vectorize/VPlan.cpp b/llvm/lib/Transforms/Vectorize/VPlan.cpp
index 1838562f26b82..8068a6b3b968f 100644
--- a/llvm/lib/Transforms/Vectorize/VPlan.cpp
+++ b/llvm/lib/Transforms/Vectorize/VPlan.cpp
@@ -484,7 +484,8 @@ void VPBasicBlock::connectToPredecessors(VPTransformState &State) {
unsigned idx = PredVPSuccessors.front() == this ? 0 : 1;
assert((TermBr && (!TermBr->getSuccessor(idx) ||
(isa<VPIRBasicBlock>(this) &&
- TermBr->getSuccessor(idx) == NewBB))) &&
+ (TermBr->getSuccessor(idx) == NewBB ||
+ PredVPBlock == getPlan()->getEntry())))) &&
"Trying to reset an existing successor block.");
TermBr->setSuccessor(idx, NewBB);
}
diff --git a/llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp b/llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp
index 593e5063802ba..a55e95ef274b7 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp
+++ b/llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp
@@ -20,6 +20,7 @@
#include "llvm/Analysis/LoopInfo.h"
#include "llvm/Analysis/LoopIterator.h"
#include "llvm/Analysis/ScalarEvolution.h"
+#include "llvm/IR/MDBuilder.h"
#define DEBUG_TYPE "vplan"
@@ -589,3 +590,41 @@ void VPlanTransforms::createLoopRegions(VPlan &Plan) {
TopRegion->setName("vector loop");
TopRegion->getEntryBasicBlock()->setName("vector.body");
}
+
+// Likelyhood of bypassing the vectorized loop because SCEV assumptions or
+// memory runtime checks.
+static constexpr uint32_t CheckBypassWeights[] = {1, 127};
+
+void VPlanTransforms::connectCheckBlocks(
+ VPlan &Plan, ArrayRef<std::pair<VPValue *, VPIRBasicBlock *>> Checks,
+ bool AddBranchWeights) {
+ VPBlockBase *VectorPH = Plan.getVectorPreheader();
+ VPBlockBase *ScalarPH = Plan.getScalarPreheader();
+ for (const auto &[Cond, CheckBlock] : Checks) {
+ VPBlockBase *PreVectorPH = VectorPH->getSinglePredecessor();
+ VPBlockUtils::insertOnEdge(PreVectorPH, VectorPH, CheckBlock);
+ VPBlockUtils::connectBlocks(CheckBlock, ScalarPH);
+ CheckBlock->swapSuccessors();
+
+ // We just connected a new block to the scalar preheader. Update all
+ // VPPhis by adding an incoming value for it, replicating the last value.
+ unsigned NumPredecessors = ScalarPH->getNumPredecessors();
+ for (VPRecipeBase &R : cast<VPBasicBlock>(ScalarPH)->phis()) {
+ assert(isa<VPPhi>(&R) && "Phi expected to be VPPhi");
+ assert(cast<VPPhi>(&R)->getNumIncoming() == NumPredecessors - 1 &&
+ "must have incoming values for all operands");
+ R.addOperand(R.getOperand(NumPredecessors - 2));
+ }
+
+ VPIRMetadata VPBranchWeights;
+ auto *Term = VPBuilder(CheckBlock)
+ .createNaryOp(VPInstruction::BranchOnCond, {Cond},
+ Plan.getCanonicalIV()->getDebugLoc());
+ if (AddBranchWeights) {
+ MDBuilder MDB(Plan.getScalarHeader()->getIRBasicBlock()->getContext());
+ MDNode *BranchWeights =
+ MDB.createBranchWeights(CheckBypassWeights, /*IsExpected=*/false);
+ Term->addMetadata(LLVMContext::MD_prof, BranchWeights);
+ }
+ }
+}
diff --git a/llvm/lib/Transforms/Vectorize/VPlanTransforms.h b/llvm/lib/Transforms/Vectorize/VPlanTransforms.h
index 34e2de4eb3b74..c13cabc87ce31 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanTransforms.h
+++ b/llvm/lib/Transforms/Vectorize/VPlanTransforms.h
@@ -74,6 +74,13 @@ struct VPlanTransforms {
/// flat CFG into a hierarchical CFG.
static void createLoopRegions(VPlan &Plan);
+ /// Connect the blocks in \p Checks to \p Plan, using the corresponding
+ /// VPValue as branch condition.
+ static void
+ connectCheckBlocks(VPlan &Plan,
+ ArrayRef<std::pair<VPValue *, VPIRBasicBlock *>> ...
[truncated]
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Connect SCEV and memory runtime check block directly in VPlan as VPIRBasicBlocks, removing ILV::emitSCEVChecks and ILV::emitMemRuntimeChecks.
The new logic is currently split across LoopVectorizationPlanner::addRuntimeChecks which collects a list of {Condition, CheckBlock} pairs and performs some checks and emits remarks if needed. The list of checks is then added to VPlan in VPlanTransforms::connectCheckBlocks.