class InnerLoopAndEpilogueVectorizer

Declaration

class InnerLoopAndEpilogueVectorizer : public InnerLoopVectorizer { /* full declaration omitted */ };

Description

An extension of the inner loop vectorizer that creates a skeleton for a vectorized loop that has its epilogue (residual) also vectorized. The idea is to run the vplan on a given loop twice, firstly to setup the skeleton and vectorize the main loop, and secondly to complete the skeleton from the first step and vectorize the epilogue. This is achieved by deriving two concrete strategy classes from this base class and invoking them in succession from the loop vectorizer planner.

Declared at: llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:784

Inherits from: InnerLoopVectorizer

Member Variables

public llvm::EpilogueLoopVectorizationInfo& EPI

Holds and updates state information required to vectorize the main loop and its epilogue in two separate passes. This setup helps us avoid regenerating and recomputing runtime safety checks. It also helps us to shorten the iteration-count-check path length for the cases where the iteration count of the loop is so small that the main vector loop is completely skipped.

Inherited from InnerLoopVectorizer:

protected OrigLoop

protected PSE

protected LI

protected DT

protected AA

protected TLI

protected TTI

protected AC

protected ORE

protected VF

protected MinProfitableTripCount

protected UF

protected Builder

protected LoopVectorPreHeader

protected LoopScalarPreHeader

protected LoopMiddleBlock

protected LoopExitBlock

protected LoopScalarBody

protected LoopBypassBlocks

protected PredicatedInstructions

protected TripCount = nullptr

protected VectorTripCount = nullptr

protected Legal

protected Cost

protected AddedSafetyChecks = false

protected IVEndValues

protected BFI

protected PSI

protected OptForSizeBasedOnProfile

protected RTChecks

protected ReductionResumeValues

Method Overview

• public InnerLoopAndEpilogueVectorizer(llvm::Loop * OrigLoop, llvm::PredicatedScalarEvolution & PSE, llvm::LoopInfo * LI, llvm::DominatorTree * DT, const llvm::TargetLibraryInfo * TLI, const llvm::TargetTransformInfo * TTI, llvm::AssumptionCache * AC, llvm::OptimizationRemarkEmitter * ORE, llvm::EpilogueLoopVectorizationInfo & EPI, llvm::LoopVectorizationLegality * LVL, llvm::LoopVectorizationCostModel * CM, llvm::BlockFrequencyInfo * BFI, llvm::ProfileSummaryInfo * PSI, GeneratedRTChecks & Checks)
• public virtual std::pair<BasicBlock *, Value *> createEpilogueVectorizedLoopSkeleton()
• public std::pair<BasicBlock *, Value *> createVectorizedLoopSkeleton()

Inherited from InnerLoopVectorizer:

• public areSafetyChecksAdded
• protected clearReductionWrapFlags
• protected collectPoisonGeneratingRecipes
• protected completeLoopSkeleton
• protected createBitOrPointerCast
• protected createInductionResumeValues
• protected createVectorLoopSkeleton
• public createVectorizedLoopSkeleton
• protected emitIterationCountCheck
• protected emitMemRuntimeChecks
• protected emitSCEVChecks
• protected fixCrossIterationPHIs
• protected fixFirstOrderRecurrence
• public fixNonInductionPHIs
• protected fixReduction
• public fixVectorizedLoop
• protected fixupIVUsers
• public getBroadcastInstrs
• protected getOrCreateTripCount
• protected getOrCreateVectorTripCount
• public getReductionResumeValue
• public packScalarIntoVectorValue
• protected printDebugTracesAtEnd
• protected printDebugTracesAtStart
• public scalarizeInstruction
• protected sinkScalarOperands
• protected truncateToMinimalBitwidths
• public useOrderedReductions
• public vectorizeInterleaveGroup
• public widenCallInstruction

Methods

¶InnerLoopAndEpilogueVectorizer(
    llvm::Loop* OrigLoop,
    llvm::PredicatedScalarEvolution& PSE,
    llvm::LoopInfo* LI,
    llvm::DominatorTree* DT,
    const llvm::TargetLibraryInfo* TLI,
    const llvm::TargetTransformInfo* TTI,
    llvm::AssumptionCache* AC,
    llvm::OptimizationRemarkEmitter* ORE,
    llvm::EpilogueLoopVectorizationInfo& EPI,
    llvm::LoopVectorizationLegality* LVL,
    llvm::LoopVectorizationCostModel* CM,
    llvm::BlockFrequencyInfo* BFI,
    llvm::ProfileSummaryInfo* PSI,
    GeneratedRTChecks& Checks)

Declared at: llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:786

Parameters

llvm::Loop* OrigLoop

llvm::PredicatedScalarEvolution& PSE

llvm::LoopInfo* LI

llvm::DominatorTree* DT

const llvm::TargetLibraryInfo* TLI

const llvm::TargetTransformInfo* TTI

llvm::AssumptionCache* AC

llvm::OptimizationRemarkEmitter* ORE

llvm::EpilogueLoopVectorizationInfo& EPI

llvm::LoopVectorizationLegality* LVL

llvm::LoopVectorizationCostModel* CM

llvm::BlockFrequencyInfo* BFI

llvm::ProfileSummaryInfo* PSI

GeneratedRTChecks& Checks

¶virtual std::pair<BasicBlock*, Value*>
createEpilogueVectorizedLoopSkeleton()

Description

The interface for creating a vectorized skeleton using one of two different strategies, each corresponding to one execution of the vplan as described above.

Declared at: llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:809

¶std::pair<BasicBlock*, Value*>
createVectorizedLoopSkeleton()

Description

Create a new empty loop that will contain vectorized instructions later on, while the old loop will be used as the scalar remainder. Control flow is generated around the vectorized (and scalar epilogue) loops consisting of various checks and bypasses. Return the pre-header block of the new loop and the start value for the canonical induction, if it is != 0. The latter is the case when vectorizing the epilogue loop. In the case of epilogue vectorization, this function is overriden to handle the more complex control flow around the loops.

Declared at: llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:801