class VPRecipeBuilder

Declaration

class VPRecipeBuilder { /* full declaration omitted */ };

Description

Helper class to create VPRecipies from IR instructions.

Declared at: llvm/lib/Transforms/Vectorize/VPRecipeBuilder.h:27

Member Variables

private llvm::Loop* OrigLoop
The loop that we evaluate.
private const llvm::TargetLibraryInfo* TLI
Target Library Info.
The legality analysis.
private llvm::LoopVectorizationCostModel& CM
The profitablity analysis.
private llvm::PredicatedScalarEvolution& PSE
private llvm::VPBuilder& Builder
private llvm::VPRecipeBuilder::EdgeMaskCacheTy EdgeMaskCache
private llvm::VPRecipeBuilder::BlockMaskCacheTy BlockMaskCache
private DenseMap<llvm::Instruction*, llvm::VPRecipeBase*> Ingredient2Recipe
private SmallVector<llvm::VPHeaderPHIRecipe*, 4> PhisToFix
Cross-iteration reduction & first-order recurrence phis for which we need to add the incoming value from the backedge after all recipes have been created.

Method Overview

  • public VPRecipeBuilder(llvm::Loop * OrigLoop, const llvm::TargetLibraryInfo * TLI, llvm::LoopVectorizationLegality * Legal, llvm::LoopVectorizationCostModel & CM, llvm::PredicatedScalarEvolution & PSE, llvm::VPBuilder & Builder)
  • public llvm::VPValue * createBlockInMask(llvm::BasicBlock * BB, llvm::VPlanPtr & Plan)
  • public llvm::VPValue * createEdgeMask(llvm::BasicBlock * Src, llvm::BasicBlock * Dst, llvm::VPlanPtr & Plan)
  • public llvm::VPRegionBlock * createReplicateRegion(llvm::VPReplicateRecipe * PredRecipe, llvm::VPlanPtr & Plan)
  • public void fixHeaderPhis()
  • public llvm::VPRecipeBase * getRecipe(llvm::Instruction * I)
  • public llvm::VPBasicBlock * handleReplication(llvm::Instruction * I, llvm::VFRange & Range, llvm::VPBasicBlock * VPBB, llvm::VPlanPtr & Plan)
  • public void recordRecipeOf(llvm::Instruction * I)
  • public void setRecipe(llvm::Instruction * I, llvm::VPRecipeBase * R)
  • private bool shouldWiden(llvm::Instruction * I, llvm::VFRange & Range) const
  • private llvm::VPRecipeOrVPValueTy toVPRecipeResult(llvm::VPRecipeBase * R) const
  • private llvm::VPRecipeOrVPValueTy tryToBlend(llvm::PHINode * Phi, ArrayRef<llvm::VPValue *> Operands, llvm::VPlanPtr & Plan)
  • public llvm::VPRecipeOrVPValueTy tryToCreateWidenRecipe(llvm::Instruction * Instr, ArrayRef<llvm::VPValue *> Operands, llvm::VFRange & Range, llvm::VPlanPtr & Plan)
  • private llvm::VPRecipeBase * tryToOptimizeInductionPHI(llvm::PHINode * Phi, ArrayRef<llvm::VPValue *> Operands, llvm::VPlan & Plan, llvm::VFRange & Range)
  • private llvm::VPWidenIntOrFpInductionRecipe * tryToOptimizeInductionTruncate(llvm::TruncInst * I, ArrayRef<llvm::VPValue *> Operands, llvm::VFRange & Range, llvm::VPlan & Plan)
  • private llvm::VPWidenRecipe * tryToWiden(llvm::Instruction * I, ArrayRef<llvm::VPValue *> Operands) const
  • private llvm::VPWidenCallRecipe * tryToWidenCall(llvm::CallInst * CI, ArrayRef<llvm::VPValue *> Operands, llvm::VFRange & Range) const
  • private llvm::VPRecipeBase * tryToWidenMemory(llvm::Instruction * I, ArrayRef<llvm::VPValue *> Operands, llvm::VFRange & Range, llvm::VPlanPtr & Plan)

Methods

VPRecipeBuilder(
    llvm::Loop* OrigLoop,
    const llvm::TargetLibraryInfo* TLI,
    llvm::LoopVectorizationLegality* Legal,
    llvm::LoopVectorizationCostModel& CM,
    llvm::PredicatedScalarEvolution& PSE,
    llvm::VPBuilder& Builder)

Declared at: llvm/lib/Transforms/Vectorize/VPRecipeBuilder.h:109

Parameters

llvm::Loop* OrigLoop
const llvm::TargetLibraryInfo* TLI
llvm::LoopVectorizationLegality* Legal
llvm::LoopVectorizationCostModel& CM
llvm::PredicatedScalarEvolution& PSE
llvm::VPBuilder& Builder

llvm::VPValue* createBlockInMask(
    llvm::BasicBlock* BB,
    llvm::VPlanPtr& Plan)

Description

A helper function that computes the predicate of the block BB, assuming that the header block of the loop is set to True. It returns the *entry* mask for the block BB.

Declared at: llvm/lib/Transforms/Vectorize/VPRecipeBuilder.h:137

Parameters

llvm::BasicBlock* BB
llvm::VPlanPtr& Plan

llvm::VPValue* createEdgeMask(
    llvm::BasicBlock* Src,
    llvm::BasicBlock* Dst,
    llvm::VPlanPtr& Plan)

Description

A helper function that computes the predicate of the edge between SRC and DST.

Declared at: llvm/lib/Transforms/Vectorize/VPRecipeBuilder.h:141

Parameters

llvm::BasicBlock* Src
llvm::BasicBlock* Dst
llvm::VPlanPtr& Plan

llvm::VPRegionBlock* createReplicateRegion(
    llvm::VPReplicateRecipe* PredRecipe,
    llvm::VPlanPtr& Plan)

Description

Create a replicating region for \p PredRecipe.

Declared at: llvm/lib/Transforms/Vectorize/VPRecipeBuilder.h:161

Parameters

llvm::VPReplicateRecipe* PredRecipe
llvm::VPlanPtr& Plan

void fixHeaderPhis()

Description

Add the incoming values from the backedge to reduction & first-order recurrence cross-iteration phis.

Declared at: llvm/lib/Transforms/Vectorize/VPRecipeBuilder.h:176

llvm::VPRecipeBase* getRecipe(
    llvm::Instruction* I)

Description

Return the recipe created for given ingredient.

Declared at: llvm/lib/Transforms/Vectorize/VPRecipeBuilder.h:152

Parameters

llvm::Instruction* I

llvm::VPBasicBlock* handleReplication(
    llvm::Instruction* I,
    llvm::VFRange& Range,
    llvm::VPBasicBlock* VPBB,
    llvm::VPlanPtr& Plan)

Description

Build a VPReplicationRecipe for \p I and enclose it within a Region if it is predicated.

Declared at: llvm/lib/Transforms/Vectorize/VPRecipeBuilder.h:170

Parameters

llvm::Instruction* I
llvm::VFRange& Range
llvm::VPBasicBlock* VPBB
llvm::VPlanPtr& Plan

Returns

a new VPBasicBlock that succeeds the new Region. Update the packing decision of predicated instructions if they feed \p I. Range.End may be decreased to ensure same recipe behavior from\p Range.Start to \p Range.End.

void recordRecipeOf(llvm::Instruction* I)

Description

Mark given ingredient for recording its recipe once one is created for it.

Declared at: llvm/lib/Transforms/Vectorize/VPRecipeBuilder.h:145

Parameters

llvm::Instruction* I

void setRecipe(llvm::Instruction* I,
               llvm::VPRecipeBase* R)

Description

Set the recipe created for given ingredient. This operation is a no-op for ingredients that were not marked using a nullptr entry in the map.

Declared at: llvm/lib/Transforms/Vectorize/VPRecipeBuilder.h:126

Parameters

llvm::Instruction* I
llvm::VPRecipeBase* R

bool shouldWiden(llvm::Instruction* I,
                 llvm::VFRange& Range) const

Description

Check if \p I can be widened at the start of \p Range and possibly decrease the range such that the returned value holds for the entire \p Range. The function should not be called for memory instructions or calls.

Declared at: llvm/lib/Transforms/Vectorize/VPRecipeBuilder.h:67

Parameters

llvm::Instruction* I
llvm::VFRange& Range

llvm::VPRecipeOrVPValueTy toVPRecipeResult(
    llvm::VPRecipeBase* R) const

Description

Return a VPRecipeOrValueTy with VPRecipeBase * being set. This can be used to force the use as VPRecipeBase* for recipe sub-types that also inherit from VPValue.

Declared at: llvm/lib/Transforms/Vectorize/VPRecipeBuilder.h:106

Parameters

llvm::VPRecipeBase* R

llvm::VPRecipeOrVPValueTy tryToBlend(
    llvm::PHINode* Phi,
    ArrayRef<llvm::VPValue*> Operands,
    llvm::VPlanPtr& Plan)

Description

Handle non-loop phi nodes. Return a VPValue, if all incoming values match or a new VPBlendRecipe otherwise. Currently all such phi nodes are turned into a sequence of select instructions as the vectorizer currently performs full if-conversion.

Declared at: llvm/lib/Transforms/Vectorize/VPRecipeBuilder.h:91

Parameters

llvm::PHINode* Phi
ArrayRef<llvm::VPValue*> Operands
llvm::VPlanPtr& Plan

llvm::VPRecipeOrVPValueTy tryToCreateWidenRecipe(
    llvm::Instruction* Instr,
    ArrayRef<llvm::VPValue*> Operands,
    llvm::VFRange& Range,
    llvm::VPlanPtr& Plan)

Description

Check if an existing VPValue can be used for \p Instr or a recipe can be create for \p I withing the given VF \p Range. If an existing VPValue can be used or if a recipe can be created, return it. Otherwise return a VPRecipeOrVPValueTy with nullptr.

Declared at: llvm/lib/Transforms/Vectorize/VPRecipeBuilder.h:120

Parameters

llvm::Instruction* Instr
ArrayRef<llvm::VPValue*> Operands
llvm::VFRange& Range
llvm::VPlanPtr& Plan

llvm::VPRecipeBase* tryToOptimizeInductionPHI(
    llvm::PHINode* Phi,
    ArrayRef<llvm::VPValue*> Operands,
    llvm::VPlan& Plan,
    llvm::VFRange& Range)

Description

Check if an induction recipe should be constructed for \p Phi. If so build and return it. If not, return null.

Declared at: llvm/lib/Transforms/Vectorize/VPRecipeBuilder.h:77

Parameters

llvm::PHINode* Phi
ArrayRef<llvm::VPValue*> Operands
llvm::VPlan& Plan
llvm::VFRange& Range

llvm::VPWidenIntOrFpInductionRecipe*
tryToOptimizeInductionTruncate(
    llvm::TruncInst* I,
    ArrayRef<llvm::VPValue*> Operands,
    llvm::VFRange& Range,
    llvm::VPlan& Plan)

Description

Optimize the special case where the operand of \p I is a constant integer induction variable.

Declared at: llvm/lib/Transforms/Vectorize/VPRecipeBuilder.h:84

Parameters

llvm::TruncInst* I
ArrayRef<llvm::VPValue*> Operands
llvm::VFRange& Range
llvm::VPlan& Plan

llvm::VPWidenRecipe* tryToWiden(
    llvm::Instruction* I,
    ArrayRef<llvm::VPValue*> Operands) const

Description

Check if \p I has an opcode that can be widened and return a VPWidenRecipe if it can. The function should only be called if the cost-model indicates that widening should be performed.

Declared at: llvm/lib/Transforms/Vectorize/VPRecipeBuilder.h:103

Parameters

llvm::Instruction* I
ArrayRef<llvm::VPValue*> Operands

llvm::VPWidenCallRecipe* tryToWidenCall(
    llvm::CallInst* CI,
    ArrayRef<llvm::VPValue*> Operands,
    llvm::VFRange& Range) const

Description

Handle call instructions. If \p CI can be widened for \p Range.Start, return a new VPWidenCallRecipe. Range.End may be decreased to ensure same decision from \p Range.Start to \p Range.End.

Declared at: llvm/lib/Transforms/Vectorize/VPRecipeBuilder.h:97

Parameters

llvm::CallInst* CI
ArrayRef<llvm::VPValue*> Operands
llvm::VFRange& Range

llvm::VPRecipeBase* tryToWidenMemory(
    llvm::Instruction* I,
    ArrayRef<llvm::VPValue*> Operands,
    llvm::VFRange& Range,
    llvm::VPlanPtr& Plan)

Description

Check if the load or store instruction \p I should widened for \p Range.Start and potentially masked. Such instructions are handled by a recipe that takes an additional VPInstruction for the mask.

Declared at: llvm/lib/Transforms/Vectorize/VPRecipeBuilder.h:72

Parameters

llvm::Instruction* I
ArrayRef<llvm::VPValue*> Operands
llvm::VFRange& Range
llvm::VPlanPtr& Plan