class TokenLexer

Declaration

class TokenLexer { /* full declaration omitted */ };

Description

TokenLexer - This implements a lexer that returns tokens from a macro body or token stream instead of lexing from a character buffer. This is used for macro expansion and _Pragma handling, for example.

Declared at: clang/include/clang/Lex/TokenLexer.h:30

Member Variables

private clang::MacroInfo* Macro = nullptr

The macro we are expanding from. This is null if expanding a token stream.

private clang::MacroArgs* ActualArgs = nullptr

The actual arguments specified for a function-like macro, or null. The TokenLexer owns the pointed-to object.

private clang::Preprocessor& PP

The current preprocessor object we are expanding for.

private const clang::Token* Tokens

This is the pointer to an array of tokens that the macro is defined to, with arguments expanded for function-like macros. If this is a token stream, these are the tokens we are returning. This points into the macro definition we are lexing from, a cache buffer that is owned by the preprocessor, or some other buffer that we may or may not own (depending on OwnsTokens). Note that if it points into Preprocessor's cache buffer, the Preprocessor may update the pointer as needed.

private unsigned int NumTokens

This is the length of the Tokens array.

private unsigned int CurTokenIdx

This is the index of the next token that Lex will return.

private clang::SourceLocation ExpandLocStart

The source location range where this macro was expanded.

private clang::SourceLocation ExpandLocEnd

The source location range where this macro was expanded.

private clang::SourceLocation MacroExpansionStart

Source location pointing at the source location entry chunk that was reserved for the current macro expansion.

private unsigned int MacroStartSLocOffset

The offset of the macro expansion in the "source location address space".

private clang::SourceLocation MacroDefStart

Location of the macro definition.

private unsigned int MacroDefLength

Length of the macro definition.

private bool AtStartOfLine

Lexical information about the expansion point of the macro: the identifier that the macro expanded from had these properties.

private bool HasLeadingSpace

private bool NextTokGetsSpace

private bool OwnsTokens

This is true if this TokenLexer allocated the Tokens array, and thus needs to free it when destroyed. For simple object-like macros (for example) we just point into the token buffer of the macro definition, we don't make a copy of it.

private bool DisableMacroExpansion

This is true when tokens lexed from the TokenLexer should not be subject to further macro expansion.

private bool IsReinject

When true, the produced tokens have Token::IsReinjected flag set. See the flag documentation for details.

Method Overview

• private void ExpandFunctionArguments()
• private void HandleMicrosoftCommentPaste(clang::Token & Tok, clang::SourceLocation OpLoc)
• public void Init(clang::Token & Tok, clang::SourceLocation ELEnd, clang::MacroInfo * MI, clang::MacroArgs * Actuals)
• public void Init(const clang::Token * TokArray, unsigned int NumToks, bool DisableMacroExpansion, bool OwnsTokens, bool IsReinject)
• public bool Lex(clang::Token & Tok)
• private bool MaybeRemoveCommaBeforeVaArgs(SmallVectorImpl<clang::Token> & ResultToks, bool HasPasteOperator, clang::MacroInfo * Macro, unsigned int MacroArgNo, clang::Preprocessor & PP)
• private void PropagateLineStartLeadingSpaceInfo(clang::Token & Result)
• public TokenLexer(const clang::Token * TokArray, unsigned int NumToks, bool DisableExpansion, bool ownsTokens, bool isReinject, clang::Preprocessor & pp)
• public TokenLexer(const clang::TokenLexer &)
• public TokenLexer(clang::Token & Tok, clang::SourceLocation ILEnd, clang::MacroInfo * MI, clang::MacroArgs * ActualArgs, clang::Preprocessor & pp)
• private void destroy()
• private clang::SourceLocation getExpansionLocForMacroDefLoc(clang::SourceLocation loc) const
• private bool isAtEnd() const
• public unsigned int isNextTokenLParen() const
• public bool isParsingPreprocessorDirective() const
• private bool pasteTokens(clang::Token & LHSTok, ArrayRef<clang::Token> TokenStream, unsigned int & CurIdx)
• private bool pasteTokens(clang::Token & Tok)
• private void stringifyVAOPTContents(SmallVectorImpl<clang::Token> & ResultToks, const clang::VAOptExpansionContext & VCtx, clang::SourceLocation VAOPTClosingParenLoc)
• private void updateLocForMacroArgTokens(clang::SourceLocation ArgIdSpellLoc, clang::Token * begin_tokens, clang::Token * end_tokens)
• public ~TokenLexer()

Methods

¶void ExpandFunctionArguments()

Description

Expand the arguments of a function-like macro so that we can quickly return preexpanded tokens from Tokens.

Declared at: clang/include/clang/Lex/TokenLexer.h:211

¶void HandleMicrosoftCommentPaste(
    clang::Token& Tok,
    clang::SourceLocation OpLoc)

Description

In microsoft compatibility mode, /##/ pastes together to form a comment that comments out everything in the current macro, other active macros, and anything left on the current physical source line of the expanded buffer. Handle this by returning the first token on the next line.

Declared at: clang/include/clang/Lex/TokenLexer.h:218

Parameters

clang::Token& Tok

clang::SourceLocation OpLoc

¶void Init(clang::Token& Tok,
          clang::SourceLocation ELEnd,
          clang::MacroInfo* MI,
          clang::MacroArgs* Actuals)

Description

Initialize this TokenLexer to expand from the specified macro with the specified argument information. Note that this ctor takes ownership of the ActualArgs pointer. ILEnd specifies the location of the ')' for a function-like macro or the identifier for an object-like macro.

Declared at: clang/include/clang/Lex/TokenLexer.h:131

Parameters

clang::Token& Tok

clang::SourceLocation ELEnd

clang::MacroInfo* MI

clang::MacroArgs* Actuals

¶void Init(const clang::Token* TokArray,
          unsigned int NumToks,
          bool DisableMacroExpansion,
          bool OwnsTokens,
          bool IsReinject)

Description

Initialize this TokenLexer with the specified token stream. This does not take ownership of the specified token vector. DisableExpansion is true when macro expansion of tokens lexed from this stream should be disabled.

Declared at: clang/include/clang/Lex/TokenLexer.h:139

Parameters

const clang::Token* TokArray

unsigned int NumToks

bool DisableMacroExpansion

bool OwnsTokens

bool IsReinject

¶bool Lex(clang::Token& Tok)

Description

Lex and return a token from this macro stream.

Declared at: clang/include/clang/Lex/TokenLexer.h:148

Parameters

clang::Token& Tok

¶bool MaybeRemoveCommaBeforeVaArgs(
    SmallVectorImpl<clang::Token>& ResultToks,
    bool HasPasteOperator,
    clang::MacroInfo* Macro,
    unsigned int MacroArgNo,
    clang::Preprocessor& PP)

Description

Remove comma ahead of __VA_ARGS__, if present, according to compiler dialect settings. Returns true if the comma is removed.

Declared at: clang/include/clang/Lex/TokenLexer.h:235

Parameters

SmallVectorImpl<clang::Token>& ResultToks

bool HasPasteOperator

clang::MacroInfo* Macro

unsigned int MacroArgNo

clang::Preprocessor& PP

¶void PropagateLineStartLeadingSpaceInfo(
    clang::Token& Result)

Declared at: clang/include/clang/Lex/TokenLexer.h:240

Parameters

clang::Token& Result

¶TokenLexer(const clang::Token* TokArray,
           unsigned int NumToks,
           bool DisableExpansion,
           bool ownsTokens,
           bool isReinject,
           clang::Preprocessor& pp)

Description

Create a TokenLexer for the specified token stream. If 'OwnsTokens' is specified, this takes ownership of the tokens and delete[]'s them when the token lexer is empty.

Declared at: clang/include/clang/Lex/TokenLexer.h:117

Parameters

const clang::Token* TokArray

unsigned int NumToks

bool DisableExpansion

bool ownsTokens

bool isReinject

clang::Preprocessor& pp

¶TokenLexer(const clang::TokenLexer&)

Declared at: clang/include/clang/Lex/TokenLexer.h:123

Parameters

const clang::TokenLexer&

¶TokenLexer(clang::Token& Tok,
           clang::SourceLocation ILEnd,
           clang::MacroInfo* MI,
           clang::MacroArgs* ActualArgs,
           clang::Preprocessor& pp)

Description

Create a TokenLexer for the specified macro with the specified actual arguments. Note that this ctor takes ownership of the ActualArgs pointer. ILEnd specifies the location of the ')' for a function-like macro or the identifier for an object-like macro.

Declared at: clang/include/clang/Lex/TokenLexer.h:108

Parameters

clang::Token& Tok

clang::SourceLocation ILEnd

clang::MacroInfo* MI

clang::MacroArgs* ActualArgs

clang::Preprocessor& pp

¶void destroy()

Declared at: clang/include/clang/Lex/TokenLexer.h:155

¶clang::SourceLocation
getExpansionLocForMacroDefLoc(
    clang::SourceLocation loc) const

Description

If \p loc is a FileID and points inside the current macro definition, returns the appropriate source location pointing at the macro expansion source location entry.

Declared at: clang/include/clang/Lex/TokenLexer.h:223

Parameters

clang::SourceLocation loc

¶bool isAtEnd() const

Description

Return true if the next lex call will pop this macro off the include stack.

Declared at: clang/include/clang/Lex/TokenLexer.h:159

¶unsigned int isNextTokenLParen() const

Description

If the next token lexed will pop this macro off the expansion stack, return 2. If the next unexpanded token is a '(', return 1, otherwise return 0.

Declared at: clang/include/clang/Lex/TokenLexer.h:145

¶bool isParsingPreprocessorDirective() const

Description

isParsingPreprocessorDirective - Return true if we are in the middle of a preprocessor directive.

Declared at: clang/include/clang/Lex/TokenLexer.h:152

¶bool pasteTokens(
    clang::Token& LHSTok,
    ArrayRef<clang::Token> TokenStream,
    unsigned int& CurIdx)

Description

Concatenates the next (sub-)sequence of \p Tokens separated by '##' starting with LHSTok - stopping when we encounter a token that is neither '##' nor preceded by '##'. Places the result back into \p LHSTok and sets\p CurIdx to point to the token following the last one that was pasted. Also performs the MSVC extension wide-literal token pasting involved with:

Declared at: clang/include/clang/Lex/TokenLexer.h:184

Parameters

clang::Token& LHSTok

- Contains the token to the left of '##' in \p Tokens upon entry and will contain the resulting concatenated Token upon exit.

ArrayRef<clang::Token> TokenStream

- The stream of Tokens we are lexing from.

unsigned int& CurIdx

- Upon entry, \pTokens[\pCurIdx] must equal '##' (with the exception of the MSVC extension mentioned above). Upon exit, it is set to the index of the token following the last token that was concatenated together.

Returns

If this returns true, the caller should immediately return the token.

¶bool pasteTokens(clang::Token& Tok)

Description

Calls pasteTokens above, passing in the '*this' object's Tokens and CurTokenIdx data members.

Declared at: clang/include/clang/Lex/TokenLexer.h:189

Parameters

clang::Token& Tok

¶void stringifyVAOPTContents(
    SmallVectorImpl<clang::Token>& ResultToks,
    const clang::VAOptExpansionContext& VCtx,
    clang::SourceLocation VAOPTClosingParenLoc)

Description

Takes the tail sequence of tokens within ReplacementToks that represent the just expanded __VA_OPT__ tokens (possibly zero tokens) and transforms them into a string. \p VCtx is used to determine which token represents the first __VA_OPT__ replacement token.

Declared at: clang/include/clang/Lex/TokenLexer.h:205

Parameters

SmallVectorImpl<clang::Token>& ResultToks

- Contains the current Replacement Tokens (prior to rescanning and token pasting), the tail end of which represents the tokens just expanded through __VA_OPT__ processing. These (sub) sequence of tokens are folded into one stringified token.

const clang::VAOptExpansionContext& VCtx

- contains relevant contextual information about the state of the tokens around and including the __VA_OPT__ token, necessary for stringification.

clang::SourceLocation VAOPTClosingParenLoc

¶void updateLocForMacroArgTokens(
    clang::SourceLocation ArgIdSpellLoc,
    clang::Token* begin_tokens,
    clang::Token* end_tokens)

Description

Creates SLocEntries and updates the locations of macro argument tokens to their new expanded locations.

Declared at: clang/include/clang/Lex/TokenLexer.h:230

Parameters

clang::SourceLocation ArgIdSpellLoc

the location of the macro argument id inside the macro definition.

clang::Token* begin_tokens

clang::Token* end_tokens

¶~TokenLexer()

Declared at: clang/include/clang/Lex/TokenLexer.h:125