class TokenBuffer

Declaration

class TokenBuffer { /* full declaration omitted */ };

Description

A list of tokens obtained by preprocessing a text buffer and operations to map between the expanded and spelled tokens, i.e. TokenBuffer has information about two token streams: 1. Expanded tokens: tokens produced by the preprocessor after all macro replacements, 2. Spelled tokens: corresponding directly to the source code of a file before any macro replacements occurred. Here's an example to illustrate a difference between those two: #define FOO 10 int a = FOO; Spelled tokens are {'#','define','FOO','10','int','a','=','FOO',';'}. Expanded tokens are {'int','a','=','10',';','eof'}. Note that the expanded token stream has a tok::eof token at the end, the spelled tokens never store a 'eof' token. The full list expanded tokens can be obtained with expandedTokens(). Spelled tokens for each of the files can be obtained via spelledTokens(FileID). To map between the expanded and spelled tokens use findSpelledByExpanded(). To build a token buffer use the TokenCollector class. You can also compute the spelled tokens of a file using the tokenize() helper. FIXME: allow mappings into macro arguments.

Declared at: clang/include/clang/Tooling/Syntax/Tokens.h:175

Member Variables

private std::vector<syntax::Token> ExpandedTokens

Token stream produced after preprocessing, conceputally this captures the same stream as 'clang -E' (excluding the preprocessor directives like #file, etc.).

private llvm::DenseMap<SourceLocation, unsigned int> ExpandedTokIndex

private llvm::DenseMap<FileID, MarkedFile> Files

private const clang::SourceManager* SourceMgr

Method Overview

• public TokenBuffer(clang::syntax::TokenBuffer &&)
• public TokenBuffer(const clang::syntax::TokenBuffer &)
• public TokenBuffer(const clang::SourceManager & SourceMgr)
• public std::string dumpForTests() const
• public llvm::SmallVector<llvm::ArrayRef<syntax::Token>, 1> expandedForSpelled(llvm::ArrayRef<syntax::Token> Spelled) const
• public llvm::ArrayRef<syntax::Token> expandedTokens() const
• public llvm::ArrayRef<syntax::Token> expandedTokens(clang::SourceRange R) const
• public llvm::Optional<Expansion> expansionStartingAt(const syntax::Token * Spelled) const
• public std::vector<Expansion> expansionsOverlapping(llvm::ArrayRef<syntax::Token> Spelled) const
• private const clang::syntax::TokenBuffer::MarkedFile & fileForSpelled(llvm::ArrayRef<syntax::Token> Spelled) const
• public void indexExpandedTokens()
• public std::vector<const syntax::Token *> macroExpansions(clang::FileID FID) const
• private clang::syntax::TokenBuffer::Expansion makeExpansion(const clang::syntax::TokenBuffer::MarkedFile &, const clang::syntax::TokenBuffer::Mapping &) const
• private static const clang::syntax::TokenBuffer::Mapping * mappingStartingBeforeSpelled(const clang::syntax::TokenBuffer::MarkedFile & F, const syntax::Token * Spelled)
• public const clang::SourceManager & sourceManager() const
• public llvm::Optional<llvm::ArrayRef<syntax::Token>> spelledForExpanded(llvm::ArrayRef<syntax::Token> Expanded) const
• private std::pair<const syntax::Token *, const Mapping *> spelledForExpandedToken(const syntax::Token * Expanded) const
• public const syntax::Token * spelledTokenAt(clang::SourceLocation Loc) const
• public llvm::ArrayRef<syntax::Token> spelledTokens(clang::FileID FID) const

Methods

¶TokenBuffer(clang::syntax::TokenBuffer&&)

Declared at: clang/include/clang/Tooling/Syntax/Tokens.h:179

Parameters

clang::syntax::TokenBuffer&&

¶TokenBuffer(const clang::syntax::TokenBuffer&)

Declared at: clang/include/clang/Tooling/Syntax/Tokens.h:180

Parameters

const clang::syntax::TokenBuffer&

¶TokenBuffer(const clang::SourceManager& SourceMgr)

Declared at: clang/include/clang/Tooling/Syntax/Tokens.h:177

Parameters

const clang::SourceManager& SourceMgr

¶std::string dumpForTests() const

Declared at: clang/include/clang/Tooling/Syntax/Tokens.h:312

¶llvm::SmallVector<llvm::ArrayRef<syntax::Token>,
                  1>
expandedForSpelled(
    llvm::ArrayRef<syntax::Token> Spelled) const

Description

Find the subranges of expanded tokens, corresponding to \p Spelled. Some spelled tokens may not be present in the expanded token stream, so this function can return an empty vector, e.g. for tokens of macro directives or disabled preprocessor branches. Some spelled tokens can be duplicated in the expanded token stream multiple times and this function will return multiple results in those cases. This happens when \p Spelled is inside a macro argument. FIXME: return correct results on macro arguments. For now, we return an empty list. (!) will return empty vector on tokens from #define body: E.g. for the following example: #define FIRST(A) f1 A = A f2 #define SECOND s a FIRST(arg) b SECOND c // expanded tokens are: a f1 arg = arg f2 b s The results would be spelled => expanded ------------------------ #define FIRST => {} a FIRST(arg) => {a f1 arg = arg f2} arg => {arg, arg} // arg #1 is before `=` and arg #2 is // after `=` in the expanded tokens.

Declared at: clang/include/clang/Tooling/Syntax/Tokens.h:263

Parameters

llvm::ArrayRef<syntax::Token> Spelled

¶llvm::ArrayRef<syntax::Token> expandedTokens()
    const

Description

All tokens produced by the preprocessor after all macro replacements, directives, etc. Source locations found in the clang AST will always point to one of these tokens. Tokens are in TU order (per SourceManager::isBeforeInTranslationUnit()). FIXME: figure out how to handle token splitting, e.g. '>>' can be split into two '>' tokens by the parser. However, TokenBuffer currently keeps it as a single '>>' token.

Declared at: clang/include/clang/Tooling/Syntax/Tokens.h:191

¶llvm::ArrayRef<syntax::Token> expandedTokens(
    clang::SourceRange R) const

Description

Returns the subrange of expandedTokens() corresponding to the closed token range R. Consider calling indexExpandedTokens() before for faster lookups.

Declared at: clang/include/clang/Tooling/Syntax/Tokens.h:202

Parameters

clang::SourceRange R

¶llvm::Optional<Expansion> expansionStartingAt(
    const syntax::Token* Spelled) const

Description

If \p Spelled starts a mapping (e.g. if it's a macro name or '#' starting a preprocessor directive) return the subrange of expanded tokens that the macro expands to.

Declared at: clang/include/clang/Tooling/Syntax/Tokens.h:282

Parameters

const syntax::Token* Spelled

¶std::vector<Expansion> expansionsOverlapping(
    llvm::ArrayRef<syntax::Token> Spelled) const

Description

Returns all expansions (partially) expanded from the specified tokens. This is the expansions whose Spelled range intersects \p Spelled.

Declared at: clang/include/clang/Tooling/Syntax/Tokens.h:286

Parameters

llvm::ArrayRef<syntax::Token> Spelled

¶const clang::syntax::TokenBuffer::MarkedFile&
fileForSpelled(
    llvm::ArrayRef<syntax::Token> Spelled) const

Description

Returns the file that the Spelled tokens are taken from. Asserts that they are non-empty, from a tracked file, and in-bounds.

Declared at: clang/include/clang/Tooling/Syntax/Tokens.h:368

Parameters

llvm::ArrayRef<syntax::Token> Spelled

¶void indexExpandedTokens()

Description

Builds a cache to make future calls to expandedToken(SourceRange) faster. Creates an index only once. Further calls to it will be no-op.

Declared at: clang/include/clang/Tooling/Syntax/Tokens.h:197

¶std::vector<const syntax::Token*> macroExpansions(
    clang::FileID FID) const

Description

Get all tokens that expand a macro in \p FID. For the following input #define FOO B #define FOO2(X) int X FOO2(XY) int B; FOO; macroExpansions() returns {"FOO2", "FOO"} (from line 3 and 5 respecitvely).

Declared at: clang/include/clang/Tooling/Syntax/Tokens.h:308

Parameters

clang::FileID FID

¶clang::syntax::TokenBuffer::Expansion
makeExpansion(
    const clang::syntax::TokenBuffer::MarkedFile&,
    const clang::syntax::TokenBuffer::Mapping&)
    const

Description

Convert a private Mapping to a public Expansion.

Declared at: clang/include/clang/Tooling/Syntax/Tokens.h:365

Parameters

const clang::syntax::TokenBuffer::MarkedFile&

const clang::syntax::TokenBuffer::Mapping&

¶static const clang::syntax::TokenBuffer::Mapping*
mappingStartingBeforeSpelled(
    const clang::syntax::TokenBuffer::MarkedFile&
        F,
    const syntax::Token* Spelled)

Description

Returns a mapping starting before \p Spelled token, or nullptr if no such mapping exists.

Declared at: clang/include/clang/Tooling/Syntax/Tokens.h:361

Parameters

const clang::syntax::TokenBuffer::MarkedFile& F

const syntax::Token* Spelled

¶const clang::SourceManager& sourceManager() const

Declared at: clang/include/clang/Tooling/Syntax/Tokens.h:310

¶llvm::Optional<llvm::ArrayRef<syntax::Token>>
spelledForExpanded(
    llvm::ArrayRef<syntax::Token> Expanded) const

Description

Returns the subrange of spelled tokens corresponding to AST node spanning\p Expanded. This is the text that should be replaced if a refactoring were to rewrite the node. If \p Expanded is empty, the returned value is llvm::None. Will fail if the expanded tokens do not correspond to a sequence of spelled tokens. E.g. for the following example: #define FIRST f1 f2 f3 #define SECOND s1 s2 s3 #define ID2(X, Y) X Y a FIRST b SECOND c // expanded tokens are: a f1 f2 f3 b s1 s2 s3 c d ID2(e f g, h) i // expanded tokens are: d e f g h i the results would be: expanded => spelled ------------------------ a => a s1 s2 s3 => SECOND a f1 f2 f3 => a FIRST a f1 => can't map s1 s2 => can't map e f => e f g h => can't map EXPECTS: \p Expanded is a subrange of expandedTokens(). Complexity is logarithmic.

Declared at: clang/include/clang/Tooling/Syntax/Tokens.h:233

Parameters

llvm::ArrayRef<syntax::Token> Expanded

¶std::pair<const syntax::Token*, const Mapping*>
spelledForExpandedToken(
    const syntax::Token* Expanded) const

Description

Maps a single expanded token to its spelled counterpart or a mapping that produced it.

Declared at: clang/include/clang/Tooling/Syntax/Tokens.h:356

Parameters

const syntax::Token* Expanded

¶const syntax::Token* spelledTokenAt(
    clang::SourceLocation Loc) const

Description

Returns the spelled Token starting at Loc, if there are no such tokens returns nullptr.

Declared at: clang/include/clang/Tooling/Syntax/Tokens.h:298

Parameters

clang::SourceLocation Loc

¶llvm::ArrayRef<syntax::Token> spelledTokens(
    clang::FileID FID) const

Description

Lexed tokens of a file before preprocessing. E.g. for the following input #define DECL(name) int name = 10 DECL(a); spelledTokens() returns {"#", "define", "DECL", "(", "name", ")", "int", "name", "=", "10", "DECL", "(", "a", ")", ";"}

Declared at: clang/include/clang/Tooling/Syntax/Tokens.h:294

Parameters

clang::FileID FID