class TextTokenRetokenizer

Declaration

class TextTokenRetokenizer { /* full declaration omitted */ };

Description

Re-lexes a sequence of tok::text tokens.

Declared at: clang/lib/AST/CommentParser.cpp:30

Member Variables

private llvm::BumpPtrAllocator& Allocator

private clang::comments::Parser& P

private bool NoMoreInterestingTokens

This flag is set when there are no more tokens we can fetch from lexer.

private SmallVector<clang::comments::Token, 16> Toks

Token buffer: tokens we have processed and lookahead.

private clang::comments::TextTokenRetokenizer::Position Pos

Current position in Toks.

Method Overview

• public TextTokenRetokenizer(llvm::BumpPtrAllocator & Allocator, clang::comments::Parser & P)
• private bool addToken()
• private void consumeChar()
• private void consumeWhitespace()
• private void formTokenWithChars(clang::comments::Token & Result, clang::SourceLocation Loc, const char * TokBegin, unsigned int TokLength, llvm::StringRef Text)
• private clang::SourceLocation getSourceLocation() const
• private bool isEnd() const
• public bool lexDelimitedSeq(clang::comments::Token & Tok, char OpenDelim, char CloseDelim)
• public bool lexWord(clang::comments::Token & Tok)
• private char peek() const
• public void putBackLeftoverTokens()
• private void setupBuffer()

Methods

¶TextTokenRetokenizer(
    llvm::BumpPtrAllocator& Allocator,
    clang::comments::Parser& P)

Declared at: clang/lib/AST/CommentParser.cpp:146

Parameters

llvm::BumpPtrAllocator& Allocator

clang::comments::Parser& P

¶bool addToken()

Description

Add a token. Returns true on success, false if there are no interesting tokens to fetch from lexer.

Declared at: clang/lib/AST/CommentParser.cpp:95

¶void consumeChar()

Declared at: clang/lib/AST/CommentParser.cpp:78

¶void consumeWhitespace()

Declared at: clang/lib/AST/CommentParser.cpp:121

¶void formTokenWithChars(
    clang::comments::Token& Result,
    clang::SourceLocation Loc,
    const char* TokBegin,
    unsigned int TokLength,
    llvm::StringRef Text)

Declared at: clang/lib/AST/CommentParser.cpp:130

Parameters

clang::comments::Token& Result

clang::SourceLocation Loc

const char* TokBegin

unsigned int TokLength

llvm::StringRef Text

¶clang::SourceLocation getSourceLocation() const

Declared at: clang/lib/AST/CommentParser.cpp:67

¶bool isEnd() const

Declared at: clang/lib/AST/CommentParser.cpp:52

¶bool lexDelimitedSeq(clang::comments::Token& Tok,
                     char OpenDelim,
                     char CloseDelim)

Declared at: clang/lib/AST/CommentParser.cpp:186

Parameters

clang::comments::Token& Tok

char OpenDelim

char CloseDelim

¶bool lexWord(clang::comments::Token& Tok)

Description

Extract a word -- sequence of non-whitespace characters.

Declared at: clang/lib/AST/CommentParser.cpp:153

Parameters

clang::comments::Token& Tok

¶char peek() const

Declared at: clang/lib/AST/CommentParser.cpp:72

¶void putBackLeftoverTokens()

Description

Put back tokens that we didn't consume.

Declared at: clang/lib/AST/CommentParser.cpp:233

¶void setupBuffer()

Description

Sets up the buffer pointers to point to current token.

Declared at: clang/lib/AST/CommentParser.cpp:57