class Token

Declaration

class Token { /* full declaration omitted */ };

Description

Token - This structure provides full information about a lexed token. It is not intended to be space efficient, it is intended to return as much information as possible about each returned token. This is expected to be compressed into a smaller form if memory footprint is important. The parser can create a special "annotation token" representing a stream of tokens that were parsed and semantically resolved, e.g.: "foo::MyClass<int>" can be represented by a single typename annotation token that carries information about the SourceRange of the tokens and the type object.

Declared at: clang/include/clang/Lex/Token.h:34

Member Variables

private SourceLocation::UIntTy Loc
The location of the token. This is actually a SourceLocation.
private SourceLocation::UIntTy UintData
UintData - This holds either the length of the token text, when a normal token, or the end of the SourceRange when an annotation token.
private void* PtrData
Other: This is null.
private tok::TokenKind Kind
Kind - The actual flavor of token this is.
private unsigned short Flags
Flags - Bits we track about this token, members of the TokenFlags enum.

Method Overview

Methods

void clearFlag(clang::Token::TokenFlags Flag)

Description

Unset the specified flag.

Declared at: clang/include/clang/Lex/Token.h:245

Parameters

clang::Token::TokenFlags Flag

bool commaAfterElided() const

Description

Returns true if the comma after this token was elided.

Declared at: clang/include/clang/Lex/Token.h:302

clang::SourceLocation getAnnotationEndLoc() const

Declared at: clang/include/clang/Lex/Token.h:139

clang::SourceRange getAnnotationRange() const

Description

SourceRange of the group of tokens that this annotation token represents.

Declared at: clang/include/clang/Lex/Token.h:159

void* getAnnotationValue() const

Declared at: clang/include/clang/Lex/Token.h:225

clang::SourceLocation getEndLoc() const

Declared at: clang/include/clang/Lex/Token.h:152

const void* getEofData() const

Declared at: clang/include/clang/Lex/Token.h:191

bool getFlag(clang::Token::TokenFlags Flag) const

Description

Get the specified flag.

Declared at: clang/include/clang/Lex/Token.h:240

Parameters

clang::Token::TokenFlags Flag

unsigned int getFlags() const

Description

Return the internal represtation of the flags. This is only intended for low-level operations such as writing tokens to disk.

Declared at: clang/include/clang/Lex/Token.h:253

clang::IdentifierInfo* getIdentifierInfo() const

Declared at: clang/include/clang/Lex/Token.h:178

tok::TokenKind getKind() const

Declared at: clang/include/clang/Lex/Token.h:92

clang::SourceLocation getLastLoc() const

Declared at: clang/include/clang/Lex/Token.h:148

unsigned int getLength() const

Declared at: clang/include/clang/Lex/Token.h:128

const char* getLiteralData() const

Description

getLiteralData - For a literal token (numeric constant, string, etc), this returns a pointer to the start of it in the text buffer if known, null otherwise.

Declared at: clang/include/clang/Lex/Token.h:216

clang::SourceLocation getLocation() const

Description

Return a source location identifier for the specified offset in the current file.

Declared at: clang/include/clang/Lex/Token.h:125

const char* getName() const

Declared at: clang/include/clang/Lex/Token.h:167

tok::ObjCKeywordKind getObjCKeywordID() const

Description

Return the ObjC keyword kind.

Declared at: clang/include/clang/Lex/Token.h:281

llvm::StringRef getRawIdentifier() const

Description

getRawIdentifier - For a raw identifier token (i.e., an identifier lexed in raw mode), returns a reference to the text substring in the buffer if known.

Declared at: clang/include/clang/Lex/Token.h:204

bool hasLeadingEmptyMacro() const

Description

Return true if this token has an empty macro before it.

Declared at: clang/include/clang/Lex/Token.h:288

bool hasLeadingSpace() const

Description

Return true if this token has whitespace before it.

Declared at: clang/include/clang/Lex/Token.h:271

bool hasUCN() const

Description

Returns true if this token contains a universal character name.

Declared at: clang/include/clang/Lex/Token.h:295

bool hasUDSuffix() const

Description

Return true if this token is a string or character literal which has a ud-suffix.

Declared at: clang/include/clang/Lex/Token.h:292

bool is(tok::TokenKind K) const

Description

is/isNot - Predicates to check if this token is a specific kind, as in "if (Tok.is(tok::l_brace)) {...}".

Declared at: clang/include/clang/Lex/Token.h:97

Parameters

tok::TokenKind K

bool isAnnotation() const

Description

Return true if this is any of tok::annot_* kind tokens.

Declared at: clang/include/clang/Lex/Token.h:119

bool isAnyIdentifier() const

Description

Return true if this is a raw identifier (when lexing in raw mode) or a non-keyword identifier (when lexing in non-raw mode).

Declared at: clang/include/clang/Lex/Token.h:108

bool isAtStartOfLine() const

Description

isAtStartOfLine - Return true if this token is at the start of a line.

Declared at: clang/include/clang/Lex/Token.h:267

bool isEditorPlaceholder() const

Description

Returns true if this token is an editor placeholder. Editor placeholders are produced by the code-completion engine and are represented as characters between ' < #' and '#>' in the source code. The lexer uses identifier tokens to represent placeholders.

Declared at: clang/include/clang/Lex/Token.h:309

bool isExpandDisabled() const

Description

Return true if this identifier token should never be expanded in the future, due to C99 6.10.3.4p2.

Declared at: clang/include/clang/Lex/Token.h:275

bool isLiteral() const

Description

Return true if this is a "literal", like a numeric constant, string, etc.

Declared at: clang/include/clang/Lex/Token.h:114

bool isNot(tok::TokenKind K) const

Declared at: clang/include/clang/Lex/Token.h:98

Parameters

tok::TokenKind K

bool isObjCAtKeyword(
    tok::ObjCKeywordKind objcKey) const

Description

Return true if we have an ObjC keyword identifier.

Declared at: clang/include/clang/Lex/Token.h:278

Parameters

tok::ObjCKeywordKind objcKey

template <typename... Ts>
bool isOneOf(tok::TokenKind K1, Ts... Ks) const

Declared at: clang/include/clang/Lex/Token.h:102

Templates

Ts

Parameters

tok::TokenKind K1
Ts... Ks

bool isOneOf(tok::TokenKind K1,
             tok::TokenKind K2) const

Declared at: clang/include/clang/Lex/Token.h:99

Parameters

tok::TokenKind K1
tok::TokenKind K2

bool needsCleaning() const

Description

Return true if this token has trigraphs or escaped newlines in it.

Declared at: clang/include/clang/Lex/Token.h:284

void setAnnotationEndLoc(clang::SourceLocation L)

Declared at: clang/include/clang/Lex/Token.h:143

Parameters

clang::SourceLocation L

void setAnnotationRange(clang::SourceRange R)

Declared at: clang/include/clang/Lex/Token.h:162

Parameters

clang::SourceRange R

void setAnnotationValue(void* val)

Declared at: clang/include/clang/Lex/Token.h:229

Parameters

void* val

void setEofData(const void* D)

Declared at: clang/include/clang/Lex/Token.h:195

Parameters

const void* D

void setFlag(clang::Token::TokenFlags Flag)

Description

Set the specified flag.

Declared at: clang/include/clang/Lex/Token.h:235

Parameters

clang::Token::TokenFlags Flag

void setFlagValue(clang::Token::TokenFlags Flag,
                  bool Val)

Description

Set a flag to either true or false.

Declared at: clang/include/clang/Lex/Token.h:258

Parameters

clang::Token::TokenFlags Flag
bool Val

void setIdentifierInfo(clang::IdentifierInfo* II)

Declared at: clang/include/clang/Lex/Token.h:187

Parameters

clang::IdentifierInfo* II

void setKind(tok::TokenKind K)

Declared at: clang/include/clang/Lex/Token.h:93

Parameters

tok::TokenKind K

void setLength(unsigned int Len)

Declared at: clang/include/clang/Lex/Token.h:134

Parameters

unsigned int Len

void setLiteralData(const char* Ptr)

Declared at: clang/include/clang/Lex/Token.h:220

Parameters

const char* Ptr

void setLocation(clang::SourceLocation L)

Declared at: clang/include/clang/Lex/Token.h:133

Parameters

clang::SourceLocation L

void setRawIdentifierData(const char* Ptr)

Declared at: clang/include/clang/Lex/Token.h:208

Parameters

const char* Ptr

void startToken()

Description

Reset all flags to cleared.

Declared at: clang/include/clang/Lex/Token.h:170

bool stringifiedInMacro() const

Description

Returns true if this token is formed by macro by stringizing or charizing operator.

Declared at: clang/include/clang/Lex/Token.h:299