class Token
Declaration
class Token { /* full declaration omitted */ };
Description
Token - This structure provides full information about a lexed token. It is not intended to be space efficient, it is intended to return as much information as possible about each returned token. This is expected to be compressed into a smaller form if memory footprint is important. The parser can create a special "annotation token" representing a stream of tokens that were parsed and semantically resolved, e.g.: "foo::MyClass<int>" can be represented by a single typename annotation token that carries information about the SourceRange of the tokens and the type object.
Declared at: clang/include/clang/Lex/Token.h:34
Member Variables
- private SourceLocation::UIntTy Loc
- The location of the token. This is actually a SourceLocation.
- private SourceLocation::UIntTy UintData
- UintData - This holds either the length of the token text, when a normal token, or the end of the SourceRange when an annotation token.
- private void* PtrData
- Other: This is null.
- private tok::TokenKind Kind
- Kind - The actual flavor of token this is.
- private unsigned short Flags
- Flags - Bits we track about this token, members of the TokenFlags enum.
Method Overview
- public void clearFlag(clang::Token::TokenFlags Flag)
- public bool commaAfterElided() const
- public clang::SourceLocation getAnnotationEndLoc() const
- public clang::SourceRange getAnnotationRange() const
- public void * getAnnotationValue() const
- public clang::SourceLocation getEndLoc() const
- public const void * getEofData() const
- public bool getFlag(clang::Token::TokenFlags Flag) const
- public unsigned int getFlags() const
- public clang::IdentifierInfo * getIdentifierInfo() const
- public tok::TokenKind getKind() const
- public clang::SourceLocation getLastLoc() const
- public unsigned int getLength() const
- public const char * getLiteralData() const
- public clang::SourceLocation getLocation() const
- public const char * getName() const
- public tok::ObjCKeywordKind getObjCKeywordID() const
- public llvm::StringRef getRawIdentifier() const
- public bool hasLeadingEmptyMacro() const
- public bool hasLeadingSpace() const
- public bool hasUCN() const
- public bool hasUDSuffix() const
- public bool is(tok::TokenKind K) const
- public bool isAnnotation() const
- public bool isAnyIdentifier() const
- public bool isAtStartOfLine() const
- public bool isEditorPlaceholder() const
- public bool isExpandDisabled() const
- public bool isLiteral() const
- public bool isNot(tok::TokenKind K) const
- public bool isObjCAtKeyword(tok::ObjCKeywordKind objcKey) const
- public template <typename... Ts>bool isOneOf(tok::TokenKind K1, Ts... Ks) const
- public bool isOneOf(tok::TokenKind K1, tok::TokenKind K2) const
- public bool needsCleaning() const
- public void setAnnotationEndLoc(clang::SourceLocation L)
- public void setAnnotationRange(clang::SourceRange R)
- public void setAnnotationValue(void * val)
- public void setEofData(const void * D)
- public void setFlag(clang::Token::TokenFlags Flag)
- public void setFlagValue(clang::Token::TokenFlags Flag, bool Val)
- public void setIdentifierInfo(clang::IdentifierInfo * II)
- public void setKind(tok::TokenKind K)
- public void setLength(unsigned int Len)
- public void setLiteralData(const char * Ptr)
- public void setLocation(clang::SourceLocation L)
- public void setRawIdentifierData(const char * Ptr)
- public void startToken()
- public bool stringifiedInMacro() const
Methods
¶void clearFlag(clang::Token::TokenFlags Flag)
void clearFlag(clang::Token::TokenFlags Flag)
Description
Unset the specified flag.
Declared at: clang/include/clang/Lex/Token.h:245
Parameters
- clang::Token::TokenFlags Flag
¶bool commaAfterElided() const
bool commaAfterElided() const
Description
Returns true if the comma after this token was elided.
Declared at: clang/include/clang/Lex/Token.h:302
¶clang::SourceLocation getAnnotationEndLoc() const
clang::SourceLocation getAnnotationEndLoc() const
Declared at: clang/include/clang/Lex/Token.h:139
¶clang::SourceRange getAnnotationRange() const
clang::SourceRange getAnnotationRange() const
Description
SourceRange of the group of tokens that this annotation token represents.
Declared at: clang/include/clang/Lex/Token.h:159
¶void* getAnnotationValue() const
void* getAnnotationValue() const
Declared at: clang/include/clang/Lex/Token.h:225
¶clang::SourceLocation getEndLoc() const
clang::SourceLocation getEndLoc() const
Declared at: clang/include/clang/Lex/Token.h:152
¶const void* getEofData() const
const void* getEofData() const
Declared at: clang/include/clang/Lex/Token.h:191
¶bool getFlag(clang::Token::TokenFlags Flag) const
bool getFlag(clang::Token::TokenFlags Flag) const
Description
Get the specified flag.
Declared at: clang/include/clang/Lex/Token.h:240
Parameters
- clang::Token::TokenFlags Flag
¶unsigned int getFlags() const
unsigned int getFlags() const
Description
Return the internal represtation of the flags. This is only intended for low-level operations such as writing tokens to disk.
Declared at: clang/include/clang/Lex/Token.h:253
¶clang::IdentifierInfo* getIdentifierInfo() const
clang::IdentifierInfo* getIdentifierInfo() const
Declared at: clang/include/clang/Lex/Token.h:178
¶tok::TokenKind getKind() const
tok::TokenKind getKind() const
Declared at: clang/include/clang/Lex/Token.h:92
¶clang::SourceLocation getLastLoc() const
clang::SourceLocation getLastLoc() const
Declared at: clang/include/clang/Lex/Token.h:148
¶unsigned int getLength() const
unsigned int getLength() const
Declared at: clang/include/clang/Lex/Token.h:128
¶const char* getLiteralData() const
const char* getLiteralData() const
Description
getLiteralData - For a literal token (numeric constant, string, etc), this returns a pointer to the start of it in the text buffer if known, null otherwise.
Declared at: clang/include/clang/Lex/Token.h:216
¶clang::SourceLocation getLocation() const
clang::SourceLocation getLocation() const
Description
Return a source location identifier for the specified offset in the current file.
Declared at: clang/include/clang/Lex/Token.h:125
¶const char* getName() const
const char* getName() const
Declared at: clang/include/clang/Lex/Token.h:167
¶tok::ObjCKeywordKind getObjCKeywordID() const
tok::ObjCKeywordKind getObjCKeywordID() const
Description
Return the ObjC keyword kind.
Declared at: clang/include/clang/Lex/Token.h:281
¶llvm::StringRef getRawIdentifier() const
llvm::StringRef getRawIdentifier() const
Description
getRawIdentifier - For a raw identifier token (i.e., an identifier lexed in raw mode), returns a reference to the text substring in the buffer if known.
Declared at: clang/include/clang/Lex/Token.h:204
¶bool hasLeadingEmptyMacro() const
bool hasLeadingEmptyMacro() const
Description
Return true if this token has an empty macro before it.
Declared at: clang/include/clang/Lex/Token.h:288
¶bool hasLeadingSpace() const
bool hasLeadingSpace() const
Description
Return true if this token has whitespace before it.
Declared at: clang/include/clang/Lex/Token.h:271
¶bool hasUCN() const
bool hasUCN() const
Description
Returns true if this token contains a universal character name.
Declared at: clang/include/clang/Lex/Token.h:295
¶bool hasUDSuffix() const
bool hasUDSuffix() const
Description
Return true if this token is a string or character literal which has a ud-suffix.
Declared at: clang/include/clang/Lex/Token.h:292
¶bool is(tok::TokenKind K) const
bool is(tok::TokenKind K) const
Description
is/isNot - Predicates to check if this token is a specific kind, as in "if (Tok.is(tok::l_brace)) {...}".
Declared at: clang/include/clang/Lex/Token.h:97
Parameters
- tok::TokenKind K
¶bool isAnnotation() const
bool isAnnotation() const
Description
Return true if this is any of tok::annot_* kind tokens.
Declared at: clang/include/clang/Lex/Token.h:119
¶bool isAnyIdentifier() const
bool isAnyIdentifier() const
Description
Return true if this is a raw identifier (when lexing in raw mode) or a non-keyword identifier (when lexing in non-raw mode).
Declared at: clang/include/clang/Lex/Token.h:108
¶bool isAtStartOfLine() const
bool isAtStartOfLine() const
Description
isAtStartOfLine - Return true if this token is at the start of a line.
Declared at: clang/include/clang/Lex/Token.h:267
¶bool isEditorPlaceholder() const
bool isEditorPlaceholder() const
Description
Returns true if this token is an editor placeholder. Editor placeholders are produced by the code-completion engine and are represented as characters between ' < #' and '#>' in the source code. The lexer uses identifier tokens to represent placeholders.
Declared at: clang/include/clang/Lex/Token.h:309
¶bool isExpandDisabled() const
bool isExpandDisabled() const
Description
Return true if this identifier token should never be expanded in the future, due to C99 6.10.3.4p2.
Declared at: clang/include/clang/Lex/Token.h:275
¶bool isLiteral() const
bool isLiteral() const
Description
Return true if this is a "literal", like a numeric constant, string, etc.
Declared at: clang/include/clang/Lex/Token.h:114
¶bool isNot(tok::TokenKind K) const
bool isNot(tok::TokenKind K) const
Declared at: clang/include/clang/Lex/Token.h:98
Parameters
- tok::TokenKind K
¶bool isObjCAtKeyword(
tok::ObjCKeywordKind objcKey) const
bool isObjCAtKeyword(
tok::ObjCKeywordKind objcKey) const
Description
Return true if we have an ObjC keyword identifier.
Declared at: clang/include/clang/Lex/Token.h:278
Parameters
- tok::ObjCKeywordKind objcKey
¶template <typename... Ts>
bool isOneOf(tok::TokenKind K1, Ts... Ks) const
template <typename... Ts>
bool isOneOf(tok::TokenKind K1, Ts... Ks) const
Declared at: clang/include/clang/Lex/Token.h:102
Templates
- Ts
Parameters
- tok::TokenKind K1
- Ts... Ks
¶bool isOneOf(tok::TokenKind K1,
tok::TokenKind K2) const
bool isOneOf(tok::TokenKind K1,
tok::TokenKind K2) const
Declared at: clang/include/clang/Lex/Token.h:99
Parameters
- tok::TokenKind K1
- tok::TokenKind K2
¶bool needsCleaning() const
bool needsCleaning() const
Description
Return true if this token has trigraphs or escaped newlines in it.
Declared at: clang/include/clang/Lex/Token.h:284
¶void setAnnotationEndLoc(clang::SourceLocation L)
void setAnnotationEndLoc(clang::SourceLocation L)
Declared at: clang/include/clang/Lex/Token.h:143
Parameters
¶void setAnnotationRange(clang::SourceRange R)
void setAnnotationRange(clang::SourceRange R)
Declared at: clang/include/clang/Lex/Token.h:162
Parameters
¶void setAnnotationValue(void* val)
void setAnnotationValue(void* val)
Declared at: clang/include/clang/Lex/Token.h:229
Parameters
- void* val
¶void setEofData(const void* D)
void setEofData(const void* D)
Declared at: clang/include/clang/Lex/Token.h:195
Parameters
- const void* D
¶void setFlag(clang::Token::TokenFlags Flag)
void setFlag(clang::Token::TokenFlags Flag)
Description
Set the specified flag.
Declared at: clang/include/clang/Lex/Token.h:235
Parameters
- clang::Token::TokenFlags Flag
¶void setFlagValue(clang::Token::TokenFlags Flag,
bool Val)
void setFlagValue(clang::Token::TokenFlags Flag,
bool Val)
Description
Set a flag to either true or false.
Declared at: clang/include/clang/Lex/Token.h:258
Parameters
- clang::Token::TokenFlags Flag
- bool Val
¶void setIdentifierInfo(clang::IdentifierInfo* II)
void setIdentifierInfo(clang::IdentifierInfo* II)
Declared at: clang/include/clang/Lex/Token.h:187
Parameters
¶void setKind(tok::TokenKind K)
void setKind(tok::TokenKind K)
Declared at: clang/include/clang/Lex/Token.h:93
Parameters
- tok::TokenKind K
¶void setLength(unsigned int Len)
void setLength(unsigned int Len)
Declared at: clang/include/clang/Lex/Token.h:134
Parameters
- unsigned int Len
¶void setLiteralData(const char* Ptr)
void setLiteralData(const char* Ptr)
Declared at: clang/include/clang/Lex/Token.h:220
Parameters
- const char* Ptr
¶void setLocation(clang::SourceLocation L)
void setLocation(clang::SourceLocation L)
Declared at: clang/include/clang/Lex/Token.h:133
Parameters
¶void setRawIdentifierData(const char* Ptr)
void setRawIdentifierData(const char* Ptr)
Declared at: clang/include/clang/Lex/Token.h:208
Parameters
- const char* Ptr
¶void startToken()
void startToken()
Description
Reset all flags to cleared.
Declared at: clang/include/clang/Lex/Token.h:170
¶bool stringifiedInMacro() const
bool stringifiedInMacro() const
Description
Returns true if this token is formed by macro by stringizing or charizing operator.
Declared at: clang/include/clang/Lex/Token.h:299