ΒΆvoid clang_tokenize(CXTranslationUnit TU,
CXSourceRange Range,
CXToken** Tokens,
unsigned int* NumTokens)
void clang_tokenize(CXTranslationUnit TU,
CXSourceRange Range,
CXToken** Tokens,
unsigned int* NumTokens)
Description
Tokenize the source code described by the given range into raw lexical tokens.
Declared at: clang/include/clang-c/Index.h:5118
Parameters
- CXTranslationUnit TU
- the translation unit whose text is being tokenized.
- CXSourceRange Range
- the source range in which text should be tokenized. All of the tokens produced by tokenization will fall within this source range,
- CXToken** Tokens
- this pointer will be set to point to the array of tokens that occur within the given source range. The returned pointer must be freed with clang_disposeTokens() before the translation unit is destroyed.
- unsigned int* NumTokens
- will be set to the number of tokens in the \c *Tokens array.