ΒΆvoid clang_tokenize(CXTranslationUnit TU,
                    CXSourceRange Range,
                    CXToken** Tokens,
                    unsigned int* NumTokens)

Description

Tokenize the source code described by the given range into raw lexical tokens.

Declared at: clang/include/clang-c/Index.h:5118

Parameters

CXTranslationUnit TU
the translation unit whose text is being tokenized.
CXSourceRange Range
the source range in which text should be tokenized. All of the tokens produced by tokenization will fall within this source range,
CXToken** Tokens
this pointer will be set to point to the array of tokens that occur within the given source range. The returned pointer must be freed with clang_disposeTokens() before the translation unit is destroyed.
unsigned int* NumTokens
will be set to the number of tokens in the \c *Tokens array.