¶std::vector<syntax::Token> tokenize(
    clang::FileID FID,
    const clang::SourceManager& SM,
    const clang::LangOptions& LO)

Description

Lex the text buffer, corresponding to \p FID, in raw mode and record the resulting spelled tokens. Does minimal post-processing on raw identifiers, setting the appropriate token kind (instead of the raw_identifier reported by lexer in raw mode). This is a very low-level function, most users should prefer to use TokenCollector. Lexing in raw mode produces wildly different results from what one might expect when running a C++ frontend, e.g. preprocessor does not run at all. The result will *not* have a 'eof' token at the end.

Declared at: clang/include/clang/Tooling/Syntax/Tokens.h:406

Parameters

clang::FileID FID

const clang::SourceManager& SM

const clang::LangOptions& LO