public tokenize(string $code, ?PhpParser\ErrorHandler $errorHandler = NULL) : array

Tokenize the provided source code.

The token array is in the same format as provided by the PhpToken::tokenize() method in PHP 8.0. The tokens are instances of PhpParser\Token, to abstract over a polyfill implementation in earlier PHP version.

The token array is terminated by a sentinel token with token ID 0. The token array does not discard any tokens (i.e. whitespace and comments are included). The token position attributes are against this token array.

  • param string $code The source code to tokenize.
  • param \ErrorHandler|null $errorHandler Error handler to use for lexing errors. Defaults to ErrorHandler\Throwing.
  • return \Token[] Tokens
protected postprocessTokens(array $tokens, PhpParser\ErrorHandler $errorHandler) : void
  • param list<\Token> $tokens
private handleInvalidCharacter(PhpParser\Token $token, PhpParser\ErrorHandler $errorHandler) : void
private isUnterminatedComment(PhpParser\Token $token) : bool
© 2024 Bruce Wells
Search Namespaces \ Classes