PhpToken represents a token produced while parsing PHP source into lexical units.
What Is PhpToken::tokenize in PHP?
PhpToken::tokenize() splits PHP source into token objects.
PhpToken::tokenize(string $code, int $flags = ...) : arrayParameters:
$code— Required. The code argument used to configure this operation; its accepted values follow the declared PHP type.$flags— Optional. A set of documented option bits controlling how the operation behaves.
Return type: array. Token IDs and whitespace tokens are significant when building analyzers or formatters.
Example 1: Understanding the Basic Behavior
<?php
$tokens = PhpToken::tokenize("<?php echo 7;");
foreach ($tokens as $token) { echo $token->getTokenName() . "\n"; }Output:
T_OPEN_TAG
T_ECHO
T_WHITESPACE
T_LNUMBER
;Explanation: PhpToken::tokenize() splits PHP source into token objects.
Example 2: Testing Another Case
<?php
$tokens = PhpToken::tokenize("<?php echo 11;");
foreach ($tokens as $token) { echo $token->getTokenName() . "\n"; }Output:
T_OPEN_TAG
T_ECHO
T_WHITESPACE
T_LNUMBER
;Explanation: Token IDs and whitespace tokens are significant when building analyzers or formatters.
Practical Example: Applying the Operation
<?php
$tokens = PhpToken::tokenize("<?php echo 42;");
foreach ($tokens as $token) { echo $token->getTokenName() . "\n"; }Output:
T_OPEN_TAG
T_ECHO
T_WHITESPACE
T_LNUMBER
;Explanation: Token IDs and token text vary with the supplied source; whitespace tokens can matter.
Important Behavior and Edge Cases
- Token IDs and whitespace tokens are significant when building analyzers or formatters.
- Token IDs and token text vary with the supplied source; whitespace tokens can matter.
Comparison With Related Approaches
| Approach | When to Use It |
|---|---|
PhpToken::tokenize | PhpToken::tokenize() splits PHP source into token objects. |
token_get_all() | PhpToken offers objects with methods, while token_get_all() returns traditional arrays. |
Common Beginner Mistakes
| Mistake | Correct Approach |
|---|---|
| Misinterpreting the documented behavior of PhpToken::tokenize() for the supplied value. | Token IDs and whitespace tokens are significant when building analyzers or formatters. |
| Choosing token_get_all() without checking whether it provides the same result. | PhpToken offers objects with methods, while token_get_all() returns traditional arrays. |
Key Takeaways
- PhpToken::tokenize() splits PHP source into token objects.
- Token IDs and whitespace tokens are significant when building analyzers or formatters.
- Token IDs and token text vary with the supplied source; whitespace tokens can matter.
The examples demonstrate where PhpToken::tokenize fits within php token inspection. Verify the relevant preconditions whenever you adapt the code to real data.