<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>Split given source into PHP tokens</title>
</head>
<body><div class="manualnavbar" style="text-align: center;">
<div class="prev" style="text-align: left; float: left;"><a href="ref.tokenizer.html">Tokenizer Functions</a></div>
<div class="next" style="text-align: right; float: right;"><a href="function.token-name.html">token_name</a></div>
<div class="up"><a href="ref.tokenizer.html">Tokenizer Functions</a></div>
<div class="home"><a href="index.html">PHP Manual</a></div>
</div><hr /><div id="function.token-get-all" class="refentry">
<div class="refnamediv">
<h1 class="refname">token_get_all</h1>
<p class="verinfo">(PHP 4 >= 4.2.0, PHP 5, PHP 7)</p><p class="refpurpose"><span class="refname">token_get_all</span> — <span class="dc-title">Split given source into PHP tokens</span></p>
</div>
<div class="refsect1 description" id="refsect1-function.token-get-all-description">
<h3 class="title">Description</h3>
<div class="methodsynopsis dc-description">
<span class="type">array</span> <span class="methodname"><strong>token_get_all</strong></span>
( <span class="methodparam"><span class="type">string</span> <code class="parameter">$source</code></span>
[, <span class="methodparam"><span class="type">int</span> <code class="parameter">$flags</code><span class="initializer"> = 0</span></span>
] )</div>
<p class="para rdfs-comment">
<span class="function"><strong>token_get_all()</strong></span> parses the given <code class="parameter">source</code>
string into PHP language tokens using the Zend engine's lexical scanner.
</p>
<p class="para">
For a list of parser tokens, see <a href="tokens.html" class="xref">List of Parser Tokens</a>, or use
<span class="function"><a href="function.token-name.html" class="function">token_name()</a></span> to translate a token value into its string
representation.
</p>
</div>
<div class="refsect1 parameters" id="refsect1-function.token-get-all-parameters">
<h3 class="title">Parameters</h3>
<p class="para">
<dl>
<dt>
<code class="parameter">source</code></dt>
<dd>
<p class="para">
The PHP source to parse.
</p>
</dd>
<dt>
<code class="parameter">flags</code></dt>
<dd>
<p class="para">
Valid flags:
<ul class="itemizedlist">
<li class="listitem">
<span class="simpara">
<strong><code>TOKEN_PARSE</code></strong> - Recognises the ability to use
reserved words in specific contexts.
</span>
</li>
</ul>
</p>
</dd>
</dl>
</p>
</div>
<div class="refsect1 returnvalues" id="refsect1-function.token-get-all-returnvalues">
<h3 class="title">Return Values</h3>
<p class="para">
An array of token identifiers. Each individual token identifier is either
a single character (i.e.: <em>;</em>, <em>.</em>,
<em>></em>, <em>!</em>, etc...),
or a three element array containing the token index in element 0, the string
content of the original token in element 1 and the line number in element 2.
</p>
</div>
<div class="refsect1 changelog" id="refsect1-function.token-get-all-changelog">
<h3 class="title">Changelog</h3>
<p class="para">
<table class="doctable informaltable">
<thead>
<tr>
<th>Version</th>
<th>Description</th>
</tr>
</thead>
<tbody class="tbody">
<tr>
<td>7.0.0</td>
<td>
Added the optional <code class="parameter">flags</code> parameter along with
the <strong><code>TOKEN_PARSE</code></strong> flag.
</td>
</tr>
<tr>
<td>5.2.2</td>
<td>
Line numbers are returned in element 2
</td>
</tr>
</tbody>
</table>
</p>
</div>
<div class="refsect1 examples" id="refsect1-function.token-get-all-examples">
<h3 class="title">Examples</h3>
<p class="para">
<div class="example" id="example-5007">
<p><strong>Example #1 <span class="function"><strong>token_get_all()</strong></span> example</strong></p>
<div class="example-contents">
<div class="phpcode"><code><span style="color: #000000">
<span style="color: #0000BB"><?php<br />$tokens </span><span style="color: #007700">= </span><span style="color: #0000BB">token_get_all</span><span style="color: #007700">(</span><span style="color: #DD0000">'<?php echo; ?>'</span><span style="color: #007700">);<br /><br />foreach (</span><span style="color: #0000BB">$tokens </span><span style="color: #007700">as </span><span style="color: #0000BB">$token</span><span style="color: #007700">) {<br /> if (</span><span style="color: #0000BB">is_array</span><span style="color: #007700">(</span><span style="color: #0000BB">$token</span><span style="color: #007700">)) {<br /> echo </span><span style="color: #DD0000">"Line </span><span style="color: #007700">{</span><span style="color: #0000BB">$token</span><span style="color: #007700">[</span><span style="color: #0000BB">2</span><span style="color: #007700">]}</span><span style="color: #DD0000">: "</span><span style="color: #007700">, </span><span style="color: #0000BB">token_name</span><span style="color: #007700">(</span><span style="color: #0000BB">$token</span><span style="color: #007700">[</span><span style="color: #0000BB">0</span><span style="color: #007700">]), </span><span style="color: #DD0000">" ('</span><span style="color: #007700">{</span><span style="color: #0000BB">$token</span><span style="color: #007700">[</span><span style="color: #0000BB">1</span><span style="color: #007700">]}</span><span style="color: #DD0000">')"</span><span style="color: #007700">, </span><span style="color: #0000BB">PHP_EOL</span><span style="color: #007700">;<br /> }<br />}<br /></span><span style="color: #0000BB">?></span>
</span>
</code></div>
</div>
<div class="example-contents"><p>The above example will output
something similar to:</p></div>
<div class="example-contents screen">
<div class="cdata"><pre>
Line 1: T_OPEN_TAG ('<?php ')
Line 1: T_ECHO ('echo')
Line 1: T_WHITESPACE (' ')
Line 1: T_CLOSE_TAG ('?>')
</pre></div>
</div>
</div>
</p>
<p class="para">
<div class="example" id="example-5008">
<p><strong>Example #2 <span class="function"><strong>token_get_all()</strong></span> incorrect usage example</strong></p>
<div class="example-contents">
<div class="phpcode"><code><span style="color: #000000">
<span style="color: #0000BB"><?php<br />$tokens </span><span style="color: #007700">= </span><span style="color: #0000BB">token_get_all</span><span style="color: #007700">(</span><span style="color: #DD0000">'/* comment */'</span><span style="color: #007700">);<br /><br />foreach (</span><span style="color: #0000BB">$tokens </span><span style="color: #007700">as </span><span style="color: #0000BB">$token</span><span style="color: #007700">) {<br /> if (</span><span style="color: #0000BB">is_array</span><span style="color: #007700">(</span><span style="color: #0000BB">$token</span><span style="color: #007700">)) {<br /> echo </span><span style="color: #DD0000">"Line </span><span style="color: #007700">{</span><span style="color: #0000BB">$token</span><span style="color: #007700">[</span><span style="color: #0000BB">2</span><span style="color: #007700">]}</span><span style="color: #DD0000">: "</span><span style="color: #007700">, </span><span style="color: #0000BB">token_name</span><span style="color: #007700">(</span><span style="color: #0000BB">$token</span><span style="color: #007700">[</span><span style="color: #0000BB">0</span><span style="color: #007700">]), </span><span style="color: #DD0000">" ('</span><span style="color: #007700">{</span><span style="color: #0000BB">$token</span><span style="color: #007700">[</span><span style="color: #0000BB">1</span><span style="color: #007700">]}</span><span style="color: #DD0000">')"</span><span style="color: #007700">, </span><span style="color: #0000BB">PHP_EOL</span><span style="color: #007700">;<br /> }<br />}<br /></span><span style="color: #0000BB">?></span>
</span>
</code></div>
</div>
<div class="example-contents"><p>The above example will output
something similar to:</p></div>
<div class="example-contents screen">
<div class="cdata"><pre>
Line 1: T_INLINE_HTML ('/* comment */')
</pre></div>
</div>
</div>
Note in the previous example that the string is parsed as
<strong><code>T_INLINE_HTML</code></strong> rather than the expected
<strong><code>T_COMMENT</code></strong>. This is because no open tag was used in the
code provided. This would be equivalent to putting a comment outside of the
PHP tags in a normal file.
</p>
<p class="para">
<div class="example" id="example-5009">
<p><strong>Example #3
<span class="function"><strong>token_get_all()</strong></span> on a class using a reserved word example
</strong></p>
<div class="example-contents">
<div class="phpcode"><code><span style="color: #000000">
<span style="color: #0000BB"><?php<br /><br />$source </span><span style="color: #007700">= <<<'code'<br /></span><span style="color: #DD0000"><?php<br /><br />class A<br />{<br /> const PUBLIC = 1;<br />}<br /></span><span style="color: #007700">code;<br /><br /></span><span style="color: #0000BB">$tokens </span><span style="color: #007700">= </span><span style="color: #0000BB">token_get_all</span><span style="color: #007700">(</span><span style="color: #0000BB">$source</span><span style="color: #007700">, </span><span style="color: #0000BB">TOKEN_PARSE</span><span style="color: #007700">);<br /><br />foreach (</span><span style="color: #0000BB">$tokens </span><span style="color: #007700">as </span><span style="color: #0000BB">$token</span><span style="color: #007700">) {<br /> if (</span><span style="color: #0000BB">is_array</span><span style="color: #007700">(</span><span style="color: #0000BB">$token</span><span style="color: #007700">)) {<br /> echo </span><span style="color: #0000BB">token_name</span><span style="color: #007700">(</span><span style="color: #0000BB">$token</span><span style="color: #007700">[</span><span style="color: #0000BB">0</span><span style="color: #007700">]) , </span><span style="color: #0000BB">PHP_EOL</span><span style="color: #007700">;<br /> }<br />}<br /></span><span style="color: #0000BB">?></span>
</span>
</code></div>
</div>
<div class="example-contents"><p>The above example will output
something similar to:</p></div>
<div class="example-contents screen">
<div class="cdata"><pre>
T_OPEN_TAG
T_WHITESPACE
T_CLASS
T_WHITESPACE
T_STRING
T_CONST
T_WHITESPACE
T_STRING
T_LNUMBER
</pre></div>
</div>
</div>
Without the <strong><code>TOKEN_PARSE</code></strong> flag, the penultimate
token (<strong><code>T_STRING</code></strong>) would have been
<strong><code>T_PUBLIC</code></strong>.
</p>
</div>
<div class="refsect1 seealso" id="refsect1-function.token-get-all-seealso">
<h3 class="title">See Also</h3>
<p class="para">
<ul class="simplelist">
<li class="member"><span class="function"><a href="function.token-name.html" class="function" rel="rdfs-seeAlso">token_name()</a> - Get the symbolic name of a given PHP token</span></li>
</ul>
</p>
</div>
</div><hr /><div class="manualnavbar" style="text-align: center;">
<div class="prev" style="text-align: left; float: left;"><a href="ref.tokenizer.html">Tokenizer Functions</a></div>
<div class="next" style="text-align: right; float: right;"><a href="function.token-name.html">token_name</a></div>
<div class="up"><a href="ref.tokenizer.html">Tokenizer Functions</a></div>
<div class="home"><a href="index.html">PHP Manual</a></div>
</div></body></html>
