<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Class Tokenizer - Class Tokenizer</title>
<link href="faldoc.css" rel="stylesheet" type="text/css"/>
<link href="tabs.css" rel="stylesheet" type="text/css"/>
</head>
<body class="faldoc_body">
<div class="navitop">
<div class="tabs">
<ul>
<li><a href="./index.html"><span>Main</span></a></li>
<li><a href="./pages.html"><span>Related pages</span></a></li>
<li><a href="./groups.html"><span>Groups</span></a></li>
<li><a href="./funset.html"><span>Function sets</span></a></li>
<li><a href="./classes.html"><span>Classes</span></a></li>
<li><a href="./objects.html"><span>Objects</span></a></li>
<li><a href="./functions.html"><span>All functions</span></a></li>
<li><a href="./globals.html"><span>Globals</span></a></li>
</ul>
</div>
</div>
<hr/>
<h1 class="faldoc_title">Class Tokenizer</h1>
<p class="faldoc_brief">Simple stream-oriented parser for efficient basic recognition of incoming data. </p>
<p class="faldoc_funcdecl">
<b>class</b> Tokenizer( [seps], [options], [tokLen], [source] )
</p>
<p class="faldoc_brief"><a href="#more">more...</a></p>
<h2 class="faldoc_title">Summary</h2>
<table class="faldoc_list">
<tr><td><a href="#hasCurrent">hasCurrent()</a></td><td>Return true if the tokenizer has a current token. </td></tr>
<tr><td><a href="#next">next()</a></td><td>Advances the tokenizer up to the next token. </td></tr>
<tr><td><a href="#nextToken">nextToken()</a></td><td>Returns the next token from the tokenizer </td></tr>
<tr><td><a href="#parse">parse()</a></td><td>Changes or set the source data for this tokenizer. </td></tr>
<tr><td><a href="#rewind">rewind()</a></td><td>Resets the status of the tokenizer. </td></tr>
<tr><td><a href="#token">token()</a></td><td>Get the current token. </td></tr>
</table>
<a name="more"><h2 class="faldoc_title">Detailed description</h2></a>
<p class="faldoc_funcdecl">
<b>class</b> Tokenizer( [seps], [options], [tokLen], [source] )
</p>
<table class="faldoc_function">
<tr><td class="faldoc_optparam">seps</td><td class="faldoc_optparamdesc">A string representing the separators. </td></tr><tr><td class="faldoc_optparam">options</td><td class="faldoc_optparamdesc">Tokenization options. </td></tr><tr><td class="faldoc_optparam">tokLen</td><td class="faldoc_optparamdesc">Maximum length of returned tokens. </td></tr><tr><td class="faldoc_optparam">source</td><td class="faldoc_optparamdesc">The string to be tokenized, or a stream to be read for tokens. </td></tr>
</table>
<br/>
<p class="item_brief">Simple stream-oriented parser for efficient basic recognition of incoming data. </p>
<p class="faldoc_text">The tokenizer class is meant to provide simple and efficient logic to parse incoming data (mainly, incoming from string). </p>
<p class="faldoc_text"> The source can also be set at a second time with the <a href="./class_Tokenizer.html#parse">Tokenizer.parse</a> method. <b>seps</b> defaults to " " if not given. </p>
<p class="faldoc_text"> The <b>options</b> parameter can be a binary combinations of the following values: </p>
<ul>
<li><b>Tokenizer.groupsep</b>: Groups different tokens into one. If not given, when a token immediately follows another, an empty field is returned. </li><li><b>Tokenizer.bindsep</b>: Return separators inbound with their token. </li><li><b>Tokenizer.trim</b>: trim whitespaces away from returned tokens. </li><li><b>Tokenizer.wsAsToken</b>: Treat a sequence of whitespaces as a single token. </li><li><b>Tokenizer.retsep</b>: Return separators as separate tokens. </li>
</ul>
<h2 class="faldoc_title">Methods</h2>
<h3 class="faldoc_funcname"><a name="hasCurrent">hasCurrent()</a></h3>
<p class="item_brief">Return true if the tokenizer has a current token. </p>
<p class="faldoc_funcdecl">Tokenizer.hasCurrent( )</p>
<table class="faldoc_function">
<tr><td class="faldoc_funcreturn">Returns:</td><td class="faldoc_funcreturndesc">True if a token is now available, false otherwise. </td></tr>
</table>
<p class="faldoc_text"><p class="faldoc_text"> Contrarily to iterators, it is necessary to call this <a href="./class_Tokenizer.html#next">Tokenizer.next</a> at least once before calling this method. </p>
</p>
<h3 class="faldoc_funcname"><a name="next">next()</a></h3>
<p class="item_brief">Advances the tokenizer up to the next token. </p>
<p class="faldoc_funcdecl">Tokenizer.next( )</p>
<table class="faldoc_function">
<tr><td class="faldoc_funcreturn">Returns:</td><td class="faldoc_funcreturndesc">True if a new token is now available, false otherwise. </td></tr>
<tr><td class="faldoc_raise">Raises:</td><td class="faldoc_raisedesc">
<table class="faldoc_raise">
<tr><td class="faldoc_raiseitem"><a href="./class_IoError.html">IoError</a></td><td class="faldoc_raisedesc">on errors on the underlying stream. </td></tr><tr><td class="faldoc_raiseitem"><a href="./class_CodeError.html">CodeError</a></td><td class="faldoc_raisedesc">if called on an unprepared Tokenizer. </td></tr>
</table>
</td></tr>
</table>
<p class="faldoc_text"><p class="faldoc_text">For example: </p>
<pre class="faldoc_code">
t = Tokenizer( source|"A string to be tokenized" )
while t.hasCurrent()
> "Token: ", t.token()
t.next()
end
</pre>
</p>
<h3 class="faldoc_funcname"><a name="nextToken">nextToken()</a></h3>
<p class="item_brief">Returns the next token from the tokenizer </p>
<p class="faldoc_funcdecl">Tokenizer.nextToken( )</p>
<table class="faldoc_function">
<tr><td class="faldoc_funcreturn">Returns:</td><td class="faldoc_funcreturndesc">A string or nil at the end of the tokenization. </td></tr>
<tr><td class="faldoc_raise">Raises:</td><td class="faldoc_raisedesc">
<table class="faldoc_raise">
<tr><td class="faldoc_raiseitem"><a href="./class_IoError.html">IoError</a></td><td class="faldoc_raisedesc">on errors on the underlying stream. </td></tr><tr><td class="faldoc_raiseitem"><a href="./class_CodeError.html">CodeError</a></td><td class="faldoc_raisedesc">if called on an unprepared Tokenizer. </td></tr>
</table>
</td></tr>
</table>
<p class="faldoc_text"><p class="faldoc_text"> This method is actually a combination of <a href="./class_Tokenizer.html#next">Tokenizer.next</a> followed by <a href="./class_Tokenizer.html#token">Tokenizer.token</a>. </p>
<p class="faldoc_text">Sample usage: </p>
<pre class="faldoc_code">
t = Tokenizer( source|"A string to be tokenized" )
while (token = t.nextToken()) != nil
> "Token: ", token
end
</pre>
<p class="faldoc_note"><span class="faldoc_notetype">Note:</span> When looping, remember to check the value of the returned token against nil, as empty strings can be legally returned multiple times, and they are considered false in logic checks. </p>
</p>
<h3 class="faldoc_funcname"><a name="parse">parse()</a></h3>
<p class="item_brief">Changes or set the source data for this tokenizer. </p>
<p class="faldoc_funcdecl">Tokenizer.parse( source )</p>
<table class="faldoc_function">
<tr><td class="faldoc_param">source</td><td class="faldoc_paramdesc">A string or a stream to be used as a source for the tokenizer. </td></tr>
<tr><td class="faldoc_raise">Raises:</td><td class="faldoc_raisedesc">
<table class="faldoc_raise">
<tr><td class="faldoc_raiseitem"><a href="./class_IoError.html">IoError</a></td><td class="faldoc_raisedesc">on errors on the underlying stream. </td></tr>
</table>
</td></tr>
</table>
<p class="faldoc_text"><p class="faldoc_text"> The first token is immediately read and set as the current token. If it's not empty, that is, if at least a token can be read, <a href="./class_Tokenizer.html#hasCurrent">Tokenizer.hasCurrent</a> returns true, and <a href="./class_Tokenizer.html#token">Tokenizer.token</a> returns its value. </p>
</p>
<h3 class="faldoc_funcname"><a name="rewind">rewind()</a></h3>
<p class="item_brief">Resets the status of the tokenizer. </p>
<p class="faldoc_funcdecl">Tokenizer.rewind( )</p>
<table class="faldoc_function">
<tr><td class="faldoc_raise">Raises:</td><td class="faldoc_raisedesc">
<table class="faldoc_raise">
<tr><td class="faldoc_raiseitem"><a href="./class_IoError.html">IoError</a></td><td class="faldoc_raisedesc">if the tokenizer is tokenizing a non-rewindable stream. </td></tr>
</table>
</td></tr>
</table>
<h3 class="faldoc_funcname"><a name="token">token()</a></h3>
<p class="item_brief">Get the current token. </p>
<p class="faldoc_funcdecl">Tokenizer.token( )</p>
<table class="faldoc_function">
<tr><td class="faldoc_funcreturn">Returns:</td><td class="faldoc_funcreturndesc">True if a new token is now available, false otherwise. </td></tr>
<tr><td class="faldoc_raise">Raises:</td><td class="faldoc_raisedesc">
<table class="faldoc_raise">
<tr><td class="faldoc_raiseitem"><a href="./class_IoError.html">IoError</a></td><td class="faldoc_raisedesc">on errors on the underlying stream. </td></tr><tr><td class="faldoc_raiseitem"><a href="./class_CodeError.html">CodeError</a></td><td class="faldoc_raisedesc">if called on an unprepared Tokenizer, or before next(). </td></tr>
</table>
</td></tr>
</table>
<p class="faldoc_text"><p class="faldoc_text">This method returns the current token. </p>
</p>
<hr/>
<div class="navibottom">
<center>
<a href="./index.html">Main</a> - <a href="./pages.html">Related pages</a> - <a href="./groups.html">Groups</a> - <a href="./funset.html">Function sets</a> - <a href="./classes.html">Classes</a> - <a href="./objects.html">Objects</a> - <a href="./functions.html">All functions</a> - <a href="./globals.html">Globals</a>
</center>
</div>
</div>
<div class="faldoc_signature">Made with <a href="http://www.falconpl.org">faldoc 2.2.0</div>
</body>
</html>
