<html> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> <title>Class Tokenizer - Class Tokenizer</title> <link href="faldoc.css" rel="stylesheet" type="text/css"/> <link href="tabs.css" rel="stylesheet" type="text/css"/> </head> <body class="faldoc_body"> <div class="navitop"> <div class="tabs"> <ul> <li><a href="./index.html"><span>Main</span></a></li> <li><a href="./pages.html"><span>Related pages</span></a></li> <li><a href="./groups.html"><span>Groups</span></a></li> <li><a href="./funset.html"><span>Function sets</span></a></li> <li><a href="./classes.html"><span>Classes</span></a></li> <li><a href="./objects.html"><span>Objects</span></a></li> <li><a href="./functions.html"><span>All functions</span></a></li> <li><a href="./globals.html"><span>Globals</span></a></li> </ul> </div> </div> <hr/> <h1 class="faldoc_title">Class Tokenizer</h1> <p class="faldoc_brief">Simple stream-oriented parser for efficient basic recognition of incoming data. </p> <p class="faldoc_funcdecl"> <b>class</b> Tokenizer( [seps], [options], [tokLen], [source] ) </p> <p class="faldoc_brief"><a href="#more">more...</a></p> <h2 class="faldoc_title">Summary</h2> <table class="faldoc_list"> <tr><td><a href="#hasCurrent">hasCurrent()</a></td><td>Return true if the tokenizer has a current token. </td></tr> <tr><td><a href="#next">next()</a></td><td>Advances the tokenizer up to the next token. </td></tr> <tr><td><a href="#nextToken">nextToken()</a></td><td>Returns the next token from the tokenizer </td></tr> <tr><td><a href="#parse">parse()</a></td><td>Changes or set the source data for this tokenizer. </td></tr> <tr><td><a href="#rewind">rewind()</a></td><td>Resets the status of the tokenizer. </td></tr> <tr><td><a href="#token">token()</a></td><td>Get the current token. </td></tr> </table> <a name="more"><h2 class="faldoc_title">Detailed description</h2></a> <p class="faldoc_funcdecl"> <b>class</b> Tokenizer( [seps], [options], [tokLen], [source] ) </p> <table class="faldoc_function"> <tr><td class="faldoc_optparam">seps</td><td class="faldoc_optparamdesc">A string representing the separators. </td></tr><tr><td class="faldoc_optparam">options</td><td class="faldoc_optparamdesc">Tokenization options. </td></tr><tr><td class="faldoc_optparam">tokLen</td><td class="faldoc_optparamdesc">Maximum length of returned tokens. </td></tr><tr><td class="faldoc_optparam">source</td><td class="faldoc_optparamdesc">The string to be tokenized, or a stream to be read for tokens. </td></tr> </table> <br/> <p class="item_brief">Simple stream-oriented parser for efficient basic recognition of incoming data. </p> <p class="faldoc_text">The tokenizer class is meant to provide simple and efficient logic to parse incoming data (mainly, incoming from string). </p> <p class="faldoc_text"> The source can also be set at a second time with the <a href="./class_Tokenizer.html#parse">Tokenizer.parse</a> method. <b>seps</b> defaults to " " if not given. </p> <p class="faldoc_text"> The <b>options</b> parameter can be a binary combinations of the following values: </p> <ul> <li><b>Tokenizer.groupsep</b>: Groups different tokens into one. If not given, when a token immediately follows another, an empty field is returned. </li><li><b>Tokenizer.bindsep</b>: Return separators inbound with their token. </li><li><b>Tokenizer.trim</b>: trim whitespaces away from returned tokens. </li><li><b>Tokenizer.wsAsToken</b>: Treat a sequence of whitespaces as a single token. </li><li><b>Tokenizer.retsep</b>: Return separators as separate tokens. </li> </ul> <h2 class="faldoc_title">Methods</h2> <h3 class="faldoc_funcname"><a name="hasCurrent">hasCurrent()</a></h3> <p class="item_brief">Return true if the tokenizer has a current token. </p> <p class="faldoc_funcdecl">Tokenizer.hasCurrent( )</p> <table class="faldoc_function"> <tr><td class="faldoc_funcreturn">Returns:</td><td class="faldoc_funcreturndesc">True if a token is now available, false otherwise. </td></tr> </table> <p class="faldoc_text"><p class="faldoc_text"> Contrarily to iterators, it is necessary to call this <a href="./class_Tokenizer.html#next">Tokenizer.next</a> at least once before calling this method. </p> </p> <h3 class="faldoc_funcname"><a name="next">next()</a></h3> <p class="item_brief">Advances the tokenizer up to the next token. </p> <p class="faldoc_funcdecl">Tokenizer.next( )</p> <table class="faldoc_function"> <tr><td class="faldoc_funcreturn">Returns:</td><td class="faldoc_funcreturndesc">True if a new token is now available, false otherwise. </td></tr> <tr><td class="faldoc_raise">Raises:</td><td class="faldoc_raisedesc"> <table class="faldoc_raise"> <tr><td class="faldoc_raiseitem"><a href="./class_IoError.html">IoError</a></td><td class="faldoc_raisedesc">on errors on the underlying stream. </td></tr><tr><td class="faldoc_raiseitem"><a href="./class_CodeError.html">CodeError</a></td><td class="faldoc_raisedesc">if called on an unprepared Tokenizer. </td></tr> </table> </td></tr> </table> <p class="faldoc_text"><p class="faldoc_text">For example: </p> <pre class="faldoc_code"> t = Tokenizer( source|"A string to be tokenized" ) while t.hasCurrent() > "Token: ", t.token() t.next() end </pre> </p> <h3 class="faldoc_funcname"><a name="nextToken">nextToken()</a></h3> <p class="item_brief">Returns the next token from the tokenizer </p> <p class="faldoc_funcdecl">Tokenizer.nextToken( )</p> <table class="faldoc_function"> <tr><td class="faldoc_funcreturn">Returns:</td><td class="faldoc_funcreturndesc">A string or nil at the end of the tokenization. </td></tr> <tr><td class="faldoc_raise">Raises:</td><td class="faldoc_raisedesc"> <table class="faldoc_raise"> <tr><td class="faldoc_raiseitem"><a href="./class_IoError.html">IoError</a></td><td class="faldoc_raisedesc">on errors on the underlying stream. </td></tr><tr><td class="faldoc_raiseitem"><a href="./class_CodeError.html">CodeError</a></td><td class="faldoc_raisedesc">if called on an unprepared Tokenizer. </td></tr> </table> </td></tr> </table> <p class="faldoc_text"><p class="faldoc_text"> This method is actually a combination of <a href="./class_Tokenizer.html#next">Tokenizer.next</a> followed by <a href="./class_Tokenizer.html#token">Tokenizer.token</a>. </p> <p class="faldoc_text">Sample usage: </p> <pre class="faldoc_code"> t = Tokenizer( source|"A string to be tokenized" ) while (token = t.nextToken()) != nil > "Token: ", token end </pre> <p class="faldoc_note"><span class="faldoc_notetype">Note:</span> When looping, remember to check the value of the returned token against nil, as empty strings can be legally returned multiple times, and they are considered false in logic checks. </p> </p> <h3 class="faldoc_funcname"><a name="parse">parse()</a></h3> <p class="item_brief">Changes or set the source data for this tokenizer. </p> <p class="faldoc_funcdecl">Tokenizer.parse( source )</p> <table class="faldoc_function"> <tr><td class="faldoc_param">source</td><td class="faldoc_paramdesc">A string or a stream to be used as a source for the tokenizer. </td></tr> <tr><td class="faldoc_raise">Raises:</td><td class="faldoc_raisedesc"> <table class="faldoc_raise"> <tr><td class="faldoc_raiseitem"><a href="./class_IoError.html">IoError</a></td><td class="faldoc_raisedesc">on errors on the underlying stream. </td></tr> </table> </td></tr> </table> <p class="faldoc_text"><p class="faldoc_text"> The first token is immediately read and set as the current token. If it's not empty, that is, if at least a token can be read, <a href="./class_Tokenizer.html#hasCurrent">Tokenizer.hasCurrent</a> returns true, and <a href="./class_Tokenizer.html#token">Tokenizer.token</a> returns its value. </p> </p> <h3 class="faldoc_funcname"><a name="rewind">rewind()</a></h3> <p class="item_brief">Resets the status of the tokenizer. </p> <p class="faldoc_funcdecl">Tokenizer.rewind( )</p> <table class="faldoc_function"> <tr><td class="faldoc_raise">Raises:</td><td class="faldoc_raisedesc"> <table class="faldoc_raise"> <tr><td class="faldoc_raiseitem"><a href="./class_IoError.html">IoError</a></td><td class="faldoc_raisedesc">if the tokenizer is tokenizing a non-rewindable stream. </td></tr> </table> </td></tr> </table> <h3 class="faldoc_funcname"><a name="token">token()</a></h3> <p class="item_brief">Get the current token. </p> <p class="faldoc_funcdecl">Tokenizer.token( )</p> <table class="faldoc_function"> <tr><td class="faldoc_funcreturn">Returns:</td><td class="faldoc_funcreturndesc">True if a new token is now available, false otherwise. </td></tr> <tr><td class="faldoc_raise">Raises:</td><td class="faldoc_raisedesc"> <table class="faldoc_raise"> <tr><td class="faldoc_raiseitem"><a href="./class_IoError.html">IoError</a></td><td class="faldoc_raisedesc">on errors on the underlying stream. </td></tr><tr><td class="faldoc_raiseitem"><a href="./class_CodeError.html">CodeError</a></td><td class="faldoc_raisedesc">if called on an unprepared Tokenizer, or before next(). </td></tr> </table> </td></tr> </table> <p class="faldoc_text"><p class="faldoc_text">This method returns the current token. </p> </p> <hr/> <div class="navibottom"> <center> <a href="./index.html">Main</a> - <a href="./pages.html">Related pages</a> - <a href="./groups.html">Groups</a> - <a href="./funset.html">Function sets</a> - <a href="./classes.html">Classes</a> - <a href="./objects.html">Objects</a> - <a href="./functions.html">All functions</a> - <a href="./globals.html">Globals</a> </center> </div> </div> <div class="faldoc_signature">Made with <a href="http://www.falconpl.org">faldoc 2.2.0</div> </body> </html>