<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> <html xmlns:fn="http://www.w3.org/2005/02/xpath-functions"> <head> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> <link rel="stylesheet" href="../../../../doc/otp_doc.css" type="text/css"> <title>Erlang -- leex</title> </head> <body bgcolor="white" text="#000000" link="#0000ff" vlink="#ff00ff" alink="#ff0000"><div id="container"> <script id="js" type="text/javascript" language="JavaScript" src="../../../../doc/js/flipmenu/flipmenu.js"></script><script id="js2" type="text/javascript" src="../../../../doc/js/erlresolvelinks.js"></script><script language="JavaScript" type="text/javascript"> <!-- function getWinHeight() { var myHeight = 0; if( typeof( window.innerHeight ) == 'number' ) { //Non-IE myHeight = window.innerHeight; } else if( document.documentElement && ( document.documentElement.clientWidth || document.documentElement.clientHeight ) ) { //IE 6+ in 'standards compliant mode' myHeight = document.documentElement.clientHeight; } else if( document.body && ( document.body.clientWidth || document.body.clientHeight ) ) { //IE 4 compatible myHeight = document.body.clientHeight; } return myHeight; } function setscrollpos() { var objf=document.getElementById('loadscrollpos'); document.getElementById("leftnav").scrollTop = objf.offsetTop - getWinHeight()/2; } function addEvent(obj, evType, fn){ if (obj.addEventListener){ obj.addEventListener(evType, fn, true); return true; } else if (obj.attachEvent){ var r = obj.attachEvent("on"+evType, fn); return r; } else { return false; } } addEvent(window, 'load', setscrollpos); //--></script><div id="leftnav"><div class="innertube"> <img alt="Erlang logo" src="../../../../doc/erlang-logo.png"><br><small><a href="index.html">Reference Manual</a><br><a href="release_notes.html">Release Notes</a><br><a href="../pdf/parsetools-2.0.7.pdf">PDF</a><br><a href="../../../../doc/index.html">Top</a></small><p><strong>Parse Tools</strong><br><strong>Reference Manual</strong><br><small>Version 2.0.7</small></p> <br><a href="javascript:openAllFlips()">Expand All</a><br><a href="javascript:closeAllFlips()">Contract All</a><p><small><strong>Table of Contents</strong></small></p> <ul class="flipMenu"> <li id="no" title="yecc " expanded="false">yecc<ul> <li><a href="yecc.html"> Top of manual page </a></li> <li title="file-1"><a href="yecc.html#file-1">file/1</a></li> <li title="format_error-1"><a href="yecc.html#format_error-1">format_error/1</a></li> </ul> </li> <li id="loadscrollpos" title="leex " expanded="true">leex<ul> <li><a href="leex.html"> Top of manual page </a></li> <li title="file-1"><a href="leex.html#file-1">file/1</a></li> <li title="file-2"><a href="leex.html#file-2">file/2</a></li> <li title="format_error-1"><a href="leex.html#format_error-1">format_error/1</a></li> <li title="string-1"><a href="leex.html#string-1">string/1</a></li> <li title="string-2"><a href="leex.html#string-2">string/2</a></li> <li title="token-2"><a href="leex.html#token-2">token/2</a></li> <li title="token-3"><a href="leex.html#token-3">token/3</a></li> <li title="tokens-2"><a href="leex.html#tokens-2">tokens/2</a></li> <li title="tokens-3"><a href="leex.html#tokens-3">tokens/3</a></li> </ul> </li> </ul> </div></div> <div id="content"> <div class="innertube"> <!-- refpage --><center><h1>leex</h1></center> <h3>MODULE</h3> <div class="REFBODY">leex</div> <h3>MODULE SUMMARY</h3> <div class="REFBODY">Lexical analyzer generator for Erlang</div> <h3>DESCRIPTION</h3> <div class="REFBODY"><p> <p>A regular expression based lexical analyzer generator for Erlang, similar to lex or flex.</p> <div class="note"> <div class="label">Note</div> <div class="content"><p><p>The Leex module should be considered experimental as it will be subject to changes in future releases.</p></p></div> </div> </p></div> <h3><a name="id62247">DATA TYPES</a></h3> <div class="REFBODY"> <div class="example"><pre> ErrorInfo = {ErrorLine,module(),error_descriptor()} ErrorLine = integer() Token = tuple()</pre></div> </div> <h3>EXPORTS</h3> <p><a name="file-1"><span class="bold_code">file(FileName) -> ok | error</span></a><br><a name="file-2"><span class="bold_code">file(FileName, Options) -> ok | error</span></a><br></p> <div class="REFBODY"> <p>Types:</p> <div class="REFTYPES"> <span class="bold_code">FileName = filename()</span><br> </div> <div class="REFTYPES"> <span class="bold_code">Options = Option | [Option]</span><br> </div> <div class="REFTYPES"> <span class="bold_code">Option = - see below -</span><br> </div> <div class="REFTYPES"> <span class="bold_code">FileReturn = {ok, Scannerfile} | {ok, Scannerfile, Warnings} | error | {error, Warnings, Errors}</span><br> </div> <div class="REFTYPES"> <span class="bold_code">Scannerfile = filename()</span><br> </div> <div class="REFTYPES"> <span class="bold_code">Warnings = Errors = [{filename(), [ErrorInfo]}]</span><br> </div> </div> <div class="REFBODY"><p> <p>Generates a lexical analyzer from the definition in the input file. The input file has the extension <span class="code">.xrl</span>. This is added to the filename if it is not given. The resulting module is the Xrl filename without the <span class="code">.xrl</span> extension.</p> <p>The current options are:</p> <dl> <dt><strong><span class="code">dfa_graph</span></strong></dt> <dd> <p>Generates a <span class="code">.dot</span> file which contains a description of the DFA in a format which can be viewed with Graphviz, <span class="code">www.graphviz.com</span>.</p> </dd> <dt><strong><span class="code">{includefile,Includefile}</span></strong></dt> <dd> <p>Uses a specific or customised prologue file instead of default <span class="code">lib/parsetools/include/leexinc.hrl</span> which is otherwise included.</p> </dd> <dt><strong><span class="code">{report_errors, bool()}</span></strong></dt> <dd> <p>Causes errors to be printed as they occur. Default is <span class="code">true</span>.</p> </dd> <dt><strong><span class="code">{report_warnings, bool()}</span></strong></dt> <dd> <p>Causes warnings to be printed as they occur. Default is <span class="code">true</span>.</p> </dd> <dt><strong><span class="code">warnings_as_errors</span></strong></dt> <dd> <p>Causes warnings to be treated as errors.</p> </dd> <dt><strong><span class="code">{report, bool()}</span></strong></dt> <dd> <p>This is a short form for both <span class="code">report_errors</span> and <span class="code">report_warnings</span>.</p> </dd> <dt><strong><span class="code">{return_errors, bool()}</span></strong></dt> <dd> <p>If this flag is set, <span class="code">{error, Errors, Warnings}</span> is returned when there are errors. Default is <span class="code">false</span>.</p> </dd> <dt><strong><span class="code">{return_warnings, bool()}</span></strong></dt> <dd> <p>If this flag is set, an extra field containing <span class="code">Warnings</span> is added to the tuple returned upon success. Default is <span class="code">false</span>.</p> </dd> <dt><strong><span class="code">{return, bool()}</span></strong></dt> <dd> <p>This is a short form for both <span class="code">return_errors</span> and <span class="code">return_warnings</span>.</p> </dd> <dt><strong><span class="code">{scannerfile, Scannerfile}</span></strong></dt> <dd> <p><span class="code">Scannerfile</span> is the name of the file that will contain the Erlang scanner code that is generated. The default (<span class="code">""</span>) is to add the extension <span class="code">.erl</span> to <span class="code">FileName</span> stripped of the <span class="code">.xrl</span> extension.</p> </dd> <dt><strong><span class="code">{verbose, bool()}</span></strong></dt> <dd> <p>Outputs information from parsing the input file and generating the internal tables.</p> </dd> </dl> <p>Any of the Boolean options can be set to <span class="code">true</span> by stating the name of the option. For example, <span class="code">verbose</span> is equivalent to <span class="code">{verbose, true}</span>.</p> <p>Leex will add the extension <span class="code">.hrl</span> to the <span class="code">Includefile</span> name and the extension <span class="code">.erl</span> to the <span class="code">Scannerfile</span> name, unless the extension is already there.</p> </p></div> <p><a name="format_error-1"><span class="bold_code">format_error(ErrorInfo) -> Chars</span></a><br></p> <div class="REFBODY"> <p>Types:</p> <div class="REFTYPES"> <span class="bold_code">Chars = [char() | Chars]</span><br> </div> </div> <div class="REFBODY"><p> <p>Returns a string which describes the error <span class="code">ErrorInfo</span> returned when there is an error in a regular expression.</p> </p></div> <h3><a name="id63091">GENERATED SCANNER EXPORTS</a></h3> <div class="REFBODY"> <p>The following functions are exported by the generated scanner.</p> </div> <h3>EXPORTS</h3> <p><a name="string-1"><span class="bold_code">string(String) -> StringRet</span></a><br><a name="string-2"><span class="bold_code">string(String, StartLine) -> StringRet</span></a><br></p> <div class="REFBODY"> <p>Types:</p> <div class="REFTYPES"> <span class="bold_code">String = string()</span><br> </div> <div class="REFTYPES"> <span class="bold_code">StringRet = {ok,Tokens,EndLine} | ErrorInfo</span><br> </div> <div class="REFTYPES"> <span class="bold_code">Tokens = [Token]</span><br> </div> <div class="REFTYPES"> <span class="bold_code">EndLine = StartLine = integer()</span><br> </div> </div> <div class="REFBODY"><p> <p>Scans <span class="code">String</span> and returns all the tokens in it, or an error.</p> <div class="note"> <div class="label">Note</div> <div class="content"><p><p>It is an error if not all of the characters in <span class="code">String</span> are consumed.</p></p></div> </div> </p></div> <p><a name="token-2"><span class="bold_code">token(Cont, Chars) -> {more,Cont1} | {done,TokenRet,RestChars} </span></a><br><a name="token-3"><span class="bold_code">token(Cont, Chars, StartLine) -> {more,Cont1} | {done,TokenRet,RestChars} </span></a><br></p> <div class="REFBODY"> <p>Types:</p> <div class="REFTYPES"> <span class="bold_code">Cont = [] | Cont1</span><br> </div> <div class="REFTYPES"> <span class="bold_code">Cont1 = tuple()</span><br> </div> <div class="REFTYPES"> <span class="bold_code">Chars = RestChars = string() | eof</span><br> </div> <div class="REFTYPES"> <span class="bold_code">TokenRet = {ok, Token, EndLine} | {eof, EndLine} | ErrorInfo</span><br> </div> <div class="REFTYPES"> <span class="bold_code">StartLine = EndLine = integer()</span><br> </div> </div> <div class="REFBODY"><p> <p>This is a re-entrant call to try and scan one token from <span class="code">Chars</span>. If there are enough characters in <span class="code">Chars</span> to either scan a token or detect an error then this will be returned with <span class="code">{done,...}</span>. Otherwise <span class="code">{cont,Cont}</span> will be returned where <span class="code">Cont</span> is used in the next call to <span class="code">token()</span> with more characters to try an scan the token. This is continued until a token has been scanned. <span class="code">Cont</span> is initially <span class="code">[]</span>.</p> <p>It is not designed to be called directly by an application but used through the i/o system where it can typically be called in an application by:</p> <div class="example"><pre> io:request(InFile, {get_until,Prompt,Module,token,[Line]}) -> TokenRet</pre></div> </p></div> <p><a name="tokens-2"><span class="bold_code">tokens(Cont, Chars) -> {more,Cont1} | {done,TokensRet,RestChars} </span></a><br><a name="tokens-3"><span class="bold_code">tokens(Cont, Chars, StartLine) -> {more,Cont1} | {done,TokensRet,RestChars} </span></a><br></p> <div class="REFBODY"> <p>Types:</p> <div class="REFTYPES"> <span class="bold_code">Cont = [] | Cont1</span><br> </div> <div class="REFTYPES"> <span class="bold_code">Cont1 = tuple()</span><br> </div> <div class="REFTYPES"> <span class="bold_code">Chars = RestChars = string() | eof</span><br> </div> <div class="REFTYPES"> <span class="bold_code">TokensRet = {ok, Tokens, EndLine} | {eof, EndLine} | ErrorInfo</span><br> </div> <div class="REFTYPES"> <span class="bold_code">Tokens = [Token]</span><br> </div> <div class="REFTYPES"> <span class="bold_code">StartLine = EndLine = integer()</span><br> </div> </div> <div class="REFBODY"><p> <p>This is a re-entrant call to try and scan tokens from <span class="code">Chars</span>. If there are enough characters in <span class="code">Chars</span> to either scan tokens or detect an error then this will be returned with <span class="code">{done,...}</span>. Otherwise <span class="code">{cont,Cont}</span> will be returned where <span class="code">Cont</span> is used in the next call to <span class="code">tokens()</span> with more characters to try an scan the tokens. This is continued until all tokens have been scanned. <span class="code">Cont</span> is initially <span class="code">[]</span>.</p> <p>This functions differs from <span class="code">token</span> in that it will continue to scan tokens upto and including an <span class="code">{end_token,Token}</span> has been scanned (see next section). It will then return all the tokens. This is typically used for scanning grammars like Erlang where there is an explicit end token, <span class="code">'.'</span>. If no end token is found then the whole file will be scanned and returned. If an error occurs then all tokens upto and including the next end token will be skipped.</p> <p>It is not designed to be called directly by an application but used through the i/o system where it can typically be called in an application by:</p> <div class="example"><pre> io:request(InFile, {get_until,Prompt,Module,tokens,[Line]}) -> TokensRet</pre></div> </p></div> <h3><a name="id63367">Input File Format</a></h3> <div class="REFBODY"> <p>Erlang style comments starting with a <span class="code">%</span> are allowed in scanner files. A definition file has the following format:</p> <div class="example"><pre> <Header> Definitions. <Macro Definitions> Rules. <Token Rules> Erlang code. <Erlang code></pre></div> <p>The "Definitions.", "Rules." and "Erlang code." headings are mandatory and must occur at the beginning of a source line. The <Header>, <Macro Definitions> and <Erlang code> sections may be empty but there must be at least one rule.</p> <p>Macro definitions have the following format:</p> <div class="example"><pre> NAME = VALUE</pre></div> <p>and there must be spaces around <span class="code">=</span>. Macros can be used in the regular expressions of rules by writing <span class="code">{NAME}</span>.</p> <div class="note"> <div class="label">Note</div> <div class="content"><p><p>When macros are expanded in expressions the macro calls are replaced by the macro value without any form of quoting or enclosing in parentheses.</p></p></div> </div> <p>Rules have the following format:</p> <div class="example"><pre> <Regexp> : <Erlang code>.</pre></div> <p>The <Regexp> must occur at the start of a line and not include any blanks; use <span class="code">\t</span> and <span class="code">\s</span> to include TAB and SPACE characters in the regular expression. If <Regexp> matches then the corresponding <Erlang code> is evaluated to generate a token. With the Erlang code the following predefined variables are available:</p> <dl> <dt><strong><span class="code">TokenChars</span></strong></dt> <dd> <p>A list of the characters in the matched token.</p> </dd> <dt><strong><span class="code">TokenLen</span></strong></dt> <dd> <p>The number of characters in the matched token.</p> </dd> <dt><strong><span class="code">TokenLine</span></strong></dt> <dd> <p>The line number where the token occurred.</p> </dd> </dl> <p>The code must return:</p> <dl> <dt><strong><span class="code">{token,Token}</span></strong></dt> <dd> <p>Return <span class="code">Token</span> to the caller.</p> </dd> <dt><strong><span class="code">{end_token,Token}</span></strong></dt> <dd> <p>Return <span class="code">Token</span> and is last token in a tokens call.</p> </dd> <dt><strong><span class="code">skip_token</span></strong></dt> <dd> <p>Skip this token completely.</p> </dd> <dt><strong><span class="code">{error,ErrString}</span></strong></dt> <dd> <p>An error in the token, <span class="code">ErrString</span> is a string describing the error.</p> </dd> </dl> <p>It is also possible to push back characters into the input characters with the following returns:</p> <ul> <li><span class="code">{token,Token,PushBackList}</span></li> <li><span class="code">{end_token,Token,PushBackList}</span></li> <li><span class="code">{skip_token,PushBackList}</span></li> </ul> <p>These have the same meanings as the normal returns but the characters in <span class="code">PushBackList</span> will be prepended to the input characters and scanned for the next token. Note that pushing back a newline will mean the line numbering will no longer be correct.</p> <div class="note"> <div class="label">Note</div> <div class="content"><p><p>Pushing back characters gives you unexpected possibilities to cause the scanner to loop!</p></p></div> </div> <p>The following example would match a simple Erlang integer or float and return a token which could be sent to the Erlang parser:</p> <div class="example"><pre> D = [0-9] {D}+ : {token,{integer,TokenLine,list_to_integer(TokenChars)}}. {D}+\.{D}+((E|e)(\+|\-)?{D}+)? : {token,{float,TokenLine,list_to_float(TokenChars)}}.</pre></div> <p>The Erlang code in the "Erlang code." section is written into the output file directly after the module declaration and predefined exports declaration so it is possible to add extra exports, define imports and other attributes which are then visible in the whole file.</p> </div> <h3><a name="id63605">Regular Expressions</a></h3> <div class="REFBODY"> <p>The regular expressions allowed here is a subset of the set found in <span class="code">egrep</span> and in the AWK programming language, as defined in the book, The AWK Programming Language, by A. V. Aho, B. W. Kernighan, P. J. Weinberger. They are composed of the following characters:</p> <dl> <dt><strong><span class="code">c</span></strong></dt> <dd> <p>Matches the non-metacharacter c.</p> </dd> <dt><strong><span class="code">\c</span></strong></dt> <dd> <p>Matches the escape sequence or literal character c.</p> </dd> <dt><strong><span class="code">.</span></strong></dt> <dd> <p>Matches any character.</p> </dd> <dt><strong><span class="code">^</span></strong></dt> <dd> <p>Matches the beginning of a string.</p> </dd> <dt><strong><span class="code">$</span></strong></dt> <dd><p>Matches the end of a string.</p></dd> <dt><strong><span class="code">[abc...]</span></strong></dt> <dd> <p>Character class, which matches any of the characters <span class="code">abc...</span>. Character ranges are specified by a pair of characters separated by a <span class="code">-</span>.</p> </dd> <dt><strong><span class="code">[^abc...]</span></strong></dt> <dd> <p>Negated character class, which matches any character except <span class="code">abc...</span>.</p> </dd> <dt><strong><span class="code">r1 | r2</span></strong></dt> <dd> <p>Alternation. It matches either <span class="code">r1</span> or <span class="code">r2</span>.</p> </dd> <dt><strong><span class="code">r1r2</span></strong></dt> <dd> <p>Concatenation. It matches <span class="code">r1</span> and then <span class="code">r2</span>.</p> </dd> <dt><strong><span class="code">r+</span></strong></dt> <dd> <p>Matches one or more <span class="code">rs</span>.</p> </dd> <dt><strong><span class="code">r*</span></strong></dt> <dd> <p>Matches zero or more <span class="code">rs</span>.</p> </dd> <dt><strong><span class="code">r?</span></strong></dt> <dd> <p>Matches zero or one <span class="code">rs</span>.</p> </dd> <dt><strong><span class="code">(r)</span></strong></dt> <dd> <p>Grouping. It matches <span class="code">r</span>.</p> </dd> </dl> <p>The escape sequences allowed are the same as for Erlang strings:</p> <dl> <dt><strong><span class="code">\b</span></strong></dt> <dd><p>Backspace.</p></dd> <dt><strong><span class="code">\f</span></strong></dt> <dd><p>Form feed.</p></dd> <dt><strong><span class="code">\n</span></strong></dt> <dd><p>Newline (line feed).</p></dd> <dt><strong><span class="code">\r</span></strong></dt> <dd><p>Carriage return.</p></dd> <dt><strong><span class="code">\t</span></strong></dt> <dd><p>Tab.</p></dd> <dt><strong><span class="code">\e</span></strong></dt> <dd><p>Escape.</p></dd> <dt><strong><span class="code">\v</span></strong></dt> <dd><p>Vertical tab.</p></dd> <dt><strong><span class="code">\s</span></strong></dt> <dd><p>Space.</p></dd> <dt><strong><span class="code">\d</span></strong></dt> <dd><p>Delete.</p></dd> <dt><strong><span class="code">\ddd</span></strong></dt> <dd><p>The octal value <span class="code">ddd</span>.</p></dd> <dt><strong><span class="code">\xhh</span></strong></dt> <dd><p>The hexadecimal value <span class="code">hh</span>.</p></dd> <dt><strong><span class="code">\x{h...}</span></strong></dt> <dd><p>The hexadecimal value <span class="code">h...</span>.</p></dd> <dt><strong><span class="code">\c</span></strong></dt> <dd> <p>Any other character literally, for example <span class="code">\\</span> for backslash, <span class="code">\"</span> for <span class="code">"</span>.</p> </dd> </dl> <p>The following examples define Erlang data types:</p> <div class="example"><pre> Atoms [a-z][0-9a-zA-Z_]* Variables [A-Z_][0-9a-zA-Z_]* Floats (\+|-)?[0-9]+\.[0-9]+((E|e)(\+|-)?[0-9]+)?</pre></div> <div class="note"> <div class="label">Note</div> <div class="content"><p><p>Anchoring a regular expression with <span class="code">^</span> and <span class="code">$</span> is not implemented in the current version of Leex and just generates a parse error.</p></p></div> </div> </div> </div> <div class="footer"> <hr> <p>Copyright © 1997-2012 Ericsson AB. All Rights Reserved.</p> </div> </div> </div></body> </html>