<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=US-ASCII">
<title>Lexer API</title>
<link rel="stylesheet" href="../../../../../../../doc/src/boostbook.css" type="text/css">
<meta name="generator" content="DocBook XSL Stylesheets V1.75.0">
<link rel="home" href="../../../index.html" title="Spirit 2.4">
<link rel="up" href="../reference.html" title="Reference">
<link rel="prev" href="lex_basics.html" title="Basics">
<link rel="next" href="primitives.html" title="Token definition Primitives">
</head>
<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
<table cellpadding="2" width="100%"><tr>
<td valign="top"><img alt="Boost C++ Libraries" width="277" height="86" src="../../../../../../../boost.png"></td>
<td align="center"><a href="../../../../../../../index.html">Home</a></td>
<td align="center"><a href="../../../../../../../libs/libraries.htm">Libraries</a></td>
<td align="center"><a href="http://www.boost.org/users/people.html">People</a></td>
<td align="center"><a href="http://www.boost.org/users/faq.html">FAQ</a></td>
<td align="center"><a href="../../../../../../../more/index.htm">More</a></td>
</tr></table>
<hr>
<div class="spirit-nav">
<a accesskey="p" href="lex_basics.html"><img src="../../../../../../../doc/src/images/prev.png" alt="Prev"></a><a accesskey="u" href="../reference.html"><img src="../../../../../../../doc/src/images/up.png" alt="Up"></a><a accesskey="h" href="../../../index.html"><img src="../../../../../../../doc/src/images/home.png" alt="Home"></a><a accesskey="n" href="primitives.html"><img src="../../../../../../../doc/src/images/next.png" alt="Next"></a>
</div>
<div class="section">
<div class="titlepage"><div><div><h4 class="title">
<a name="spirit.lex.reference.lexer_api"></a><a class="link" href="lexer_api.html" title="Lexer API"> Lexer API</a>
</h4></div></div></div>
<a name="spirit.lex.reference.lexer_api.description"></a><h6>
<a name="id1191225"></a>
<a class="link" href="lexer_api.html#spirit.lex.reference.lexer_api.description">Description</a>
</h6>
<p>
The library provides a couple of free functions to make using the lexer
a snap. These functions have three forms. The first form, <code class="computeroutput"><span class="identifier">tokenize</span></code>, simplifies the usage of a standalone
lexer (without parsing). The second form, <code class="computeroutput"><span class="identifier">tokenize_and_parse</span></code>,
combines a lexer step with parsing on the token level (without a skipper).
The third, <code class="computeroutput"><span class="identifier">tokenize_and_phrase_parse</span></code>,
works on the token level as well, but additionally employs a skip parser.
The latter two versions can take in attributes by reference that will hold
the parsed values on a successful parse.
</p>
<a name="spirit.lex.reference.lexer_api.header"></a><h6>
<a name="id1191263"></a>
<a class="link" href="lexer_api.html#spirit.lex.reference.lexer_api.header">Header</a>
</h6>
<pre class="programlisting"><span class="comment">// forwards to <boost/spirit/home/lex/tokenize_and_parse.hpp>
</span><span class="preprocessor">#include</span> <span class="special"><</span><span class="identifier">boost</span><span class="special">/</span><span class="identifier">spirit</span><span class="special">/</span><span class="identifier">include</span><span class="special">/</span><span class="identifier">lex_tokenize_and_parse</span><span class="special">.</span><span class="identifier">hpp</span><span class="special">></span>
</pre>
<p>
For variadic attributes:
</p>
<pre class="programlisting"><span class="comment">// forwards to <boost/spirit/home/lex/tokenize_and_parse_attr.hpp>
</span><span class="preprocessor">#include</span> <span class="special"><</span><span class="identifier">boost</span><span class="special">/</span><span class="identifier">spirit</span><span class="special">/</span><span class="identifier">include</span><span class="special">/</span><span class="identifier">lex_tokenize_and_parse_attr</span><span class="special">.</span><span class="identifier">hpp</span><span class="special">></span>
</pre>
<p>
The variadic attributes version of the API allows one or more attributes
to be passed into the API functions. The functions taking two or more attributes
are usable when the parser expression is a <a class="link" href="../../qi/reference/operator/sequence.html" title="Sequence (a >> b)">Sequence</a>
only. In this case each of the attributes passed have to match the corresponding
part of the sequence.
</p>
<p>
Also, see <a class="link" href="../../structure/include.html" title="Include">Include Structure</a>.
</p>
<a name="spirit.lex.reference.lexer_api.namespace"></a><h6>
<a name="id1191402"></a>
<a class="link" href="lexer_api.html#spirit.lex.reference.lexer_api.namespace">Namespace</a>
</h6>
<div class="informaltable"><table class="table">
<colgroup><col></colgroup>
<thead><tr><th>
<p>
Name
</p>
</th></tr></thead>
<tbody>
<tr><td>
<p>
<code class="computeroutput"><span class="identifier">boost</span><span class="special">::</span><span class="identifier">spirit</span><span class="special">::</span><span class="identifier">lex</span><span class="special">::</span><span class="identifier">tokenize</span></code>
</p>
</td></tr>
<tr><td>
<p>
<code class="computeroutput"><span class="identifier">boost</span><span class="special">::</span><span class="identifier">spirit</span><span class="special">::</span><span class="identifier">lex</span><span class="special">::</span><span class="identifier">tokenize_and_parse</span></code>
</p>
</td></tr>
<tr><td>
<p>
<code class="computeroutput"><span class="identifier">boost</span><span class="special">::</span><span class="identifier">spirit</span><span class="special">::</span><span class="identifier">lex</span><span class="special">::</span><span class="identifier">tokenize_and_phrase_parse</span></code>
</p>
</td></tr>
<tr><td>
<p>
<code class="computeroutput"><span class="identifier">boost</span><span class="special">::</span><span class="identifier">spirit</span><span class="special">::</span><span class="identifier">qi</span><span class="special">::</span><span class="identifier">skip_flag</span><span class="special">::</span><span class="identifier">postskip</span></code>
</p>
</td></tr>
<tr><td>
<p>
<code class="computeroutput"><span class="identifier">boost</span><span class="special">::</span><span class="identifier">spirit</span><span class="special">::</span><span class="identifier">qi</span><span class="special">::</span><span class="identifier">skip_flag</span><span class="special">::</span><span class="identifier">dont_postskip</span></code>
</p>
</td></tr>
</tbody>
</table></div>
<a name="spirit.lex.reference.lexer_api.synopsis"></a><h6>
<a name="id1191656"></a>
<a class="link" href="lexer_api.html#spirit.lex.reference.lexer_api.synopsis">Synopsis</a>
</h6>
<p>
The <code class="computeroutput"><span class="identifier">tokenize</span></code> function is
one of the main lexer API functions. It simplifies using a lexer to tokenize
a given input sequence. It's main purpose is to use the lexer to tokenize
all the input.
</p>
<p>
Both functions take a pair of iterators spanning the underlying input stream
to scan, the lexer object (built from the token definitions), and an (optional)
functor being called for each of the generated tokens. If no function object
<code class="computeroutput"><span class="identifier">f</span></code> is given, the generated
tokens will be discarded.
</p>
<p>
The functions return <code class="computeroutput"><span class="keyword">true</span></code>
if the scanning of the input succeeded (the given input sequence has been
successfully matched by the given token definitions).
</p>
<p>
The argument <code class="computeroutput"><span class="identifier">f</span></code> is expected
to be a function (callable) object taking a single argument of the token
type and returning a bool, indicating whether the tokenization should be
canceled. If it returns <code class="computeroutput"><span class="keyword">false</span></code>
the function <code class="computeroutput"><span class="identifier">tokenize</span></code> will
return <code class="computeroutput"><span class="keyword">false</span></code> as well.
</p>
<p>
The <code class="computeroutput"><span class="identifier">initial_state</span></code> argument
forces lexing to start with the given lexer state. If this is omitted lexing
starts in the <code class="computeroutput"><span class="string">"INITIAL"</span></code>
state.
</p>
<pre class="programlisting"><span class="keyword">template</span> <span class="special"><</span><span class="keyword">typename</span> <span class="identifier">Iterator</span><span class="special">,</span> <span class="keyword">typename</span> <span class="identifier">Lexer</span><span class="special">></span>
<span class="keyword">inline</span> <span class="keyword">bool</span>
<span class="identifier">tokenize</span><span class="special">(</span>
<span class="identifier">Iterator</span><span class="special">&</span> <span class="identifier">first</span>
<span class="special">,</span> <span class="identifier">Iterator</span> <span class="identifier">last</span>
<span class="special">,</span> <span class="identifier">Lexer</span> <span class="keyword">const</span><span class="special">&</span> <span class="identifier">lex</span>
<span class="special">,</span> <span class="keyword">typename</span> <span class="identifier">Lexer</span><span class="special">::</span><span class="identifier">char_type</span> <span class="keyword">const</span><span class="special">*</span> <span class="identifier">initial_state</span> <span class="special">=</span> <span class="number">0</span><span class="special">);</span>
<span class="keyword">template</span> <span class="special"><</span><span class="keyword">typename</span> <span class="identifier">Iterator</span><span class="special">,</span> <span class="keyword">typename</span> <span class="identifier">Lexer</span><span class="special">,</span> <span class="keyword">typename</span> <span class="identifier">F</span><span class="special">></span>
<span class="keyword">inline</span> <span class="keyword">bool</span>
<span class="identifier">tokenize</span><span class="special">(</span>
<span class="identifier">Iterator</span><span class="special">&</span> <span class="identifier">first</span>
<span class="special">,</span> <span class="identifier">Iterator</span> <span class="identifier">last</span>
<span class="special">,</span> <span class="identifier">Lexer</span> <span class="keyword">const</span><span class="special">&</span> <span class="identifier">lex</span>
<span class="special">,</span> <span class="identifier">F</span> <span class="identifier">f</span>
<span class="special">,</span> <span class="keyword">typename</span> <span class="identifier">Lexer</span><span class="special">::</span><span class="identifier">char_type</span> <span class="keyword">const</span><span class="special">*</span> <span class="identifier">initial_state</span> <span class="special">=</span> <span class="number">0</span><span class="special">);</span>
</pre>
<p>
The <code class="computeroutput"><span class="identifier">tokenize_and_parse</span></code>
function is one of the main lexer API functions. It simplifies using a
lexer as the underlying token source while parsing a given input sequence.
</p>
<p>
The functions take a pair of iterators spanning the underlying input stream
to parse, the lexer object (built from the token definitions) and a parser
object (built from the parser grammar definition). Additionally they may
take the attributes for the parser step.
</p>
<p>
The function returns <code class="computeroutput"><span class="keyword">true</span></code>
if the parsing succeeded (the given input sequence has been successfully
matched by the given grammar).
</p>
<pre class="programlisting"><span class="keyword">template</span> <span class="special"><</span><span class="keyword">typename</span> <span class="identifier">Iterator</span><span class="special">,</span> <span class="keyword">typename</span> <span class="identifier">Lexer</span><span class="special">,</span> <span class="keyword">typename</span> <span class="identifier">ParserExpr</span><span class="special">></span>
<span class="keyword">inline</span> <span class="keyword">bool</span>
<span class="identifier">tokenize_and_parse</span><span class="special">(</span>
<span class="identifier">Iterator</span><span class="special">&</span> <span class="identifier">first</span>
<span class="special">,</span> <span class="identifier">Iterator</span> <span class="identifier">last</span>
<span class="special">,</span> <span class="identifier">Lexer</span> <span class="keyword">const</span><span class="special">&</span> <span class="identifier">lex</span>
<span class="special">,</span> <span class="identifier">ParserExpr</span> <span class="keyword">const</span><span class="special">&</span> <span class="identifier">expr</span><span class="special">)</span>
<span class="keyword">template</span> <span class="special"><</span><span class="keyword">typename</span> <span class="identifier">Iterator</span><span class="special">,</span> <span class="keyword">typename</span> <span class="identifier">Lexer</span><span class="special">,</span> <span class="keyword">typename</span> <span class="identifier">ParserExpr</span>
<span class="special">,</span> <span class="keyword">typename</span> <span class="identifier">Attr1</span><span class="special">,</span> <span class="keyword">typename</span> <span class="identifier">Attr2</span><span class="special">,</span> <span class="special">...,</span> <span class="keyword">typename</span> <span class="identifier">AttrN</span><span class="special">></span>
<span class="keyword">inline</span> <span class="keyword">bool</span>
<span class="identifier">tokenize_and_parse</span><span class="special">(</span>
<span class="identifier">Iterator</span><span class="special">&</span> <span class="identifier">first</span>
<span class="special">,</span> <span class="identifier">Iterator</span> <span class="identifier">last</span>
<span class="special">,</span> <span class="identifier">Lexer</span> <span class="keyword">const</span><span class="special">&</span> <span class="identifier">lex</span>
<span class="special">,</span> <span class="identifier">ParserExpr</span> <span class="keyword">const</span><span class="special">&</span> <span class="identifier">expr</span>
<span class="special">,</span> <span class="identifier">Attr1</span> <span class="keyword">const</span><span class="special">&</span> <span class="identifier">attr1</span><span class="special">,</span> <span class="identifier">Attr2</span> <span class="keyword">const</span><span class="special">&</span> <span class="identifier">attr2</span><span class="special">,</span> <span class="special">...,</span> <span class="identifier">AttrN</span> <span class="keyword">const</span><span class="special">&</span> <span class="identifier">attrN</span><span class="special">);</span>
</pre>
<p>
The functions <code class="computeroutput"><span class="identifier">tokenize_and_phrase_parse</span></code>
take a pair of iterators spanning the underlying input stream to parse,
the lexer object (built from the token definitions) and a parser object
(built from the parser grammar definition). The additional skipper parameter
will be used as the skip parser during the parsing process. Additionally
they may take the attributes for the parser step.
</p>
<p>
The function returns <code class="computeroutput"><span class="keyword">true</span></code>
if the parsing succeeded (the given input sequence has been successfully
matched by the given grammar).
</p>
<pre class="programlisting"><span class="keyword">template</span> <span class="special"><</span><span class="keyword">typename</span> <span class="identifier">Iterator</span><span class="special">,</span> <span class="keyword">typename</span> <span class="identifier">Lexer</span><span class="special">,</span> <span class="keyword">typename</span> <span class="identifier">ParserExpr</span>
<span class="special">,</span> <span class="keyword">typename</span> <span class="identifier">Skipper</span><span class="special">></span>
<span class="keyword">inline</span> <span class="keyword">bool</span>
<span class="identifier">tokenize_and_phrase_parse</span><span class="special">(</span>
<span class="identifier">Iterator</span><span class="special">&</span> <span class="identifier">first</span>
<span class="special">,</span> <span class="identifier">Iterator</span> <span class="identifier">last</span>
<span class="special">,</span> <span class="identifier">Lexer</span> <span class="keyword">const</span><span class="special">&</span> <span class="identifier">lex</span>
<span class="special">,</span> <span class="identifier">ParserExpr</span> <span class="keyword">const</span><span class="special">&</span> <span class="identifier">expr</span>
<span class="special">,</span> <span class="identifier">Skipper</span> <span class="keyword">const</span><span class="special">&</span> <span class="identifier">skipper</span>
<span class="special">,</span> <span class="identifier">BOOST_SCOPED_ENUM</span><span class="special">(</span><span class="identifier">skip_flag</span><span class="special">)</span> <span class="identifier">post_skip</span> <span class="special">=</span> <span class="identifier">skip_flag</span><span class="special">::</span><span class="identifier">postskip</span><span class="special">);</span>
<span class="keyword">template</span> <span class="special"><</span><span class="keyword">typename</span> <span class="identifier">Iterator</span><span class="special">,</span> <span class="keyword">typename</span> <span class="identifier">Lexer</span><span class="special">,</span> <span class="keyword">typename</span> <span class="identifier">ParserExpr</span>
<span class="special">,</span> <span class="keyword">typename</span> <span class="identifier">Skipper</span><span class="special">,</span> <span class="keyword">typename</span> <span class="identifier">Attr1</span><span class="special">,</span> <span class="keyword">typename</span> <span class="identifier">Attr2</span><span class="special">,</span> <span class="special">...,</span> <span class="keyword">typename</span> <span class="identifier">AttrN</span><span class="special">></span>
<span class="keyword">inline</span> <span class="keyword">bool</span>
<span class="identifier">tokenize_and_phrase_parse</span><span class="special">(</span>
<span class="identifier">Iterator</span><span class="special">&</span> <span class="identifier">first</span>
<span class="special">,</span> <span class="identifier">Iterator</span> <span class="identifier">last</span>
<span class="special">,</span> <span class="identifier">Lexer</span> <span class="keyword">const</span><span class="special">&</span> <span class="identifier">lex</span>
<span class="special">,</span> <span class="identifier">ParserExpr</span> <span class="keyword">const</span><span class="special">&</span> <span class="identifier">expr</span>
<span class="special">,</span> <span class="identifier">Skipper</span> <span class="keyword">const</span><span class="special">&</span> <span class="identifier">skipper</span>
<span class="special">,</span> <span class="identifier">Attr1</span> <span class="keyword">const</span><span class="special">&</span> <span class="identifier">attr1</span><span class="special">,</span> <span class="identifier">Attr2</span> <span class="keyword">const</span><span class="special">&</span> <span class="identifier">attr2</span><span class="special">,</span> <span class="special">...,</span> <span class="identifier">AttrN</span> <span class="keyword">const</span><span class="special">&</span> <span class="identifier">attrN</span><span class="special">);</span>
<span class="keyword">template</span> <span class="special"><</span><span class="keyword">typename</span> <span class="identifier">Iterator</span><span class="special">,</span> <span class="keyword">typename</span> <span class="identifier">Lexer</span><span class="special">,</span> <span class="keyword">typename</span> <span class="identifier">ParserExpr</span>
<span class="special">,</span> <span class="keyword">typename</span> <span class="identifier">Skipper</span><span class="special">,</span> <span class="keyword">typename</span> <span class="identifier">Attr1</span><span class="special">,</span> <span class="keyword">typename</span> <span class="identifier">Attr2</span><span class="special">,</span> <span class="special">...,</span> <span class="keyword">typename</span> <span class="identifier">AttrN</span><span class="special">></span>
<span class="keyword">inline</span> <span class="keyword">bool</span>
<span class="identifier">tokenize_and_phrase_parse</span><span class="special">(</span>
<span class="identifier">Iterator</span><span class="special">&</span> <span class="identifier">first</span>
<span class="special">,</span> <span class="identifier">Iterator</span> <span class="identifier">last</span>
<span class="special">,</span> <span class="identifier">Lexer</span> <span class="keyword">const</span><span class="special">&</span> <span class="identifier">lex</span>
<span class="special">,</span> <span class="identifier">ParserExpr</span> <span class="keyword">const</span><span class="special">&</span> <span class="identifier">expr</span>
<span class="special">,</span> <span class="identifier">Skipper</span> <span class="keyword">const</span><span class="special">&</span> <span class="identifier">skipper</span>
<span class="special">,</span> <span class="identifier">BOOST_SCOPED_ENUM</span><span class="special">(</span><span class="identifier">skip_flag</span><span class="special">)</span> <span class="identifier">post_skip</span>
<span class="special">,</span> <span class="identifier">Attr1</span> <span class="keyword">const</span><span class="special">&</span> <span class="identifier">attr1</span><span class="special">,</span> <span class="identifier">Attr2</span> <span class="keyword">const</span><span class="special">&</span> <span class="identifier">attr2</span><span class="special">,</span> <span class="special">...,</span> <span class="identifier">AttrN</span> <span class="keyword">const</span><span class="special">&</span> <span class="identifier">attrN</span><span class="special">);</span>
</pre>
<p>
The maximum number of supported arguments is limited by the preprocessor
constant <code class="computeroutput"><span class="identifier">SPIRIT_ARGUMENTS_LIMIT</span></code>.
This constant defaults to the value defined by the preprocessor constant
<code class="computeroutput"><span class="identifier">PHOENIX_LIMIT</span></code> (which in
turn defaults to <code class="computeroutput"><span class="number">10</span></code>).
</p>
<div class="note"><table border="0" summary="Note">
<tr>
<td rowspan="2" align="center" valign="top" width="25"><img alt="[Note]" src="../../../images/note.png"></td>
<th align="left">Note</th>
</tr>
<tr><td align="left" valign="top"><p>
The variadic function with two or more attributes internally combine
references to all passed attributes into a <code class="computeroutput"><span class="identifier">fusion</span><span class="special">::</span><span class="identifier">vector</span></code>
and forward this as a combined attribute to the corresponding one attribute
function.
</p></td></tr>
</table></div>
<p>
The <code class="computeroutput"><span class="identifier">tokenize_and_phrase_parse</span></code>
functions not taking an explicit <code class="computeroutput"><span class="identifier">skip_flag</span></code>
as one of their arguments invoke the passed skipper after a successful
match of the parser expression. This can be inhbited by using the other
versions of that function while passing <code class="computeroutput"><span class="identifier">skip_flag</span><span class="special">::</span><span class="identifier">dont_postskip</span></code>
to the corresponding argument.
</p>
<a name="spirit.lex.reference.lexer_api.template_parameters"></a><h6>
<a name="id1194012"></a>
<a class="link" href="lexer_api.html#spirit.lex.reference.lexer_api.template_parameters">Template
parameters</a>
</h6>
<div class="informaltable"><table class="table">
<colgroup>
<col>
<col>
</colgroup>
<thead><tr>
<th>
<p>
Parameter
</p>
</th>
<th>
<p>
Description
</p>
</th>
</tr></thead>
<tbody>
<tr>
<td>
<p>
<code class="computeroutput"><span class="identifier">Iterator</span></code>
</p>
</td>
<td>
<p>
<a href="http://www.sgi.com/tech/stl/ForwardIterator.html" target="_top"><code class="computeroutput"><span class="identifier">ForwardIterator</span></code></a> pointing
to the underlying input sequence to parse.
</p>
</td>
</tr>
<tr>
<td>
<p>
<code class="computeroutput"><span class="identifier">Lexer</span></code>
</p>
</td>
<td>
<p>
A lexer (token definition) object.
</p>
</td>
</tr>
<tr>
<td>
<p>
<code class="computeroutput"><span class="identifier">F</span></code>
</p>
</td>
<td>
<p>
A function object called for each generated token.
</p>
</td>
</tr>
<tr>
<td>
<p>
<code class="computeroutput"><span class="identifier">ParserExpr</span></code>
</p>
</td>
<td>
<p>
An expression that can be converted to a Qi parser.
</p>
</td>
</tr>
<tr>
<td>
<p>
<code class="computeroutput"><span class="identifier">Skipper</span></code>
</p>
</td>
<td>
<p>
Parser used to skip white spaces.
</p>
</td>
</tr>
<tr>
<td>
<p>
<code class="computeroutput"><span class="identifier">Attr1</span></code>, <code class="computeroutput"><span class="identifier">Attr2</span></code>, ..., <code class="computeroutput"><span class="identifier">AttrN</span></code>
</p>
</td>
<td>
<p>
One or more attributes.
</p>
</td>
</tr>
</tbody>
</table></div>
</div>
<table xmlns:rev="http://www.cs.rpi.edu/~gregod/boost/tools/doc/revision" width="100%"><tr>
<td align="left"></td>
<td align="right"><div class="copyright-footer">Copyright © 2001-2010 Joel de Guzman, Hartmut Kaiser<p>
Distributed under the Boost Software License, Version 1.0. (See accompanying
file LICENSE_1_0.txt or copy at <a href="http://www.boost.org/LICENSE_1_0.txt" target="_top">http://www.boost.org/LICENSE_1_0.txt</a>)
</p>
</div></td>
</tr></table>
<hr>
<div class="spirit-nav">
<a accesskey="p" href="lex_basics.html"><img src="../../../../../../../doc/src/images/prev.png" alt="Prev"></a><a accesskey="u" href="../reference.html"><img src="../../../../../../../doc/src/images/up.png" alt="Up"></a><a accesskey="h" href="../../../index.html"><img src="../../../../../../../doc/src/images/home.png" alt="Home"></a><a accesskey="n" href="primitives.html"><img src="../../../../../../../doc/src/images/next.png" alt="Next"></a>
</div>
</body>
</html>
