<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 3.2//EN">
<html>
<head>
<title>Notes for using the ANTLR Python Code Generator</title>
</head>
<body bgcolor="#FFFFFF">
<h1><strong>Python Code Generator for ANTLR 2.7.7</strong></h1>
<p>As of ANTLR 2.7.5, you can generate your Lexers,
Parsers and TreeParsers in Python. This feature extends the benefits of
ANTLR's predicated-LL(k) parsing technology to the Python language and
platform.</p>
<p>To be able to build and use the Python language Lexers, Parsers and
TreeParsers, you will need to have the ANTLR Python runtime library
installed in your Python path. The Python runtime model is based on the
existing runtime model for <a href="runtime.html">Java</a> and is thus
immediately familiar. The Python runtime and the Java runtime are very
similar although there a number of subtle (and not so subtle)
differences. Some of these result from differences in the respective
runtime environments.</p>
<p>ANTLR Python support was contributed (and is to be maintained) by
Wolfgang Haefelinger and Marq Kole.</p>
<h2><a name="#building">Building the ANTLR Python Runtime</a></h2>
<p>The ANTLR Python runtime source and build files are completely
integrated in the ANTLR build process.The ANTLR runtime support module
for Python is located in the <b><code>lib/python</code></b> subdirectory
of the ANTLR distribution. Installation of the Python runtime support
is enabled automatically if Python can be found on your system by the
<tt>configure</tt> script.</p>
<p>With Python support enabled the current distribution will look for
the presence of a python executable of version 2.2 or higher. If it has
found such a beast, it will generate and install the ANTLR Python
runtime as part of the overall ANTLR building and installation
process.</p>
<p>If the python distribution you are using is at an unusual location,
perhaps because you are using a local installation instead of a
system-wide one, you can provide the location of that python executable
using the <tt>--with-python=<path></tt> option for the configure
script, for instance:</p>
<blockquote>
<pre>
./configure --with-python=$HOME/bin/python2.3
</pre>
</blockquote>
<p>Also, if the python executable is at a regular location, but has a
name that differs from "python", you can specify the correct
name through the <tt>--with-python=<path></tt>, as shown above, or through
environment variable <tt>$PYTHON</tt></p>
<blockquote>
<pre>
PYTHON=python2.3
export PYTHON
./configure
</pre>
</blockquote>
<p>All the example grammars for the ANTLR Python runtime are built when
ANTLR itself is built. They can be run in one go by running <tt>make
test</tt> in the same directory where you ran the configure script in
the ANTLR distribution. So after you've run <tt>configure</tt> you can
do:</p>
<blockquote>
<pre>
# Build ANTLR and all examples
make
# Run them
make test
# Install everything
make install
</pre>
</blockquote>
<p>Note that <tt>make install</tt> will not add the ANTLR Python
runtime (i.e. antlr.py) to your Python installation but rather install
antlr.py in <tt>${prefix}/lib</tt>. To be able to use antlr.py you
would need to adjust Python's <tt>sys.path<tt>.</p>
<p>However, there a script is provided that let's you easily add antlr.py
as module to your Python installation. After installation just run</p>
<blockquote>
<pre>
${prefix}/sbin/pyantlr.sh install
</pre>
</blockquote>
<p>Note that usually you need to be superuser in order to succeed. Also
note that you can run this command later at any time again, for
example, if you have a second Python installation etc. Just make sure
that <tt>python</tt> is in your $PATH when running pyantlr.sh.</p>
<p>Note further that you can also do this to install ANTLR Python
runtime immediatly after having called <tt>./configure</tt>:</p>
<blockquote>
<pre>
scripts/pyantlr.sh install
</pre>
</blockquote>
<h2><a name="#codegen">Specifying Code Generation</a></h2>
<p>You can instruct ANTLR to generate your Lexers, Parsers and
TreeParsers using the Python code generator by adding the following
entry to the global options section at the beginning of your grammar
file.</p>
<blockquote>
<pre>
{
language="Python";
}
</pre>
</blockquote>
<p>After that things are pretty much the same as in the default
<b>java</b> code generation mode. See the examples in
<code>examples/python</code> for some illustrations.</p>
<p>One particular issue that is worth mentioning is the handling of
comments in ANTLR Python. Java, C++, and C# all use the same lexical
structures to define comments: <code>//</code> for single-line
comments, and <code>/* ... */</code> for block comments. Unfortunately,
Python does not handle comments this way. It only knows about
single-line comments, and these start off with a <code>#</code>
symbol.</p>
<p>Normally, all comments outside of actions are actually comments in
the ANTLR input language. These comments, and that is both block
comments and single-line comments are translated into Python
single-line comments.</p>
<p>Secondly, all comments inside actions should be comments in the
target language, Python in this case. Unfortunately, if the actions
contain ANTLR actions, such as <code>$getText</code>, the code
generator seems to choke on Python comments as the <code>#</code> sign
is also used in tree construction. The solution is to use Java/C++-style
comments in all actions; these will be translated into Python comments
by the ANTLR as it checks these actions for the presence of predefined
action symbols such as <code>$getText</code>.</p>
<p>So, as a general issue: all comments in an ANTLR grammar for the
Python target should be in Java/C++ style, not in Python style.</p>
<h2><a name="#sections">Python-Specific ANTLR Sections</a></h2>
<ul>
<li>
<b>header - specify additional <code>import</code> directives</b>
<p>You can instruct the ANTLR Python code generator to import
additional Python packages in your generated
Lexer/Parser/TreeParser by adding code to the header section which
must be the first section at the beginning of your ANTLR grammar
file, apart from any other header sections.</p>
<blockquote>
<pre>
header {
import os, sys
}
</pre>
</blockquote>
</li>
<li>
<b>header "__init__" - specify additional code in the
<code>__init__</code> method</b>
<p>You can instruct the ANTLR Python code generator to include
additional Python code in your generated Lexer/Parser/TreeParser by
adding code to the <code>init</code> header section which must
be the first section at the beginning of your ANTLR grammar file,
apart from any other header sections. The code in the header is
appended to the end of the <code>__init__</code> method.</p>
<blockquote>
<pre>
header "__init__" {
self.message = "This is the default message"
}
</pre>
</blockquote>
<p>If your grammar file contains both a Lexer and a Parser (or any
other multiple of definitions), the code in the
<code>__init__</code> header will be reproduced in the
<code>__init__</code> methods of all of these definitions without
change. If you really want to update only one of the definitions,
for instance, the <code>__init__</code> method of the Lexer class
you are creating, use</p>
<blockquote>
<pre>
header "<LexerGrammar>.__init__" {
self.message = "This is the default message"
}
</pre>
</blockquote>
<p>where <tt><LexerGrammar></tt> is the name of the Lexer
grammar. The same construction also works with the Parsers and
TreeParsers, of course.</p>
<p>In the case both a generic init header and a grammar-specific
header are present, the grammar-specific one will override the
generic one.</p>
</li>
<li>
<b>header "__main__" - specify additional code after the class
definition</b>
<p>You can instruct the ANTLR Python code generator to add
additional Python code at the end of your generated
Lexer/Parser/TreeParser, so after the class definition itself by
adding code to the <code>__main__</code> header section which must
be the first section at the beginning of your ANTLR grammar file,
apart from any other header sections.</p>
<blockquote>
<pre>
header "__main__" {
print "You cannot execute this file!"
}
</pre>
</blockquote>
<p>If your grammar file contains both a Lexer and a Parser (or any
other multiple of definitions), the code in the <code>__main__</code>
header will be reproduced at the end of all of the generated class
definitions. If you really want to add code after only one of the
definitions, for instance, after the Lexer class, use</p>
<blockquote>
<pre>
header "<LexerGrammar>.__main__" {
print "You cannot execute this file!"
}
</pre>
</blockquote>
<p>where <tt><LexerGrammar></tt> is the name of the Lexer
grammar. The same construction also works with the Parsers and
TreeParsers, of course.</p>
<p>In the case both a generic init header and a grammar-specific
header are present, the grammar-specific one will override the
generic one. If no <code>__main__</code> headers are present and the
grammar is for a Lexer, automated test code for that lexer is
automatically added at the end of the generated module. This can be
prevented by providing an empty <code>__main__</code> header. In the
latter case it is good practise to provide a comment explaining why
an empty header is present.</p>
<blockquote>
<pre>
header "<LexerGrammar>.__main__" {
// Empty main header to prevent automatic test code from being added
// to the generated lexer module.
}
</pre>
</blockquote>
<p>This automated test code can be executed by running Python with
the generated lexer file (<tt><LexerGrammar>.py</tt>where
<tt><LexerGrammar></tt> is the name of the Lexer grammar) and
providing some test input on <tt>stdin</tt>:</p>
<blockquote>
<pre>
python <LexerGrammar>.py < test.in
</pre>
</blockquote>
</li>
</ul>
<h2><a name="#options">Python-Specific ANTLR Options</a></h2>
<ul>
<li>
<b>className - change the default name of the generated class</b>
<p></p>
<blockquote>
<pre>
options {
className="Scanner";
}
</pre>
</blockquote>
<p>If you are using the <tt>className</tt> option conjunction with the
Python specific header options, there will be no collisions. The
<tt>className</tt> option changes the class name, while the
<code>main</code> headers require the use of the grammar name which
will become the module name after code generation.</p>
<blockquote>
<pre>
header "ParrotSketch.init" {
self.state = JohnCleese.select("dead", "pushing up daisies", \
"no longer", "in Parrot Heaven")
print "This parrot is", self.state
}
class ParrotSketch extends Lexer;
options {
className="Scanner";
}
</pre>
</blockquote>
</li>
</ul>
<h2><a name="#template">A Template Python ANTLR Grammar File</a></h2>
<p>As the handling of modules &emdash; packages in Java speak &emdash;
in Python differs from that in Java, the current approach in ANTLR to
call both the file and the class they contain after the name of the
grammar is kind of awkward. Instead, a different approach is chosen that
better reflects the handling of modules in Python. The name of the
generated Python file is still derived from the name of the grammar, but
the name of the class is fixed to the particular kind of grammar. A
lexer grammar will be used to generate a class <tt>Lexer</tt>; a parser
grammar will be used to generate a class <tt>Parser</tt>; and a
treeparser grammar will be used to generate a class <tt>Walker</tt>.</p>
<blockquote>
<pre>
header {
// gets inserted in the Python source file before any generated
// declarations
...
}
header "__init__" {
// gets inserted in the __init__ method of each of the generated Python
// classes
...
}
header "MyParser.__init__" {
// gets inserted in the __init__ method of the generated Python class
// for the MyParser grammar
...
}
header "__main__" {
// gets inserted at the end of each of the generated Python files in an
// indented section preceeded by the conditional:
// if __name__ == "__main__":
...
}
header "MyLexer.__init__" {
// gets inserted at the end of the generated Python file for the MyLexer
// grammar in an indented section preceeded by the conditional:
// if __name__ == "__main__":
// and preventing the insertion of automatic test code in the same place.
...
}
options {
language = "Python";
}
{
// global code stuff that will be included in the 'MyParser.py' source
// file just before the 'Parser' class below
...
}
class MyParser extends Parser;
options {
exportVocab=My;
}
{
// additional methods and members for the generated 'Parser' class
...
}
... generated RULES go here ...
{
// global code stuff that will be included in the 'MyLexer' source file
// just before the 'Lexer' class below
...
}
class MyLexer extends Lexer;
options {
exportVocab=My;
}
{
// additional methods and members for the generated 'Lexer' class
...
}
... generated RULES go here ...
{
// global code stuff that will be included in the 'MyTreeParser' source
// file just before the 'Walker' class below
...
}
class MyTreeParser extends TreeParser;
options {
exportVocab=My;
}
{
// additional methods and members for the generated 'Walker' class
...
}
... generated RULES go here ...
</pre>
<p>Version number in parentheses shows the tool version used to
develop and test. It may work with older versions as well. Python 2.2
or better is required as some recent Python features (like
<tt>super()</tt> for example) are being used.</p>
<h2><a name="#notes">More notes on using ANTLR Python</a></h2>
<ul>
<li>
<p>The API of the generated lexers, parsers, and treeparsers is
supposed to be similar to the Java ones. However, calling a lexer
is somewhat simplified:</p>
<blockquote>
<pre>
### class "calcLexer extends Lexer" will generate python
### module "calcLexer" with class "Lexer".
import calcLexer
### read from stdin ..
L = calcLexer.Lexer()
### read from file "test.in" ..
L = calcLexer.Lexer("test.in")
### open a file and read from it ..
f = file("test.in", "r")
L = calcLexer.Lexer(f)
### this works of course as well
import sys
L = calcLexer.Lexer(sys.stdin)
### use a shared input state
L1 = calcLexer.Lexer(...)
state = L1.inputState
L2 = calcLexer.Lexer(state)
</pre>
</blockquote>
</li>
<li>
<p>The loop for the lexer to retrieve token by token can be written
as:</p>
<blockquote>
<pre>
lexer = calcLexer.Lexer() ### create a lexer for calculator
for token in lexer:
## do something with token
print token
</pre>
</blockquote>
or even:
<blockquote>
<pre>
for token in calcLexer.Lexer(): ### create a lexer for calculator
## do something with token
print token
</pre>
</blockquote>
<p>As an iterator is available for all <tt>TokenStreams</tt>, you
can apply the same technique with a <tt>TokenStreamSelector</tt>.</p>
</li>
<li>
<p>However, writing this particular lexer loop is rarely necessary as it
is generated by default in each generated lexer. Just run:</p>
<blockquote>
<pre>
python calcLexer.py < calc.in
</pre>
</blockquote>
to test the generated lexer.<br>
</li>
<li>
<p>Symbolic token number, table of literals bitsets and bitset data
functions are generated on file (module) scope instead of class
scope. For example:</p>
<blockquote>
<pre>
import calcLexer # import calc lexer module
calcLexer.EOF_TYPE # prints 1
calcLexer.literals # { ';': 11, 'end': 12, 'begin': 10 }
</pre>
</blockquote>
</li>
<li>
<p>Comments in action should be in Java/C++ formats, ie. <tt>//</tt>
and <tt>/* ... */</tt> are valid comments. However, make sure
that you put a comment before or after a statement, but not
within. For example, this will not work:</p>
<blockquote>
<pre>
x = /* one */ 1
</pre>
</blockquote>
<p>The reason is that Python only supports single-line comments. Such
a Python comment skips everything till end-of-line. Therefore in
the translation of the comment a newline will be introduced on
reaching <tt>*/</tt>. The code above would result in the following
Python code in the generated file:</p>
<blockquote>
<pre>
x = # one
1
</pre>
</blockquote>
<p>which is probably not what you want.</p>
</li>
<li>The Lexer actions <tt>$newline</tt>, <tt>$nl</tt> and
<tt>$skip</tt> have been introduced as language independent
shortcuts for calling <tt>self.newline()</tt> (<tt>$newline</tt>,
<tt>$nl</tt>) and <tt>_ttype = SKIP</tt> (<tt>$skip</tt>).<br>
</li>
<li>
<p>In Python arguments to function and method calls do not have a
declared type. Also, functionns and methdos do not have to declare
a return type. If you want to pass a value to a rule in your grammar,
you can do so by providing simply the name of a variable.</p>
<blockquote>
<pre>
ident [symtable]
: ( 'a'..'z' | '0'..'9' )+
;
</pre>
</blockquote>
<p>Similarly, is you want a rule to pass a return value, you do not
have to provide a type either. It is possible to provide a default
value.</p>
<blockquote>
<pre>
sign returns [isPos = False]
: '-' { /* default value is OK */ }
| '+' { isPos = True }
;
</pre>
</blockquote>
</li>
<li>
<p>The <tt>__init__</tt> method of the generated Lexer, Parser, or
TreeParser has the following heading:</p>
<blockquote>
<pre>
def __init__(self, *args, **kwargs):
...
</pre>
</blockquote>
<p>So if you need to pass special arguments to your generated class,
you can use the <tt>**kwargs</tt> to check for a particular keyword
argument, irrespective of any non-keyword arguments that you did
provide. So if you have a <tt>TokenStreamSelector</tt> that you want
to access locally, you can pass it to the Lexer in the following
call:</p>
<blockquote>
<pre>
MySpecialLexer.Lexer(sys.stdin, selector=TokenStreamSelector())
</pre>
</blockquote>
<p>while in the <tt>__init__</tt> header of this particular grammar
you can specify the handling of the <tt>selector</tt> keyword
argument in the following way:</p>
<blockquote>
<pre>
header "MyParser.__init__" {
self.selector = None
if kwargs.has_key("selector"):
self.selector = kwargs["selector"]
assert(isinstance(self.selector, TokenStreamSelector))
}
</pre>
</blockquote>
</li>
<li>
<p>Because of limitations in the lexer of the ANTLR compiler
generator itself, you cannot use single quoted strings of more than
one character in your Python code.<br>
So if you use a Python string like <tt>'wink, wink, nudge,
nudge'</tt> in one of your actions, ANTLR will give a parse error
when you try to compile this grammar. Instead you should use double
quotes: <tt>"wink, wink, nudge, nudge"</tt>.</p>
</li>
<li>
<p>Unicode is supported but it's easy to run into errors if your
terminal(output device) is not able to handle unicode chars.<br>
Here are some rules when using Unicode input:</p>
<ol>
<li>
You need to wrap your input stream by a stream reader which
translates bytes into unicode chars. This requires usually
knowledge about your input's encoding. Assume for example
that your input is 'latin1', you would do this:
<blockquote>
<pre>
### replace stdin with a wrapper that spits out
### unicode chars.
sys.stdin = codecs.lookup('latin1')[-2](sys.stdin)
</pre>
</blockquote>
Here reading from stdin gets wrapped.
</li>
<li>
When printing tokens etc containing Unicode chars it appears
to be best to translate explicit to a unicode string before
printing. Consider:
<blockquote>
<pre>
for token in unicode_l.Lexer() :
print unicode(token) ## explict cast
</pre>
</blockquote>
This explicit cast appears to be a bug in Python found during
development (discussion still in progress).
</li>
</ol>
</li>
</ul>
</blockquote>
</body>
</html>
distrib
>
Mandriva
>
2010.1
>
x86_64
>
media
>
main-release
>
by-pkgid
>
303fbf88f78bb63b946e00c7f4e66ae1
>
files
>
21
