<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"> <html> <head> <title>Builtin Tokens — Pygments</title> <meta http-equiv="content-type" content="text/html; charset=utf-8"> <style type="text/css"> body { background-color: #f2f2f2; margin: 0; padding: 0; font-family: 'Georgia', serif; color: #111; } #content { background-color: white; padding: 20px; margin: 20px auto 20px auto; max-width: 800px; border: 4px solid #ddd; } h1 { font-weight: normal; font-size: 40px; color: #09839A; } h2 { font-weight: normal; font-size: 30px; color: #C73F00; } h1.heading { margin: 0 0 30px 0; } h2.subheading { margin: -30px 0 0 45px; } h3 { margin-top: 30px; } table.docutils { border-collapse: collapse; border: 2px solid #aaa; margin: 0.5em 1.5em 0.5em 1.5em; } table.docutils td { padding: 2px; border: 1px solid #ddd; } p, li, dd, dt, blockquote { font-size: 15px; color: #333; } p { line-height: 150%; margin-bottom: 0; margin-top: 10px; } hr { border-top: 1px solid #ccc; border-bottom: 0; border-right: 0; border-left: 0; margin-bottom: 10px; margin-top: 20px; } dl { margin-left: 10px; } li, dt { margin-top: 5px; } dt { font-weight: bold; } th { text-align: left; } a { color: #990000; } a:hover { color: #c73f00; } pre { background-color: #f9f9f9; border-top: 1px solid #ccc; border-bottom: 1px solid #ccc; padding: 5px; font-size: 13px; font-family: Bitstream Vera Sans Mono,monospace; } tt { font-size: 13px; font-family: Bitstream Vera Sans Mono,monospace; color: black; padding: 1px 2px 1px 2px; background-color: #f0f0f0; } cite { /* abusing <cite>, it's generated by ReST for `x` */ font-size: 13px; font-family: Bitstream Vera Sans Mono,monospace; font-weight: bold; font-style: normal; } #backlink { float: right; font-size: 11px; color: #888; } div.toc { margin: 0 0 10px 0; } div.toc h2 { font-size: 20px; } .syntax .hll { background-color: #ffffcc } .syntax { background: #ffffff; } .syntax .c { color: #888888 } /* Comment */ .syntax .err { color: #a61717; background-color: #e3d2d2 } /* Error */ .syntax .k { color: #008800; font-weight: bold } /* Keyword */ .syntax .cm { color: #888888 } /* Comment.Multiline */ .syntax .cp { color: #cc0000; font-weight: bold } /* Comment.Preproc */ .syntax .c1 { color: #888888 } /* Comment.Single */ .syntax .cs { color: #cc0000; font-weight: bold; background-color: #fff0f0 } /* Comment.Special */ .syntax .gd { color: #000000; background-color: #ffdddd } /* Generic.Deleted */ .syntax .ge { font-style: italic } /* Generic.Emph */ .syntax .gr { color: #aa0000 } /* Generic.Error */ .syntax .gh { color: #303030 } /* Generic.Heading */ .syntax .gi { color: #000000; background-color: #ddffdd } /* Generic.Inserted */ .syntax .go { color: #888888 } /* Generic.Output */ .syntax .gp { color: #555555 } /* Generic.Prompt */ .syntax .gs { font-weight: bold } /* Generic.Strong */ .syntax .gu { color: #606060 } /* Generic.Subheading */ .syntax .gt { color: #aa0000 } /* Generic.Traceback */ .syntax .kc { color: #008800; font-weight: bold } /* Keyword.Constant */ .syntax .kd { color: #008800; font-weight: bold } /* Keyword.Declaration */ .syntax .kn { color: #008800; font-weight: bold } /* Keyword.Namespace */ .syntax .kp { color: #008800 } /* Keyword.Pseudo */ .syntax .kr { color: #008800; font-weight: bold } /* Keyword.Reserved */ .syntax .kt { color: #888888; font-weight: bold } /* Keyword.Type */ .syntax .m { color: #0000DD; font-weight: bold } /* Literal.Number */ .syntax .s { color: #dd2200; background-color: #fff0f0 } /* Literal.String */ .syntax .na { color: #336699 } /* Name.Attribute */ .syntax .nb { color: #003388 } /* Name.Builtin */ .syntax .nc { color: #bb0066; font-weight: bold } /* Name.Class */ .syntax .no { color: #003366; font-weight: bold } /* Name.Constant */ .syntax .nd { color: #555555 } /* Name.Decorator */ .syntax .ne { color: #bb0066; font-weight: bold } /* Name.Exception */ .syntax .nf { color: #0066bb; font-weight: bold } /* Name.Function */ .syntax .nl { color: #336699; font-style: italic } /* Name.Label */ .syntax .nn { color: #bb0066; font-weight: bold } /* Name.Namespace */ .syntax .py { color: #336699; font-weight: bold } /* Name.Property */ .syntax .nt { color: #bb0066; font-weight: bold } /* Name.Tag */ .syntax .nv { color: #336699 } /* Name.Variable */ .syntax .ow { color: #008800 } /* Operator.Word */ .syntax .w { color: #bbbbbb } /* Text.Whitespace */ .syntax .mf { color: #0000DD; font-weight: bold } /* Literal.Number.Float */ .syntax .mh { color: #0000DD; font-weight: bold } /* Literal.Number.Hex */ .syntax .mi { color: #0000DD; font-weight: bold } /* Literal.Number.Integer */ .syntax .mo { color: #0000DD; font-weight: bold } /* Literal.Number.Oct */ .syntax .sb { color: #dd2200; background-color: #fff0f0 } /* Literal.String.Backtick */ .syntax .sc { color: #dd2200; background-color: #fff0f0 } /* Literal.String.Char */ .syntax .sd { color: #dd2200; background-color: #fff0f0 } /* Literal.String.Doc */ .syntax .s2 { color: #dd2200; background-color: #fff0f0 } /* Literal.String.Double */ .syntax .se { color: #0044dd; background-color: #fff0f0 } /* Literal.String.Escape */ .syntax .sh { color: #dd2200; background-color: #fff0f0 } /* Literal.String.Heredoc */ .syntax .si { color: #3333bb; background-color: #fff0f0 } /* Literal.String.Interpol */ .syntax .sx { color: #22bb22; background-color: #f0fff0 } /* Literal.String.Other */ .syntax .sr { color: #008800; background-color: #fff0ff } /* Literal.String.Regex */ .syntax .s1 { color: #dd2200; background-color: #fff0f0 } /* Literal.String.Single */ .syntax .ss { color: #aa6600; background-color: #fff0f0 } /* Literal.String.Symbol */ .syntax .bp { color: #003388 } /* Name.Builtin.Pseudo */ .syntax .vc { color: #336699 } /* Name.Variable.Class */ .syntax .vg { color: #dd7700 } /* Name.Variable.Global */ .syntax .vi { color: #3333bb } /* Name.Variable.Instance */ .syntax .il { color: #0000DD; font-weight: bold } /* Literal.Number.Integer.Long */ </style> </head> <body> <div id="content"> <h1 class="heading">Pygments</h1> <h2 class="subheading">Builtin Tokens</h2> <a id="backlink" href="index.html">« Back To Index</a> <div class="toc"> <h2>Contents</h2> <ul class="contents"> <li><a href="#keyword-tokens">Keyword Tokens</a></li> <li><a href="#name-tokens">Name Tokens</a></li> <li><a href="#literals">Literals</a></li> <li><a href="#operators">Operators</a></li> <li><a href="#punctuation">Punctuation</a></li> <li><a href="#comments">Comments</a></li> <li><a href="#generic-tokens">Generic Tokens</a></li> </ul> </div> <!-- -*- mode: rst -*- --> <p>Inside the <cite>pygments.token</cite> module, there is a special object called <cite>Token</cite> that is used to create token types.</p> <p>You can create a new token type by accessing an attribute of <cite>Token</cite>:</p> <div class="syntax"><pre><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">pygments.token</span> <span class="kn">import</span> <span class="n">Token</span> <span class="gp">>>> </span><span class="n">Token</span><span class="o">.</span><span class="n">String</span> <span class="go">Token.String</span> <span class="gp">>>> </span><span class="n">Token</span><span class="o">.</span><span class="n">String</span> <span class="ow">is</span> <span class="n">Token</span><span class="o">.</span><span class="n">String</span> <span class="go">True</span> </pre></div> <p>Note that tokens are singletons so you can use the <tt class="docutils literal">is</tt> operator for comparing token types.</p> <p>As of Pygments 0.7 you can also use the <tt class="docutils literal">in</tt> operator to perform set tests:</p> <div class="syntax"><pre><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">pygments.token</span> <span class="kn">import</span> <span class="n">Comment</span> <span class="gp">>>> </span><span class="n">Comment</span><span class="o">.</span><span class="n">Single</span> <span class="ow">in</span> <span class="n">Comment</span> <span class="go">True</span> <span class="gp">>>> </span><span class="n">Comment</span> <span class="ow">in</span> <span class="n">Comment</span><span class="o">.</span><span class="n">Multi</span> <span class="go">False</span> </pre></div> <p>This can be useful in <a class="reference external" href="./filters.html">filters</a> and if you write lexers on your own without using the base lexers.</p> <p>You can also split a token type into a hierarchy, and get the parent of it:</p> <div class="syntax"><pre><span class="gp">>>> </span><span class="n">String</span><span class="o">.</span><span class="n">split</span><span class="p">()</span> <span class="go">[Token, Token.Literal, Token.Literal.String]</span> <span class="gp">>>> </span><span class="n">String</span><span class="o">.</span><span class="n">parent</span> <span class="go">Token.Literal</span> </pre></div> <p>In principle, you can create an unlimited number of token types but nobody can guarantee that a style would define style rules for a token type. Because of that, Pygments proposes some global token types defined in the <cite>pygments.token.STANDARD_TYPES</cite> dict.</p> <p>For some tokens aliases are already defined:</p> <div class="syntax"><pre><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">pygments.token</span> <span class="kn">import</span> <span class="n">String</span> <span class="gp">>>> </span><span class="n">String</span> <span class="go">Token.Literal.String</span> </pre></div> <p>Inside the <cite>pygments.token</cite> module the following aliases are defined:</p> <table border="1" class="docutils"> <colgroup> <col width="17%" /> <col width="36%" /> <col width="47%" /> </colgroup> <tbody valign="top"> <tr><td><cite>Text</cite></td> <td><cite>Token.Text</cite></td> <td>for any type of text data</td> </tr> <tr><td><cite>Whitespace</cite></td> <td><cite>Token.Text.Whitespace</cite></td> <td>for specially highlighted whitespace</td> </tr> <tr><td><cite>Error</cite></td> <td><cite>Token.Error</cite></td> <td>represents lexer errors</td> </tr> <tr><td><cite>Other</cite></td> <td><cite>Token.Other</cite></td> <td>special token for data not matched by a parser (e.g. HTML markup in PHP code)</td> </tr> <tr><td><cite>Keyword</cite></td> <td><cite>Token.Keyword</cite></td> <td>any kind of keywords</td> </tr> <tr><td><cite>Name</cite></td> <td><cite>Token.Name</cite></td> <td>variable/function names</td> </tr> <tr><td><cite>Literal</cite></td> <td><cite>Token.Literal</cite></td> <td>Any literals</td> </tr> <tr><td><cite>String</cite></td> <td><cite>Token.Literal.String</cite></td> <td>string literals</td> </tr> <tr><td><cite>Number</cite></td> <td><cite>Token.Literal.Number</cite></td> <td>number literals</td> </tr> <tr><td><cite>Operator</cite></td> <td><cite>Token.Operator</cite></td> <td>operators (<tt class="docutils literal">+</tt>, <tt class="docutils literal">not</tt>...)</td> </tr> <tr><td><cite>Punctuation</cite></td> <td><cite>Token.Punctuation</cite></td> <td>punctuation (<tt class="docutils literal">[</tt>, <tt class="docutils literal">(</tt>...)</td> </tr> <tr><td><cite>Comment</cite></td> <td><cite>Token.Comment</cite></td> <td>any kind of comments</td> </tr> <tr><td><cite>Generic</cite></td> <td><cite>Token.Generic</cite></td> <td>generic tokens (have a look at the explanation below)</td> </tr> </tbody> </table> <p>The <cite>Whitespace</cite> token type is new in Pygments 0.8. It is used only by the <cite>VisibleWhitespaceFilter</cite> currently.</p> <p>Normally you just create token types using the already defined aliases. For each of those token aliases, a number of subtypes exists (excluding the special tokens <cite>Token.Text</cite>, <cite>Token.Error</cite> and <cite>Token.Other</cite>)</p> <p>The <cite>is_token_subtype()</cite> function in the <cite>pygments.token</cite> module can be used to test if a token type is a subtype of another (such as <cite>Name.Tag</cite> and <cite>Name</cite>). (This is the same as <tt class="docutils literal">Name.Tag in Name</tt>. The overloaded <cite>in</cite> operator was newly introduced in Pygments 0.7, the function still exists for backwards compatiblity.)</p> <p>With Pygments 0.7, it's also possible to convert strings to token types (for example if you want to supply a token from the command line):</p> <div class="syntax"><pre><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">pygments.token</span> <span class="kn">import</span> <span class="n">String</span><span class="p">,</span> <span class="n">string_to_tokentype</span> <span class="gp">>>> </span><span class="n">string_to_tokentype</span><span class="p">(</span><span class="s">"String"</span><span class="p">)</span> <span class="go">Token.Literal.String</span> <span class="gp">>>> </span><span class="n">string_to_tokentype</span><span class="p">(</span><span class="s">"Token.Literal.String"</span><span class="p">)</span> <span class="go">Token.Literal.String</span> <span class="gp">>>> </span><span class="n">string_to_tokentype</span><span class="p">(</span><span class="n">String</span><span class="p">)</span> <span class="go">Token.Literal.String</span> </pre></div> <div class="section" id="keyword-tokens"> <h3>Keyword Tokens</h3> <dl class="docutils"> <dt><cite>Keyword</cite></dt> <dd>For any kind of keyword (especially if it doesn't match any of the subtypes of course).</dd> <dt><cite>Keyword.Constant</cite></dt> <dd>For keywords that are constants (e.g. <tt class="docutils literal">None</tt> in future Python versions).</dd> <dt><cite>Keyword.Declaration</cite></dt> <dd>For keywords used for variable declaration (e.g. <tt class="docutils literal">var</tt> in some programming languages like JavaScript).</dd> <dt><cite>Keyword.Namespace</cite></dt> <dd>For keywords used for namespace declarations (e.g. <tt class="docutils literal">import</tt> in Python and Java and <tt class="docutils literal">package</tt> in Java).</dd> <dt><cite>Keyword.Pseudo</cite></dt> <dd>For keywords that aren't really keywords (e.g. <tt class="docutils literal">None</tt> in old Python versions).</dd> <dt><cite>Keyword.Reserved</cite></dt> <dd>For reserved keywords.</dd> <dt><cite>Keyword.Type</cite></dt> <dd>For builtin types that can't be used as identifiers (e.g. <tt class="docutils literal">int</tt>, <tt class="docutils literal">char</tt> etc. in C).</dd> </dl> </div> <div class="section" id="name-tokens"> <h3>Name Tokens</h3> <dl class="docutils"> <dt><cite>Name</cite></dt> <dd>For any name (variable names, function names, classes).</dd> <dt><cite>Name.Attribute</cite></dt> <dd>For all attributes (e.g. in HTML tags).</dd> <dt><cite>Name.Builtin</cite></dt> <dd>Builtin names; names that are available in the global namespace.</dd> <dt><cite>Name.Builtin.Pseudo</cite></dt> <dd>Builtin names that are implicit (e.g. <tt class="docutils literal">self</tt> in Ruby, <tt class="docutils literal">this</tt> in Java).</dd> <dt><cite>Name.Class</cite></dt> <dd>Class names. Because no lexer can know if a name is a class or a function or something else this token is meant for class declarations.</dd> <dt><cite>Name.Constant</cite></dt> <dd>Token type for constants. In some languages you can recognise a token by the way it's defined (the value after a <tt class="docutils literal">const</tt> keyword for example). In other languages constants are uppercase by definition (Ruby).</dd> <dt><cite>Name.Decorator</cite></dt> <dd>Token type for decorators. Decorators are synatic elements in the Python language. Similar syntax elements exist in C# and Java.</dd> <dt><cite>Name.Entity</cite></dt> <dd>Token type for special entities. (e.g. <tt class="docutils literal">&nbsp;</tt> in HTML).</dd> <dt><cite>Name.Exception</cite></dt> <dd>Token type for exception names (e.g. <tt class="docutils literal">RuntimeError</tt> in Python). Some languages define exceptions in the function signature (Java). You can highlight the name of that exception using this token then.</dd> <dt><cite>Name.Function</cite></dt> <dd>Token type for function names.</dd> <dt><cite>Name.Label</cite></dt> <dd>Token type for label names (e.g. in languages that support <tt class="docutils literal">goto</tt>).</dd> <dt><cite>Name.Namespace</cite></dt> <dd>Token type for namespaces. (e.g. import paths in Java/Python), names following the <tt class="docutils literal">module</tt>/<tt class="docutils literal">namespace</tt> keyword in other languages.</dd> <dt><cite>Name.Other</cite></dt> <dd>Other names. Normally unused.</dd> <dt><cite>Name.Tag</cite></dt> <dd>Tag names (in HTML/XML markup or configuration files).</dd> <dt><cite>Name.Variable</cite></dt> <dd>Token type for variables. Some languages have prefixes for variable names (PHP, Ruby, Perl). You can highlight them using this token.</dd> <dt><cite>Name.Variable.Class</cite></dt> <dd>same as <cite>Name.Variable</cite> but for class variables (also static variables).</dd> <dt><cite>Name.Variable.Global</cite></dt> <dd>same as <cite>Name.Variable</cite> but for global variables (used in Ruby, for example).</dd> <dt><cite>Name.Variable.Instance</cite></dt> <dd>same as <cite>Name.Variable</cite> but for instance variables.</dd> </dl> </div> <div class="section" id="literals"> <h3>Literals</h3> <dl class="docutils"> <dt><cite>Literal</cite></dt> <dd>For any literal (if not further defined).</dd> <dt><cite>Literal.Date</cite></dt> <dd>for date literals (e.g. <tt class="docutils literal">42d</tt> in Boo).</dd> <dt><cite>String</cite></dt> <dd>For any string literal.</dd> <dt><cite>String.Backtick</cite></dt> <dd>Token type for strings enclosed in backticks.</dd> <dt><cite>String.Char</cite></dt> <dd>Token type for single characters (e.g. Java, C).</dd> <dt><cite>String.Doc</cite></dt> <dd>Token type for documentation strings (for example Python).</dd> <dt><cite>String.Double</cite></dt> <dd>Double quoted strings.</dd> <dt><cite>String.Escape</cite></dt> <dd>Token type for escape sequences in strings.</dd> <dt><cite>String.Heredoc</cite></dt> <dd>Token type for "heredoc" strings (e.g. in Ruby or Perl).</dd> <dt><cite>String.Interpol</cite></dt> <dd>Token type for interpolated parts in strings (e.g. <tt class="docutils literal">#{foo}</tt> in Ruby).</dd> <dt><cite>String.Other</cite></dt> <dd>Token type for any other strings (for example <tt class="docutils literal">%q{foo}</tt> string constructs in Ruby).</dd> <dt><cite>String.Regex</cite></dt> <dd>Token type for regular expression literals (e.g. <tt class="docutils literal">/foo/</tt> in JavaScript).</dd> <dt><cite>String.Single</cite></dt> <dd>Token type for single quoted strings.</dd> <dt><cite>String.Symbol</cite></dt> <dd>Token type for symbols (e.g. <tt class="docutils literal">:foo</tt> in LISP or Ruby).</dd> <dt><cite>Number</cite></dt> <dd>Token type for any number literal.</dd> <dt><cite>Number.Float</cite></dt> <dd>Token type for float literals (e.g. <tt class="docutils literal">42.0</tt>).</dd> <dt><cite>Number.Hex</cite></dt> <dd>Token type for hexadecimal number literals (e.g. <tt class="docutils literal">0xdeadbeef</tt>).</dd> <dt><cite>Number.Integer</cite></dt> <dd>Token type for integer literals (e.g. <tt class="docutils literal">42</tt>).</dd> <dt><cite>Number.Integer.Long</cite></dt> <dd>Token type for long integer literals (e.g. <tt class="docutils literal">42L</tt> in Python).</dd> <dt><cite>Number.Oct</cite></dt> <dd>Token type for octal literals.</dd> </dl> </div> <div class="section" id="operators"> <h3>Operators</h3> <dl class="docutils"> <dt><cite>Operator</cite></dt> <dd>For any punctuation operator (e.g. <tt class="docutils literal">+</tt>, <tt class="docutils literal">-</tt>).</dd> <dt><cite>Operator.Word</cite></dt> <dd>For any operator that is a word (e.g. <tt class="docutils literal">not</tt>).</dd> </dl> </div> <div class="section" id="punctuation"> <h3>Punctuation</h3> <p><em>New in Pygments 0.7.</em></p> <dl class="docutils"> <dt><cite>Punctuation</cite></dt> <dd>For any punctuation which is not an operator (e.g. <tt class="docutils literal">[</tt>, <tt class="docutils literal">(</tt>...)</dd> </dl> </div> <div class="section" id="comments"> <h3>Comments</h3> <dl class="docutils"> <dt><cite>Comment</cite></dt> <dd>Token type for any comment.</dd> <dt><cite>Comment.Multiline</cite></dt> <dd>Token type for multiline comments.</dd> <dt><cite>Comment.Preproc</cite></dt> <dd>Token type for preprocessor comments (also <tt class="docutils literal"><span class="pre"><?php</span></tt>/<tt class="docutils literal"><%</tt> constructs).</dd> <dt><cite>Comment.Single</cite></dt> <dd>Token type for comments that end at the end of a line (e.g. <tt class="docutils literal"># foo</tt>).</dd> <dt><cite>Comment.Special</cite></dt> <dd>Special data in comments. For example code tags, author and license informations etc.</dd> </dl> </div> <div class="section" id="generic-tokens"> <h3>Generic Tokens</h3> <p>Generic tokens are for special lexers like the <cite>DiffLexer</cite> that doesn't really highlight a programming language but a patch file.</p> <dl class="docutils"> <dt><cite>Generic</cite></dt> <dd>A generic, unstyled token. Normally you don't use this token type.</dd> <dt><cite>Generic.Deleted</cite></dt> <dd>Marks the token value as deleted.</dd> <dt><cite>Generic.Emph</cite></dt> <dd>Marks the token value as emphasized.</dd> <dt><cite>Generic.Error</cite></dt> <dd>Marks the token value as an error message.</dd> <dt><cite>Generic.Heading</cite></dt> <dd>Marks the token value as headline.</dd> <dt><cite>Generic.Inserted</cite></dt> <dd>Marks the token value as inserted.</dd> <dt><cite>Generic.Output</cite></dt> <dd>Marks the token value as program output (e.g. for python cli lexer).</dd> <dt><cite>Generic.Prompt</cite></dt> <dd>Marks the token value as command prompt (e.g. bash lexer).</dd> <dt><cite>Generic.Strong</cite></dt> <dd>Marks the token value as bold (e.g. for rst lexer).</dd> <dt><cite>Generic.Subheading</cite></dt> <dd>Marks the token value as subheadline.</dd> <dt><cite>Generic.Traceback</cite></dt> <dd>Marks the token value as a part of an error traceback.</dd> </dl> </div> </div> </body> <!-- generated on: 2010-03-01 21:18:23.269356 file id: tokens --> </html>