<!DOCTYPE html> <html> <head> <title>lexer.coffee</title> <meta http-equiv="content-type" content="text/html; charset=UTF-8"> <meta name="viewport" content="width=device-width, target-densitydpi=160dpi, initial-scale=1.0; maximum-scale=1.0; user-scalable=0;"> <link rel="stylesheet" media="all" href="docco.css" /> </head> <body> <div id="container"> <div id="background"></div> <ul id="jump_to"> <li> <a class="large" href="javascript:void(0);">Jump To …</a> <a class="small" href="javascript:void(0);">+</a> <div id="jump_wrapper"> <div id="jump_page"> <a class="source" href="browser.html"> browser.coffee </a> <a class="source" href="cake.html"> cake.coffee </a> <a class="source" href="coffee-script.html"> coffee-script.coffee </a> <a class="source" href="command.html"> command.coffee </a> <a class="source" href="grammar.html"> grammar.coffee </a> <a class="source" href="helpers.html"> helpers.coffee </a> <a class="source" href="index.html"> index.coffee </a> <a class="source" href="lexer.html"> lexer.coffee </a> <a class="source" href="nodes.html"> nodes.coffee </a> <a class="source" href="optparse.html"> optparse.coffee </a> <a class="source" href="repl.html"> repl.coffee </a> <a class="source" href="rewriter.html"> rewriter.coffee </a> <a class="source" href="scope.html"> scope.litcoffee </a> <a class="source" href="sourcemap.html"> sourcemap.litcoffee </a> </div> </li> </ul> <ul class="sections"> <li id="title"> <div class="annotation"> <h1>lexer.coffee</h1> </div> </li> <li id="section-1"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-1">¶</a> </div> <p>The CoffeeScript Lexer. Uses a series of token-matching regexes to attempt matches against the beginning of the source code. When a match is found, a token is produced, we consume the match, and start again. Tokens are in the form: </p> <pre><code>[tag, value, locationData]</code></pre> <p>where locationData is {first_line, first_column, last_line, last_column}, which is a format that can be fed directly into <a href="http://github.com/zaach/jison">Jison</a>. These are read by jison in the <code>parser.lexer</code> function defined in coffee-script.coffee. </p> </div> <div class="content"><div class='highlight'><pre> {Rewriter, INVERSES} = require <span class="string">'./rewriter'</span></pre></div></div> </li> <li id="section-2"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-2">¶</a> </div> <p>Import the helpers we need. </p> </div> <div class="content"><div class='highlight'><pre>{count, starts, compact, last, repeat, invertLiterate, locationDataToString, throwSyntaxError} = require <span class="string">'./helpers'</span></pre></div></div> </li> <li id="section-3"> <div class="annotation"> <div class="pilwrap for-h2"> <a class="pilcrow" href="#section-3">¶</a> </div> <h2>The Lexer Class</h2> </div> </li> <li id="section-4"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-4">¶</a> </div> </div> </li> <li id="section-5"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-5">¶</a> </div> <p>The Lexer class reads a stream of CoffeeScript and divvies it up into tagged tokens. Some potential ambiguity in the grammar has been avoided by pushing some extra smarts into the Lexer. </p> </div> <div class="content"><div class='highlight'><pre>exports.Lexer = <span class="class"><span class="keyword">class</span> <span class="title">Lexer</span></span></pre></div></div> </li> <li id="section-6"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-6">¶</a> </div> <p><strong>tokenize</strong> is the Lexer's main method. Scan by attempting to match tokens one at a time, using a regular expression anchored at the start of the remaining code, or a custom recursive token-matching method (for interpolations). When the next token has been recorded, we move forward within the code past the token, and begin again. </p> <p>Each tokenizing method is responsible for returning the number of characters it has consumed. </p> <p>Before returning the token stream, run it through the <a href="rewriter.html">Rewriter</a> unless explicitly asked not to. </p> </div> <div class="content"><div class='highlight'><pre> tokenize: (code, opts = {}) -> <span class="property">@literate</span> = opts.literate <span class="comment"># Are we lexing literate CoffeeScript?</span> <span class="property">@indent</span> = <span class="number">0</span> <span class="comment"># The current indentation level.</span> <span class="property">@indebt</span> = <span class="number">0</span> <span class="comment"># The over-indentation at the current level.</span> <span class="property">@outdebt</span> = <span class="number">0</span> <span class="comment"># The under-outdentation at the current level.</span> <span class="property">@indents</span> = [] <span class="comment"># The stack of all current indentation levels.</span> <span class="property">@ends</span> = [] <span class="comment"># The stack for pairing up tokens.</span> <span class="property">@tokens</span> = [] <span class="comment"># Stream of parsed tokens in the form `['TYPE', value, location data]`.</span> <span class="property">@chunkLine</span> = opts.line <span class="keyword">or</span> <span class="number">0</span> <span class="comment"># The start line for the current @chunk.</span> <span class="property">@chunkColumn</span> = opts.column <span class="keyword">or</span> <span class="number">0</span> <span class="comment"># The start column of the current @chunk.</span> code = <span class="property">@clean</span> code <span class="comment"># The stripped, cleaned original source code.</span></pre></div></div> </li> <li id="section-7"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-7">¶</a> </div> <p>At every position, run through this list of attempted matches, short-circuiting if any of them succeed. Their order determines precedence: <code>@literalToken</code> is the fallback catch-all. </p> </div> <div class="content"><div class='highlight'><pre> i = <span class="number">0</span> <span class="keyword">while</span> <span class="property">@chunk</span> = code[i..] consumed = \ <span class="property">@identifierToken</span>() <span class="keyword">or</span> <span class="property">@commentToken</span>() <span class="keyword">or</span> <span class="property">@whitespaceToken</span>() <span class="keyword">or</span> <span class="property">@lineToken</span>() <span class="keyword">or</span> <span class="property">@heredocToken</span>() <span class="keyword">or</span> <span class="property">@stringToken</span>() <span class="keyword">or</span> <span class="property">@numberToken</span>() <span class="keyword">or</span> <span class="property">@regexToken</span>() <span class="keyword">or</span> <span class="property">@jsToken</span>() <span class="keyword">or</span> <span class="property">@literalToken</span>()</pre></div></div> </li> <li id="section-8"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-8">¶</a> </div> <p>Update position </p> </div> <div class="content"><div class='highlight'><pre> [<span class="property">@chunkLine</span>, <span class="property">@chunkColumn</span>] = <span class="property">@getLineAndColumnFromChunk</span> consumed i += consumed <span class="property">@closeIndentation</span>() <span class="property">@error</span> <span class="string">"missing <span class="subst">#{tag}</span>"</span> <span class="keyword">if</span> tag = <span class="property">@ends</span>.pop() <span class="keyword">return</span> <span class="property">@tokens</span> <span class="keyword">if</span> opts.rewrite <span class="keyword">is</span> <span class="literal">off</span> (<span class="keyword">new</span> Rewriter).rewrite <span class="property">@tokens</span></pre></div></div> </li> <li id="section-9"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-9">¶</a> </div> <p>Preprocess the code to remove leading and trailing whitespace, carriage returns, etc. If we're lexing literate CoffeeScript, strip external Markdown by removing all lines that aren't indented by at least four spaces or a tab. </p> </div> <div class="content"><div class='highlight'><pre> clean: (code) -> code = code.slice(<span class="number">1</span>) <span class="keyword">if</span> code.charCodeAt(<span class="number">0</span>) <span class="keyword">is</span> BOM code = code.replace(<span class="regexp">/\r/g</span>, <span class="string">''</span>).replace TRAILING_SPACES, <span class="string">''</span> <span class="keyword">if</span> WHITESPACE.test code code = <span class="string">"\n<span class="subst">#{code}</span>"</span> <span class="property">@chunkLine</span>-- code = invertLiterate code <span class="keyword">if</span> <span class="property">@literate</span> code</pre></div></div> </li> <li id="section-10"> <div class="annotation"> <div class="pilwrap for-h2"> <a class="pilcrow" href="#section-10">¶</a> </div> <h2>Tokenizers</h2> </div> </li> <li id="section-11"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-11">¶</a> </div> </div> </li> <li id="section-12"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-12">¶</a> </div> <p>Matches identifying literals: variables, keywords, method names, etc. Check to ensure that JavaScript reserved words aren't being used as identifiers. Because CoffeeScript reserves a handful of keywords that are allowed in JavaScript, we're careful not to tag them as keywords when referenced as property names here, so you can still do <code>jQuery.is()</code> even though <code>is</code> means <code>===</code> otherwise. </p> </div> <div class="content"><div class='highlight'><pre> identifierToken: -> <span class="keyword">return</span> <span class="number">0</span> <span class="keyword">unless</span> match = IDENTIFIER.exec <span class="property">@chunk</span> [input, id, colon] = match</pre></div></div> </li> <li id="section-13"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-13">¶</a> </div> <p>Preserve length of id for location data </p> </div> <div class="content"><div class='highlight'><pre> idLength = id.length poppedToken = <span class="literal">undefined</span> <span class="keyword">if</span> id <span class="keyword">is</span> <span class="string">'own'</span> <span class="keyword">and</span> <span class="property">@tag</span>() <span class="keyword">is</span> <span class="string">'FOR'</span> <span class="property">@token</span> <span class="string">'OWN'</span>, id <span class="keyword">return</span> id.length forcedIdentifier = colon <span class="keyword">or</span> (prev = last <span class="property">@tokens</span>) <span class="keyword">and</span> (prev[<span class="number">0</span>] <span class="keyword">in</span> [<span class="string">'.'</span>, <span class="string">'?.'</span>, <span class="string">'::'</span>, <span class="string">'?::'</span>] <span class="keyword">or</span> <span class="keyword">not</span> prev.spaced <span class="keyword">and</span> prev[<span class="number">0</span>] <span class="keyword">is</span> <span class="string">'@'</span>) tag = <span class="string">'IDENTIFIER'</span> <span class="keyword">if</span> <span class="keyword">not</span> forcedIdentifier <span class="keyword">and</span> (id <span class="keyword">in</span> JS_KEYWORDS <span class="keyword">or</span> id <span class="keyword">in</span> COFFEE_KEYWORDS) tag = id.toUpperCase() <span class="keyword">if</span> tag <span class="keyword">is</span> <span class="string">'WHEN'</span> <span class="keyword">and</span> <span class="property">@tag</span>() <span class="keyword">in</span> LINE_BREAK tag = <span class="string">'LEADING_WHEN'</span> <span class="keyword">else</span> <span class="keyword">if</span> tag <span class="keyword">is</span> <span class="string">'FOR'</span> <span class="property">@seenFor</span> = <span class="literal">yes</span> <span class="keyword">else</span> <span class="keyword">if</span> tag <span class="keyword">is</span> <span class="string">'UNLESS'</span> tag = <span class="string">'IF'</span> <span class="keyword">else</span> <span class="keyword">if</span> tag <span class="keyword">in</span> UNARY tag = <span class="string">'UNARY'</span> <span class="keyword">else</span> <span class="keyword">if</span> tag <span class="keyword">in</span> RELATION <span class="keyword">if</span> tag <span class="keyword">isnt</span> <span class="string">'INSTANCEOF'</span> <span class="keyword">and</span> <span class="property">@seenFor</span> tag = <span class="string">'FOR'</span> + tag <span class="property">@seenFor</span> = <span class="literal">no</span> <span class="keyword">else</span> tag = <span class="string">'RELATION'</span> <span class="keyword">if</span> <span class="property">@value</span>() <span class="keyword">is</span> <span class="string">'!'</span> poppedToken = <span class="property">@tokens</span>.pop() id = <span class="string">'!'</span> + id <span class="keyword">if</span> id <span class="keyword">in</span> JS_FORBIDDEN <span class="keyword">if</span> forcedIdentifier tag = <span class="string">'IDENTIFIER'</span> id = <span class="keyword">new</span> String id id.reserved = <span class="literal">yes</span> <span class="keyword">else</span> <span class="keyword">if</span> id <span class="keyword">in</span> RESERVED <span class="property">@error</span> <span class="string">"reserved word \"<span class="subst">#{id}</span>\""</span> <span class="keyword">unless</span> forcedIdentifier id = COFFEE_ALIAS_MAP[id] <span class="keyword">if</span> id <span class="keyword">in</span> COFFEE_ALIASES tag = <span class="keyword">switch</span> id <span class="keyword">when</span> <span class="string">'!'</span> <span class="keyword">then</span> <span class="string">'UNARY'</span> <span class="keyword">when</span> <span class="string">'=='</span>, <span class="string">'!='</span> <span class="keyword">then</span> <span class="string">'COMPARE'</span> <span class="keyword">when</span> <span class="string">'&&'</span>, <span class="string">'||'</span> <span class="keyword">then</span> <span class="string">'LOGIC'</span> <span class="keyword">when</span> <span class="string">'true'</span>, <span class="string">'false'</span> <span class="keyword">then</span> <span class="string">'BOOL'</span> <span class="keyword">when</span> <span class="string">'break'</span>, <span class="string">'continue'</span> <span class="keyword">then</span> <span class="string">'STATEMENT'</span> <span class="keyword">else</span> tag tagToken = <span class="property">@token</span> tag, id, <span class="number">0</span>, idLength <span class="keyword">if</span> poppedToken [tagToken[<span class="number">2</span>].first_line, tagToken[<span class="number">2</span>].first_column] = [poppedToken[<span class="number">2</span>].first_line, poppedToken[<span class="number">2</span>].first_column] <span class="keyword">if</span> colon colonOffset = input.lastIndexOf <span class="string">':'</span> <span class="property">@token</span> <span class="string">':'</span>, <span class="string">':'</span>, colonOffset, colon.length input.length</pre></div></div> </li> <li id="section-14"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-14">¶</a> </div> <p>Matches numbers, including decimals, hex, and exponential notation. Be careful not to interfere with ranges-in-progress. </p> </div> <div class="content"><div class='highlight'><pre> numberToken: -> <span class="keyword">return</span> <span class="number">0</span> <span class="keyword">unless</span> match = NUMBER.exec <span class="property">@chunk</span> number = match[<span class="number">0</span>] <span class="keyword">if</span> <span class="regexp">/^0[BOX]/</span>.test number <span class="property">@error</span> <span class="string">"radix prefix '<span class="subst">#{number}</span>' must be lowercase"</span> <span class="keyword">else</span> <span class="keyword">if</span> <span class="regexp">/E/.test(number) and not /^0x/</span>.test number <span class="property">@error</span> <span class="string">"exponential notation '<span class="subst">#{number}</span>' must be indicated with a lowercase 'e'"</span> <span class="keyword">else</span> <span class="keyword">if</span> <span class="regexp">/^0\d*[89]/</span>.test number <span class="property">@error</span> <span class="string">"decimal literal '<span class="subst">#{number}</span>' must not be prefixed with '0'"</span> <span class="keyword">else</span> <span class="keyword">if</span> <span class="regexp">/^0\d+/</span>.test number <span class="property">@error</span> <span class="string">"octal literal '<span class="subst">#{number}</span>' must be prefixed with '0o'"</span> lexedLength = number.length <span class="keyword">if</span> octalLiteral = <span class="regexp">/^0o([0-7]+)/</span>.exec number number = <span class="string">'0x'</span> + parseInt(octalLiteral[<span class="number">1</span>], <span class="number">8</span>).toString <span class="number">16</span> <span class="keyword">if</span> binaryLiteral = <span class="regexp">/^0b([01]+)/</span>.exec number number = <span class="string">'0x'</span> + parseInt(binaryLiteral[<span class="number">1</span>], <span class="number">2</span>).toString <span class="number">16</span> <span class="property">@token</span> <span class="string">'NUMBER'</span>, number, <span class="number">0</span>, lexedLength lexedLength</pre></div></div> </li> <li id="section-15"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-15">¶</a> </div> <p>Matches strings, including multi-line strings. Ensures that quotation marks are balanced within the string's contents, and within nested interpolations. </p> </div> <div class="content"><div class='highlight'><pre> stringToken: -> <span class="keyword">switch</span> <span class="property">@chunk</span>.charAt <span class="number">0</span> <span class="keyword">when</span> <span class="string">"'"</span> <span class="keyword">return</span> <span class="number">0</span> <span class="keyword">unless</span> match = SIMPLESTR.exec <span class="property">@chunk</span> string = match[<span class="number">0</span>] <span class="property">@token</span> <span class="string">'STRING'</span>, string.replace(MULTILINER, <span class="string">'\\\n'</span>), <span class="number">0</span>, string.length <span class="keyword">when</span> <span class="string">'"'</span> <span class="keyword">return</span> <span class="number">0</span> <span class="keyword">unless</span> string = <span class="property">@balancedString</span> <span class="property">@chunk</span>, <span class="string">'"'</span> <span class="keyword">if</span> <span class="number">0</span> < string.indexOf <span class="string">'#{'</span>, <span class="number">1</span> <span class="property">@interpolateString</span> string[<span class="number">1.</span>..-<span class="number">1</span>], strOffset: <span class="number">1</span>, lexedLength: string.length <span class="keyword">else</span> <span class="property">@token</span> <span class="string">'STRING'</span>, <span class="property">@escapeLines</span> string, <span class="number">0</span>, string.length <span class="keyword">else</span> <span class="keyword">return</span> <span class="number">0</span> <span class="keyword">if</span> octalEsc = <span class="regexp">/^(?:\\.|[^\\])*\\(?:0[0-7]|[1-7])/</span>.test string <span class="property">@error</span> <span class="string">"octal escape sequences <span class="subst">#{string}</span> are not allowed"</span> string.length</pre></div></div> </li> <li id="section-16"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-16">¶</a> </div> <p>Matches heredocs, adjusting indentation to the correct level, as heredocs preserve whitespace, but ignore indentation to the left. </p> </div> <div class="content"><div class='highlight'><pre> heredocToken: -> <span class="keyword">return</span> <span class="number">0</span> <span class="keyword">unless</span> match = HEREDOC.exec <span class="property">@chunk</span> heredoc = match[<span class="number">0</span>] quote = heredoc.charAt <span class="number">0</span> doc = <span class="property">@sanitizeHeredoc</span> match[<span class="number">2</span>], quote: quote, indent: <span class="literal">null</span> <span class="keyword">if</span> quote <span class="keyword">is</span> <span class="string">'"'</span> <span class="keyword">and</span> <span class="number">0</span> <= doc.indexOf <span class="string">'#{'</span> <span class="property">@interpolateString</span> doc, heredoc: <span class="literal">yes</span>, strOffset: <span class="number">3</span>, lexedLength: heredoc.length <span class="keyword">else</span> <span class="property">@token</span> <span class="string">'STRING'</span>, <span class="property">@makeString</span>(doc, quote, <span class="literal">yes</span>), <span class="number">0</span>, heredoc.length heredoc.length</pre></div></div> </li> <li id="section-17"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-17">¶</a> </div> <p>Matches and consumes comments. </p> </div> <div class="content"><div class='highlight'><pre> commentToken: -> <span class="keyword">return</span> <span class="number">0</span> <span class="keyword">unless</span> match = <span class="property">@chunk</span>.match COMMENT [comment, here] = match <span class="keyword">if</span> here <span class="property">@token</span> <span class="string">'HERECOMMENT'</span>, (<span class="property">@sanitizeHeredoc</span> here, herecomment: <span class="literal">true</span>, indent: repeat <span class="string">' '</span>, <span class="property">@indent</span>), <span class="number">0</span>, comment.length comment.length</pre></div></div> </li> <li id="section-18"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-18">¶</a> </div> <p>Matches JavaScript interpolated directly into the source via backticks. </p> </div> <div class="content"><div class='highlight'><pre> jsToken: -> <span class="keyword">return</span> <span class="number">0</span> <span class="keyword">unless</span> <span class="property">@chunk</span>.charAt(<span class="number">0</span>) <span class="keyword">is</span> <span class="string">'`'</span> <span class="keyword">and</span> match = JSTOKEN.exec <span class="property">@chunk</span> <span class="property">@token</span> <span class="string">'JS'</span>, (script = match[<span class="number">0</span>])[<span class="number">1.</span>..-<span class="number">1</span>], <span class="number">0</span>, script.length script.length</pre></div></div> </li> <li id="section-19"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-19">¶</a> </div> <p>Matches regular expression literals. Lexing regular expressions is difficult to distinguish from division, so we borrow some basic heuristics from JavaScript and Ruby. </p> </div> <div class="content"><div class='highlight'><pre> regexToken: -> <span class="keyword">return</span> <span class="number">0</span> <span class="keyword">if</span> <span class="property">@chunk</span>.charAt(<span class="number">0</span>) <span class="keyword">isnt</span> <span class="string">'/'</span> <span class="keyword">if</span> match = HEREGEX.exec <span class="property">@chunk</span> length = <span class="property">@heregexToken</span> match <span class="keyword">return</span> length prev = last <span class="property">@tokens</span> <span class="keyword">return</span> <span class="number">0</span> <span class="keyword">if</span> prev <span class="keyword">and</span> (prev[<span class="number">0</span>] <span class="keyword">in</span> (<span class="keyword">if</span> prev.spaced <span class="keyword">then</span> NOT_REGEX <span class="keyword">else</span> NOT_SPACED_REGEX)) <span class="keyword">return</span> <span class="number">0</span> <span class="keyword">unless</span> match = REGEX.exec <span class="property">@chunk</span> [match, regex, flags] = match <span class="keyword">if</span> regex[.<span class="number">.1</span>] <span class="keyword">is</span> <span class="string">'/*'</span> <span class="keyword">then</span> <span class="property">@error</span> <span class="string">'regular expressions cannot begin with `*`'</span> <span class="keyword">if</span> regex <span class="keyword">is</span> <span class="string">'//'</span> <span class="keyword">then</span> regex = <span class="string">'/(?:)/'</span> <span class="property">@token</span> <span class="string">'REGEX'</span>, <span class="string">"<span class="subst">#{regex}</span><span class="subst">#{flags}</span>"</span>, <span class="number">0</span>, match.length match.length</pre></div></div> </li> <li id="section-20"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-20">¶</a> </div> <p>Matches multiline extended regular expressions. </p> </div> <div class="content"><div class='highlight'><pre> heregexToken: (match) -> [heregex, body, flags] = match if 0 > body.indexOf '#{' re = body.replace(HEREGEX_OMIT, '').replace(/\//g, '\\/') if re.match /^\*/ then @error 'regular expressions cannot begin with `*`' @token 'REGEX', "/#{ re or '(?:)' }/#{flags}", 0, heregex.length return heregex.length @token 'IDENTIFIER', 'RegExp', 0, 0 @token 'CALL_START', '(', 0, 0 tokens = [] for token in @interpolateString(body, regex: yes) [tag, value] = token if tag is 'TOKENS' tokens.push value... else if tag is 'NEOSTRING' continue unless value = value.replace HEREGEX_OMIT, ''</pre></div></div> </li> <li id="section-21"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-21">¶</a> </div> <p>Convert NEOSTRING into STRING </p> </div> <div class="content"><div class='highlight'><pre> value = value.replace <span class="regexp">/\\/g</span>, <span class="string">'\\\\'</span> token[<span class="number">0</span>] = <span class="string">'STRING'</span> token[<span class="number">1</span>] = <span class="property">@makeString</span>(value, <span class="string">'"'</span>, <span class="literal">yes</span>) tokens.push token <span class="keyword">else</span> <span class="property">@error</span> <span class="string">"Unexpected <span class="subst">#{tag}</span>"</span> prev = last <span class="property">@tokens</span> plusToken = [<span class="string">'+'</span>, <span class="string">'+'</span>] plusToken[<span class="number">2</span>] = prev[<span class="number">2</span>] <span class="comment"># Copy location data</span> tokens.push plusToken</pre></div></div> </li> <li id="section-22"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-22">¶</a> </div> <p>Remove the extra "+" </p> </div> <div class="content"><div class='highlight'><pre> tokens.pop() <span class="keyword">unless</span> tokens[<span class="number">0</span>]?[<span class="number">0</span>] <span class="keyword">is</span> <span class="string">'STRING'</span> <span class="property">@token</span> <span class="string">'STRING'</span>, <span class="string">'""'</span>, <span class="number">0</span>, <span class="number">0</span> <span class="property">@token</span> <span class="string">'+'</span>, <span class="string">'+'</span>, <span class="number">0</span>, <span class="number">0</span> <span class="property">@tokens</span>.push tokens... <span class="keyword">if</span> flags</pre></div></div> </li> <li id="section-23"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-23">¶</a> </div> <p>Find the flags in the heregex </p> </div> <div class="content"><div class='highlight'><pre> flagsOffset = heregex.lastIndexOf flags <span class="property">@token</span> <span class="string">','</span>, <span class="string">','</span>, flagsOffset, <span class="number">0</span> <span class="property">@token</span> <span class="string">'STRING'</span>, <span class="string">'"'</span> + flags + <span class="string">'"'</span>, flagsOffset, flags.length <span class="property">@token</span> <span class="string">')'</span>, <span class="string">')'</span>, heregex.length-<span class="number">1</span>, <span class="number">0</span> heregex.length</pre></div></div> </li> <li id="section-24"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-24">¶</a> </div> <p>Matches newlines, indents, and outdents, and determines which is which. If we can detect that the current line is continued onto the the next line, then the newline is suppressed: </p> <pre><code>elements .each( ... ) .map( ... )</code></pre> <p>Keeps track of the level of indentation, because a single outdent token can close multiple indents, so we need to know how far in we happen to be. </p> </div> <div class="content"><div class='highlight'><pre> lineToken: -> <span class="keyword">return</span> <span class="number">0</span> <span class="keyword">unless</span> match = MULTI_DENT.exec <span class="property">@chunk</span> indent = match[<span class="number">0</span>] <span class="property">@seenFor</span> = <span class="literal">no</span> size = indent.length - <span class="number">1</span> - indent.lastIndexOf <span class="string">'\n'</span> noNewlines = <span class="property">@unfinished</span>() <span class="keyword">if</span> size - <span class="property">@indebt</span> <span class="keyword">is</span> <span class="property">@indent</span> <span class="keyword">if</span> noNewlines <span class="keyword">then</span> <span class="property">@suppressNewlines</span>() <span class="keyword">else</span> <span class="property">@newlineToken</span> <span class="number">0</span> <span class="keyword">return</span> indent.length <span class="keyword">if</span> size > <span class="property">@indent</span> <span class="keyword">if</span> noNewlines <span class="property">@indebt</span> = size - <span class="property">@indent</span> <span class="property">@suppressNewlines</span>() <span class="keyword">return</span> indent.length diff = size - <span class="property">@indent</span> + <span class="property">@outdebt</span> <span class="property">@token</span> <span class="string">'INDENT'</span>, diff, indent.length - size, size <span class="property">@indents</span>.push diff <span class="property">@ends</span>.push <span class="string">'OUTDENT'</span> <span class="property">@outdebt</span> = <span class="property">@indebt</span> = <span class="number">0</span> <span class="keyword">else</span> <span class="property">@indebt</span> = <span class="number">0</span> <span class="property">@outdentToken</span> <span class="property">@indent</span> - size, noNewlines, indent.length <span class="property">@indent</span> = size indent.length</pre></div></div> </li> <li id="section-25"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-25">¶</a> </div> <p>Record an outdent token or multiple tokens, if we happen to be moving back inwards past several recorded indents. </p> </div> <div class="content"><div class='highlight'><pre> outdentToken: (moveOut, noNewlines, outdentLength) -> <span class="keyword">while</span> moveOut > <span class="number">0</span> len = <span class="property">@indents</span>.length - <span class="number">1</span> <span class="keyword">if</span> <span class="property">@indents</span>[len] <span class="keyword">is</span> <span class="literal">undefined</span> moveOut = <span class="number">0</span> <span class="keyword">else</span> <span class="keyword">if</span> <span class="property">@indents</span>[len] <span class="keyword">is</span> <span class="property">@outdebt</span> moveOut -= <span class="property">@outdebt</span> <span class="property">@outdebt</span> = <span class="number">0</span> <span class="keyword">else</span> <span class="keyword">if</span> <span class="property">@indents</span>[len] < <span class="property">@outdebt</span> <span class="property">@outdebt</span> -= <span class="property">@indents</span>[len] moveOut -= <span class="property">@indents</span>[len] <span class="keyword">else</span> dent = <span class="property">@indents</span>.pop() + <span class="property">@outdebt</span> moveOut -= dent <span class="property">@outdebt</span> = <span class="number">0</span> <span class="property">@pair</span> <span class="string">'OUTDENT'</span> <span class="property">@token</span> <span class="string">'OUTDENT'</span>, dent, <span class="number">0</span>, outdentLength <span class="property">@outdebt</span> -= moveOut <span class="keyword">if</span> dent <span class="property">@tokens</span>.pop() <span class="keyword">while</span> <span class="property">@value</span>() <span class="keyword">is</span> <span class="string">';'</span> <span class="property">@token</span> <span class="string">'TERMINATOR'</span>, <span class="string">'\n'</span>, outdentLength, <span class="number">0</span> <span class="keyword">unless</span> <span class="property">@tag</span>() <span class="keyword">is</span> <span class="string">'TERMINATOR'</span> <span class="keyword">or</span> noNewlines <span class="keyword">this</span></pre></div></div> </li> <li id="section-26"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-26">¶</a> </div> <p>Matches and consumes non-meaningful whitespace. Tag the previous token as being "spaced", because there are some cases where it makes a difference. </p> </div> <div class="content"><div class='highlight'><pre> whitespaceToken: -> <span class="keyword">return</span> <span class="number">0</span> <span class="keyword">unless</span> (match = WHITESPACE.exec <span class="property">@chunk</span>) <span class="keyword">or</span> (nline = <span class="property">@chunk</span>.charAt(<span class="number">0</span>) <span class="keyword">is</span> <span class="string">'\n'</span>) prev = last <span class="property">@tokens</span> prev[<span class="keyword">if</span> match <span class="keyword">then</span> <span class="string">'spaced'</span> <span class="keyword">else</span> <span class="string">'newLine'</span>] = <span class="literal">true</span> <span class="keyword">if</span> prev <span class="keyword">if</span> match <span class="keyword">then</span> match[<span class="number">0</span>].length <span class="keyword">else</span> <span class="number">0</span></pre></div></div> </li> <li id="section-27"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-27">¶</a> </div> <p>Generate a newline token. Consecutive newlines get merged together. </p> </div> <div class="content"><div class='highlight'><pre> newlineToken: (offset) -> <span class="property">@tokens</span>.pop() <span class="keyword">while</span> <span class="property">@value</span>() <span class="keyword">is</span> <span class="string">';'</span> <span class="property">@token</span> <span class="string">'TERMINATOR'</span>, <span class="string">'\n'</span>, offset, <span class="number">0</span> <span class="keyword">unless</span> <span class="property">@tag</span>() <span class="keyword">is</span> <span class="string">'TERMINATOR'</span> <span class="keyword">this</span></pre></div></div> </li> <li id="section-28"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-28">¶</a> </div> <p>Use a <code>\</code> at a line-ending to suppress the newline. The slash is removed here once its job is done. </p> </div> <div class="content"><div class='highlight'><pre> suppressNewlines: -> <span class="property">@tokens</span>.pop() <span class="keyword">if</span> <span class="property">@value</span>() <span class="keyword">is</span> <span class="string">'\\'</span> <span class="keyword">this</span></pre></div></div> </li> <li id="section-29"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-29">¶</a> </div> <p>We treat all other single characters as a token. E.g.: <code>( ) , . !</code> Multi-character operators are also literal tokens, so that Jison can assign the proper order of operations. There are some symbols that we tag specially here. <code>;</code> and newlines are both treated as a <code>TERMINATOR</code>, we distinguish parentheses that indicate a method call from regular parentheses, and so on. </p> </div> <div class="content"><div class='highlight'><pre> literalToken: -> <span class="keyword">if</span> match = OPERATOR.exec <span class="property">@chunk</span> [value] = match <span class="property">@tagParameters</span>() <span class="keyword">if</span> CODE.test value <span class="keyword">else</span> value = <span class="property">@chunk</span>.charAt <span class="number">0</span> tag = value prev = last <span class="property">@tokens</span> <span class="keyword">if</span> value <span class="keyword">is</span> <span class="string">'='</span> <span class="keyword">and</span> prev <span class="keyword">if</span> <span class="keyword">not</span> prev[<span class="number">1</span>].reserved <span class="keyword">and</span> prev[<span class="number">1</span>] <span class="keyword">in</span> JS_FORBIDDEN <span class="property">@error</span> <span class="string">"reserved word \"<span class="subst">#{@value()}</span>\" can't be assigned"</span> <span class="keyword">if</span> prev[<span class="number">1</span>] <span class="keyword">in</span> [<span class="string">'||'</span>, <span class="string">'&&'</span>] prev[<span class="number">0</span>] = <span class="string">'COMPOUND_ASSIGN'</span> prev[<span class="number">1</span>] += <span class="string">'='</span> <span class="keyword">return</span> value.length <span class="keyword">if</span> value <span class="keyword">is</span> <span class="string">';'</span> <span class="property">@seenFor</span> = <span class="literal">no</span> tag = <span class="string">'TERMINATOR'</span> <span class="keyword">else</span> <span class="keyword">if</span> value <span class="keyword">in</span> MATH <span class="keyword">then</span> tag = <span class="string">'MATH'</span> <span class="keyword">else</span> <span class="keyword">if</span> value <span class="keyword">in</span> COMPARE <span class="keyword">then</span> tag = <span class="string">'COMPARE'</span> <span class="keyword">else</span> <span class="keyword">if</span> value <span class="keyword">in</span> COMPOUND_ASSIGN <span class="keyword">then</span> tag = <span class="string">'COMPOUND_ASSIGN'</span> <span class="keyword">else</span> <span class="keyword">if</span> value <span class="keyword">in</span> UNARY <span class="keyword">then</span> tag = <span class="string">'UNARY'</span> <span class="keyword">else</span> <span class="keyword">if</span> value <span class="keyword">in</span> SHIFT <span class="keyword">then</span> tag = <span class="string">'SHIFT'</span> <span class="keyword">else</span> <span class="keyword">if</span> value <span class="keyword">in</span> LOGIC <span class="keyword">or</span> value <span class="keyword">is</span> <span class="string">'?'</span> <span class="keyword">and</span> prev?.spaced <span class="keyword">then</span> tag = <span class="string">'LOGIC'</span> <span class="keyword">else</span> <span class="keyword">if</span> prev <span class="keyword">and</span> <span class="keyword">not</span> prev.spaced <span class="keyword">if</span> value <span class="keyword">is</span> <span class="string">'('</span> <span class="keyword">and</span> prev[<span class="number">0</span>] <span class="keyword">in</span> CALLABLE prev[<span class="number">0</span>] = <span class="string">'FUNC_EXIST'</span> <span class="keyword">if</span> prev[<span class="number">0</span>] <span class="keyword">is</span> <span class="string">'?'</span> tag = <span class="string">'CALL_START'</span> <span class="keyword">else</span> <span class="keyword">if</span> value <span class="keyword">is</span> <span class="string">'['</span> <span class="keyword">and</span> prev[<span class="number">0</span>] <span class="keyword">in</span> INDEXABLE tag = <span class="string">'INDEX_START'</span> <span class="keyword">switch</span> prev[<span class="number">0</span>] <span class="keyword">when</span> <span class="string">'?'</span> <span class="keyword">then</span> prev[<span class="number">0</span>] = <span class="string">'INDEX_SOAK'</span> <span class="keyword">switch</span> value <span class="keyword">when</span> <span class="string">'('</span>, <span class="string">'{'</span>, <span class="string">'['</span> <span class="keyword">then</span> <span class="property">@ends</span>.push INVERSES[value] <span class="keyword">when</span> <span class="string">')'</span>, <span class="string">'}'</span>, <span class="string">']'</span> <span class="keyword">then</span> <span class="property">@pair</span> value <span class="property">@token</span> tag, value value.length</pre></div></div> </li> <li id="section-30"> <div class="annotation"> <div class="pilwrap for-h2"> <a class="pilcrow" href="#section-30">¶</a> </div> <h2>Token Manipulators</h2> </div> </li> <li id="section-31"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-31">¶</a> </div> </div> </li> <li id="section-32"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-32">¶</a> </div> <p>Sanitize a heredoc or herecomment by erasing all external indentation on the left-hand side. </p> </div> <div class="content"><div class='highlight'><pre> sanitizeHeredoc: (doc, options) -> {indent, herecomment} = options <span class="keyword">if</span> herecomment <span class="keyword">if</span> HEREDOC_ILLEGAL.test doc <span class="property">@error</span> <span class="string">"block comment cannot contain \"*/\", starting"</span> <span class="keyword">return</span> doc <span class="keyword">if</span> doc.indexOf(<span class="string">'\n'</span>) < <span class="number">0</span> <span class="keyword">else</span> <span class="keyword">while</span> match = HEREDOC_INDENT.exec doc attempt = match[<span class="number">1</span>] indent = attempt <span class="keyword">if</span> indent <span class="keyword">is</span> <span class="literal">null</span> <span class="keyword">or</span> <span class="number">0</span> < attempt.length < indent.length doc = doc.replace <span class="regexp">/// \n <span class="comment">#{indent} ///g, '\n' if indent</span> doc = doc.replace /^\n/, '' unless herecomment doc</pre></div></div> </li> <li id="section-33"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-33">¶</a> </div> <p>A source of ambiguity in our grammar used to be parameter lists in function definitions versus argument lists in function calls. Walk backwards, tagging parameters specially in order to make things easier for the parser. </p> </div> <div class="content"><div class='highlight'><pre> tagParameters: -> <span class="keyword">return</span> <span class="keyword">this</span> <span class="keyword">if</span> <span class="property">@tag</span>() <span class="keyword">isnt</span> <span class="string">')'</span> stack = [] {tokens} = <span class="keyword">this</span> i = tokens.length tokens[--i][<span class="number">0</span>] = <span class="string">'PARAM_END'</span> <span class="keyword">while</span> tok = tokens[--i] <span class="keyword">switch</span> tok[<span class="number">0</span>] <span class="keyword">when</span> <span class="string">')'</span> stack.push tok <span class="keyword">when</span> <span class="string">'('</span>, <span class="string">'CALL_START'</span> <span class="keyword">if</span> stack.length <span class="keyword">then</span> stack.pop() <span class="keyword">else</span> <span class="keyword">if</span> tok[<span class="number">0</span>] <span class="keyword">is</span> <span class="string">'('</span> tok[<span class="number">0</span>] = <span class="string">'PARAM_START'</span> <span class="keyword">return</span> <span class="keyword">this</span> <span class="keyword">else</span> <span class="keyword">return</span> <span class="keyword">this</span> <span class="keyword">this</span></pre></div></div> </li> <li id="section-34"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-34">¶</a> </div> <p>Close up all remaining open blocks at the end of the file. </p> </div> <div class="content"><div class='highlight'><pre> closeIndentation: -> <span class="property">@outdentToken</span> <span class="property">@indent</span></pre></div></div> </li> <li id="section-35"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-35">¶</a> </div> <p>Matches a balanced group such as a single or double-quoted string. Pass in a series of delimiters, all of which must be nested correctly within the contents of the string. This method allows us to have strings within interpolations within strings, ad infinitum. </p> </div> <div class="content"><div class='highlight'><pre> balancedString: (str, end) -> continueCount = <span class="number">0</span> stack = [end] <span class="keyword">for</span> i <span class="keyword">in</span> [<span class="number">1.</span>..str.length] <span class="keyword">if</span> continueCount --continueCount <span class="keyword">continue</span> <span class="keyword">switch</span> letter = str.charAt i <span class="keyword">when</span> <span class="string">'\\'</span> ++continueCount <span class="keyword">continue</span> <span class="keyword">when</span> end stack.pop() <span class="keyword">unless</span> stack.length <span class="keyword">return</span> str[<span class="number">0.</span>.i] end = stack[stack.length - <span class="number">1</span>] <span class="keyword">continue</span> <span class="keyword">if</span> end <span class="keyword">is</span> <span class="string">'}'</span> <span class="keyword">and</span> letter <span class="keyword">in</span> [<span class="string">'"'</span>, <span class="string">"'"</span>] stack.push end = letter <span class="keyword">else</span> <span class="keyword">if</span> end <span class="keyword">is</span> <span class="string">'}'</span> <span class="keyword">and</span> letter <span class="keyword">is</span> <span class="string">'/'</span> <span class="keyword">and</span> match = (HEREGEX.exec(str[i..]) <span class="keyword">or</span> REGEX.exec(str[i..])) continueCount += match[<span class="number">0</span>].length - <span class="number">1</span> <span class="keyword">else</span> <span class="keyword">if</span> end <span class="keyword">is</span> <span class="string">'}'</span> <span class="keyword">and</span> letter <span class="keyword">is</span> <span class="string">'{'</span> stack.push end = <span class="string">'}'</span> <span class="keyword">else</span> <span class="keyword">if</span> end <span class="keyword">is</span> <span class="string">'"'</span> <span class="keyword">and</span> prev <span class="keyword">is</span> <span class="string">'#'</span> <span class="keyword">and</span> letter <span class="keyword">is</span> <span class="string">'{'</span> stack.push end = <span class="string">'}'</span> prev = letter <span class="property">@error</span> <span class="string">"missing <span class="subst">#{ stack.pop() }</span>, starting"</span></pre></div></div> </li> <li id="section-36"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-36">¶</a> </div> <p>Expand variables and expressions inside double-quoted strings using Ruby-like notation for substitution of arbitrary expressions. </p> <pre><code>"Hello #{name.capitalize()}."</code></pre> <p>If it encounters an interpolation, this method will recursively create a new Lexer, tokenize the interpolated contents, and merge them into the token stream. </p> <ul> <li><code>str</code> is the start of the string contents (IE with the " or """ stripped off.)</li> <li><code>options.offsetInChunk</code> is the start of the interpolated string in the current chunk, including the " or """, etc... If not provided, this is assumed to be 0. <code>options.lexedLength</code> is the length of the interpolated string, including both the start and end quotes. Both of these values are ignored if <code>options.regex</code> is true.</li> <li><code>options.strOffset</code> is the offset of str, relative to the start of the current chunk.</li> </ul> </div> <div class="content"><div class='highlight'><pre> interpolateString: (str, options = {}) -> {heredoc, regex, offsetInChunk, strOffset, lexedLength} = options offsetInChunk = offsetInChunk || <span class="number">0</span> strOffset = strOffset || <span class="number">0</span> lexedLength = lexedLength || str.length</pre></div></div> </li> <li id="section-37"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-37">¶</a> </div> <p>Clip leading \n from heredoc </p> </div> <div class="content"><div class='highlight'><pre> <span class="keyword">if</span> heredoc <span class="keyword">and</span> str.length > <span class="number">0</span> <span class="keyword">and</span> str[<span class="number">0</span>] == <span class="string">'\n'</span> str = str[<span class="number">1.</span>..] strOffset++</pre></div></div> </li> <li id="section-38"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-38">¶</a> </div> <p>Parse the string. </p> </div> <div class="content"><div class='highlight'><pre> tokens = [] pi = <span class="number">0</span> i = -<span class="number">1</span> <span class="keyword">while</span> letter = str.charAt i += <span class="number">1</span> <span class="keyword">if</span> letter <span class="keyword">is</span> <span class="string">'\\'</span> i += <span class="number">1</span> <span class="keyword">continue</span> <span class="keyword">unless</span> letter <span class="keyword">is</span> <span class="string">'#'</span> <span class="keyword">and</span> str.charAt(i+<span class="number">1</span>) <span class="keyword">is</span> <span class="string">'{'</span> <span class="keyword">and</span> (expr = <span class="property">@balancedString</span> str[i + <span class="number">1.</span>.], <span class="string">'}'</span>) <span class="keyword">continue</span></pre></div></div> </li> <li id="section-39"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-39">¶</a> </div> <p>NEOSTRING is a fake token. This will be converted to a string below. </p> </div> <div class="content"><div class='highlight'><pre> tokens.push <span class="property">@makeToken</span>(<span class="string">'NEOSTRING'</span>, str[pi...i], strOffset + pi) <span class="keyword">if</span> pi < i inner = expr[<span class="number">1.</span>..-<span class="number">1</span>] <span class="keyword">if</span> inner.length [line, column] = <span class="property">@getLineAndColumnFromChunk</span>(strOffset + i + <span class="number">1</span>) nested = <span class="keyword">new</span> Lexer().tokenize inner, line: line, column: column, rewrite: <span class="literal">off</span> popped = nested.pop() popped = nested.shift() <span class="keyword">if</span> nested[<span class="number">0</span>]?[<span class="number">0</span>] <span class="keyword">is</span> <span class="string">'TERMINATOR'</span> <span class="keyword">if</span> len = nested.length <span class="keyword">if</span> len > <span class="number">1</span> nested.unshift <span class="property">@makeToken</span> <span class="string">'('</span>, <span class="string">'('</span>, strOffset + i + <span class="number">1</span>, <span class="number">0</span> nested.push <span class="property">@makeToken</span> <span class="string">')'</span>, <span class="string">')'</span>, strOffset + i + <span class="number">1</span> + inner.length, <span class="number">0</span></pre></div></div> </li> <li id="section-40"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-40">¶</a> </div> <p>Push a fake 'TOKENS' token, which will get turned into real tokens below. </p> </div> <div class="content"><div class='highlight'><pre> tokens.push [<span class="string">'TOKENS'</span>, nested] i += expr.length pi = i + <span class="number">1</span> tokens.push <span class="property">@makeToken</span>(<span class="string">'NEOSTRING'</span>, str[pi..], strOffset + pi) <span class="keyword">if</span> i > pi < str.length</pre></div></div> </li> <li id="section-41"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-41">¶</a> </div> <p>If regex, then return now and let the regex code deal with all these fake tokens </p> </div> <div class="content"><div class='highlight'><pre> <span class="keyword">return</span> tokens <span class="keyword">if</span> regex</pre></div></div> </li> <li id="section-42"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-42">¶</a> </div> <p>If we didn't find any tokens, then just return an empty string. </p> </div> <div class="content"><div class='highlight'><pre> <span class="keyword">return</span> <span class="property">@token</span> <span class="string">'STRING'</span>, <span class="string">'""'</span>, offsetInChunk, lexedLength <span class="keyword">unless</span> tokens.length</pre></div></div> </li> <li id="section-43"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-43">¶</a> </div> <p>If the first token is not a string, add a fake empty string to the beginning. </p> </div> <div class="content"><div class='highlight'><pre> tokens.unshift <span class="property">@makeToken</span>(<span class="string">'NEOSTRING'</span>, <span class="string">''</span>, offsetInChunk) <span class="keyword">unless</span> tokens[<span class="number">0</span>][<span class="number">0</span>] <span class="keyword">is</span> <span class="string">'NEOSTRING'</span> <span class="property">@token</span> <span class="string">'('</span>, <span class="string">'('</span>, offsetInChunk, <span class="number">0</span> <span class="keyword">if</span> interpolated = tokens.length > <span class="number">1</span></pre></div></div> </li> <li id="section-44"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-44">¶</a> </div> <p>Push all the tokens </p> </div> <div class="content"><div class='highlight'><pre> <span class="keyword">for</span> token, i <span class="keyword">in</span> tokens [tag, value] = token <span class="keyword">if</span> i</pre></div></div> </li> <li id="section-45"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-45">¶</a> </div> <p>Create a 0-length "+" token. </p> </div> <div class="content"><div class='highlight'><pre> plusToken = <span class="property">@token</span> <span class="string">'+'</span>, <span class="string">'+'</span> <span class="keyword">if</span> i locationToken = <span class="keyword">if</span> tag == <span class="string">'TOKENS'</span> <span class="keyword">then</span> value[<span class="number">0</span>] <span class="keyword">else</span> token plusToken[<span class="number">2</span>] = first_line: locationToken[<span class="number">2</span>].first_line first_column: locationToken[<span class="number">2</span>].first_column last_line: locationToken[<span class="number">2</span>].first_line last_column: locationToken[<span class="number">2</span>].first_column <span class="keyword">if</span> tag <span class="keyword">is</span> <span class="string">'TOKENS'</span></pre></div></div> </li> <li id="section-46"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-46">¶</a> </div> <p>Push all the tokens in the fake 'TOKENS' token. These already have sane location data. </p> </div> <div class="content"><div class='highlight'><pre> <span class="property">@tokens</span>.push value... <span class="keyword">else</span> <span class="keyword">if</span> tag <span class="keyword">is</span> <span class="string">'NEOSTRING'</span></pre></div></div> </li> <li id="section-47"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-47">¶</a> </div> <p>Convert NEOSTRING into STRING </p> </div> <div class="content"><div class='highlight'><pre> token[<span class="number">0</span>] = <span class="string">'STRING'</span> token[<span class="number">1</span>] = <span class="property">@makeString</span> value, <span class="string">'"'</span>, heredoc <span class="property">@tokens</span>.push token <span class="keyword">else</span> <span class="property">@error</span> <span class="string">"Unexpected <span class="subst">#{tag}</span>"</span> <span class="keyword">if</span> interpolated rparen = <span class="property">@makeToken</span> <span class="string">')'</span>, <span class="string">')'</span>, offsetInChunk + lexedLength, <span class="number">0</span> rparen.stringEnd = <span class="literal">true</span> <span class="property">@tokens</span>.push rparen tokens</pre></div></div> </li> <li id="section-48"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-48">¶</a> </div> <p>Pairs up a closing token, ensuring that all listed pairs of tokens are correctly balanced throughout the course of the token stream. </p> </div> <div class="content"><div class='highlight'><pre> pair: (tag) -> <span class="keyword">unless</span> tag <span class="keyword">is</span> wanted = last <span class="property">@ends</span> <span class="property">@error</span> <span class="string">"unmatched <span class="subst">#{tag}</span>"</span> <span class="keyword">unless</span> <span class="string">'OUTDENT'</span> <span class="keyword">is</span> wanted</pre></div></div> </li> <li id="section-49"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-49">¶</a> </div> <p>Auto-close INDENT to support syntax like this: </p> <pre><code>el.click((event) -> el.hide())</code></pre> </div> <div class="content"><div class='highlight'><pre> <span class="property">@indent</span> -= size = last <span class="property">@indents</span> <span class="property">@outdentToken</span> size, <span class="literal">true</span> <span class="keyword">return</span> <span class="property">@pair</span> tag <span class="property">@ends</span>.pop()</pre></div></div> </li> <li id="section-50"> <div class="annotation"> <div class="pilwrap for-h2"> <a class="pilcrow" href="#section-50">¶</a> </div> <h2>Helpers</h2> </div> </li> <li id="section-51"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-51">¶</a> </div> </div> </li> <li id="section-52"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-52">¶</a> </div> <p>Returns the line and column number from an offset into the current chunk. </p> <p><code>offset</code> is a number of characters into @chunk. </p> </div> <div class="content"><div class='highlight'><pre> getLineAndColumnFromChunk: (offset) -> <span class="keyword">if</span> offset <span class="keyword">is</span> <span class="number">0</span> <span class="keyword">return</span> [<span class="property">@chunkLine</span>, <span class="property">@chunkColumn</span>] <span class="keyword">if</span> offset >= <span class="property">@chunk</span>.length string = <span class="property">@chunk</span> <span class="keyword">else</span> string = <span class="property">@chunk</span>[..offset-<span class="number">1</span>] lineCount = count string, <span class="string">'\n'</span> column = <span class="property">@chunkColumn</span> <span class="keyword">if</span> lineCount > <span class="number">0</span> lines = string.split <span class="string">'\n'</span> column = last(lines).length <span class="keyword">else</span> column += string.length [<span class="property">@chunkLine</span> + lineCount, column]</pre></div></div> </li> <li id="section-53"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-53">¶</a> </div> <p>Same as "token", exception this just returns the token without adding it to the results. </p> </div> <div class="content"><div class='highlight'><pre> makeToken: (tag, value, offsetInChunk = <span class="number">0</span>, length = value.length) -> locationData = {} [locationData.first_line, locationData.first_column] = <span class="property">@getLineAndColumnFromChunk</span> offsetInChunk</pre></div></div> </li> <li id="section-54"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-54">¶</a> </div> <p>Use length - 1 for the final offset - we're supplying the last_line and the last_column, so if last_column == first_column, then we're looking at a character of length 1. </p> </div> <div class="content"><div class='highlight'><pre> lastCharacter = Math.max <span class="number">0</span>, length - <span class="number">1</span> [locationData.last_line, locationData.last_column] = <span class="property">@getLineAndColumnFromChunk</span> offsetInChunk + lastCharacter token = [tag, value, locationData] token</pre></div></div> </li> <li id="section-55"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-55">¶</a> </div> <p>Add a token to the results. <code>offset</code> is the offset into the current @chunk where the token starts. <code>length</code> is the length of the token in the @chunk, after the offset. If not specified, the length of <code>value</code> will be used. </p> <p>Returns the new token. </p> </div> <div class="content"><div class='highlight'><pre> token: (tag, value, offsetInChunk, length) -> token = <span class="property">@makeToken</span> tag, value, offsetInChunk, length <span class="property">@tokens</span>.push token token</pre></div></div> </li> <li id="section-56"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-56">¶</a> </div> <p>Peek at a tag in the current token stream. </p> </div> <div class="content"><div class='highlight'><pre> tag: (index, tag) -> (tok = last <span class="property">@tokens</span>, index) <span class="keyword">and</span> <span class="keyword">if</span> tag <span class="keyword">then</span> tok[<span class="number">0</span>] = tag <span class="keyword">else</span> tok[<span class="number">0</span>]</pre></div></div> </li> <li id="section-57"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-57">¶</a> </div> <p>Peek at a value in the current token stream. </p> </div> <div class="content"><div class='highlight'><pre> value: (index, val) -> (tok = last <span class="property">@tokens</span>, index) <span class="keyword">and</span> <span class="keyword">if</span> val <span class="keyword">then</span> tok[<span class="number">1</span>] = val <span class="keyword">else</span> tok[<span class="number">1</span>]</pre></div></div> </li> <li id="section-58"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-58">¶</a> </div> <p>Are we in the midst of an unfinished expression? </p> </div> <div class="content"><div class='highlight'><pre> unfinished: -> LINE_CONTINUER.test(<span class="property">@chunk</span>) <span class="keyword">or</span> <span class="property">@tag</span>() <span class="keyword">in</span> [<span class="string">'\\'</span>, <span class="string">'.'</span>, <span class="string">'?.'</span>, <span class="string">'?::'</span>, <span class="string">'UNARY'</span>, <span class="string">'MATH'</span>, <span class="string">'+'</span>, <span class="string">'-'</span>, <span class="string">'SHIFT'</span>, <span class="string">'RELATION'</span> <span class="string">'COMPARE'</span>, <span class="string">'LOGIC'</span>, <span class="string">'THROW'</span>, <span class="string">'EXTENDS'</span>]</pre></div></div> </li> <li id="section-59"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-59">¶</a> </div> <p>Converts newlines for string literals. </p> </div> <div class="content"><div class='highlight'><pre> escapeLines: (str, heredoc) -> str.replace MULTILINER, <span class="keyword">if</span> heredoc <span class="keyword">then</span> <span class="string">'\\n'</span> <span class="keyword">else</span> <span class="string">''</span></pre></div></div> </li> <li id="section-60"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-60">¶</a> </div> <p>Constructs a string token by escaping quotes and newlines. </p> </div> <div class="content"><div class='highlight'><pre> makeString: (body, quote, heredoc) -> <span class="keyword">return</span> quote + quote <span class="keyword">unless</span> body body = body.replace <span class="regexp">/\\([\s\S])/g</span>, (match, contents) -> <span class="keyword">if</span> contents <span class="keyword">in</span> [<span class="string">'\n'</span>, quote] <span class="keyword">then</span> contents <span class="keyword">else</span> match body = body.replace <span class="regexp">/// <span class="comment">#{quote} ///g, '\\$&'</span> quote + @escapeLines(body, heredoc) + quote</pre></div></div> </li> <li id="section-61"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-61">¶</a> </div> <p>Throws a compiler error on the current position. </p> </div> <div class="content"><div class='highlight'><pre> error: (message) -></pre></div></div> </li> <li id="section-62"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-62">¶</a> </div> <p>TODO: Are there some cases we could improve the error line number by passing the offset in the chunk where the error happened? </p> </div> <div class="content"><div class='highlight'><pre> throwSyntaxError message, first_line: <span class="property">@chunkLine</span>, first_column: <span class="property">@chunkColumn</span></pre></div></div> </li> <li id="section-63"> <div class="annotation"> <div class="pilwrap for-h2"> <a class="pilcrow" href="#section-63">¶</a> </div> <h2>Constants</h2> </div> </li> <li id="section-64"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-64">¶</a> </div> </div> </li> <li id="section-65"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-65">¶</a> </div> <p>Keywords that CoffeeScript shares in common with JavaScript. </p> </div> <div class="content"><div class='highlight'><pre>JS_KEYWORDS = [ <span class="string">'true'</span>, <span class="string">'false'</span>, <span class="string">'null'</span>, <span class="string">'this'</span> <span class="string">'new'</span>, <span class="string">'delete'</span>, <span class="string">'typeof'</span>, <span class="string">'in'</span>, <span class="string">'instanceof'</span> <span class="string">'return'</span>, <span class="string">'throw'</span>, <span class="string">'break'</span>, <span class="string">'continue'</span>, <span class="string">'debugger'</span> <span class="string">'if'</span>, <span class="string">'else'</span>, <span class="string">'switch'</span>, <span class="string">'for'</span>, <span class="string">'while'</span>, <span class="string">'do'</span>, <span class="string">'try'</span>, <span class="string">'catch'</span>, <span class="string">'finally'</span> <span class="string">'class'</span>, <span class="string">'extends'</span>, <span class="string">'super'</span> ]</pre></div></div> </li> <li id="section-66"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-66">¶</a> </div> <p>CoffeeScript-only keywords. </p> </div> <div class="content"><div class='highlight'><pre>COFFEE_KEYWORDS = [<span class="string">'undefined'</span>, <span class="string">'then'</span>, <span class="string">'unless'</span>, <span class="string">'until'</span>, <span class="string">'loop'</span>, <span class="string">'of'</span>, <span class="string">'by'</span>, <span class="string">'when'</span>] COFFEE_ALIAS_MAP = <span class="keyword">and</span> : <span class="string">'&&'</span> <span class="keyword">or</span> : <span class="string">'||'</span> <span class="keyword">is</span> : <span class="string">'=='</span> <span class="keyword">isnt</span> : <span class="string">'!='</span> <span class="keyword">not</span> : <span class="string">'!'</span> <span class="literal">yes</span> : <span class="string">'true'</span> <span class="literal">no</span> : <span class="string">'false'</span> <span class="literal">on</span> : <span class="string">'true'</span> <span class="literal">off</span> : <span class="string">'false'</span> COFFEE_ALIASES = (key <span class="keyword">for</span> key <span class="keyword">of</span> COFFEE_ALIAS_MAP) COFFEE_KEYWORDS = COFFEE_KEYWORDS.concat COFFEE_ALIASES</pre></div></div> </li> <li id="section-67"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-67">¶</a> </div> <p>The list of keywords that are reserved by JavaScript, but not used, or are used by CoffeeScript internally. We throw an error when these are encountered, to avoid having a JavaScript error at runtime. </p> </div> <div class="content"><div class='highlight'><pre>RESERVED = [ <span class="string">'case'</span>, <span class="string">'default'</span>, <span class="string">'function'</span>, <span class="string">'var'</span>, <span class="string">'void'</span>, <span class="string">'with'</span>, <span class="string">'const'</span>, <span class="string">'let'</span>, <span class="string">'enum'</span> <span class="string">'export'</span>, <span class="string">'import'</span>, <span class="string">'native'</span>, <span class="string">'__hasProp'</span>, <span class="string">'__extends'</span>, <span class="string">'__slice'</span>, <span class="string">'__bind'</span> <span class="string">'__indexOf'</span>, <span class="string">'implements'</span>, <span class="string">'interface'</span>, <span class="string">'package'</span>, <span class="string">'private'</span>, <span class="string">'protected'</span> <span class="string">'public'</span>, <span class="string">'static'</span>, <span class="string">'yield'</span> ] STRICT_PROSCRIBED = [<span class="string">'arguments'</span>, <span class="string">'eval'</span>]</pre></div></div> </li> <li id="section-68"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-68">¶</a> </div> <p>The superset of both JavaScript keywords and reserved words, none of which may be used as identifiers or properties. </p> </div> <div class="content"><div class='highlight'><pre>JS_FORBIDDEN = JS_KEYWORDS.concat(RESERVED).concat(STRICT_PROSCRIBED) exports.RESERVED = RESERVED.concat(JS_KEYWORDS).concat(COFFEE_KEYWORDS).concat(STRICT_PROSCRIBED) exports.STRICT_PROSCRIBED = STRICT_PROSCRIBED</pre></div></div> </li> <li id="section-69"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-69">¶</a> </div> <p>The character code of the nasty Microsoft madness otherwise known as the BOM. </p> </div> <div class="content"><div class='highlight'><pre>BOM = <span class="number">65279</span></pre></div></div> </li> <li id="section-70"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-70">¶</a> </div> <p>Token matching regexes. </p> </div> <div class="content"><div class='highlight'><pre>IDENTIFIER = <span class="regexp">/// ^ ( [$A-Za-z_\x7f-\uffff][$\w\x7f-\uffff]* ) ( [^\n\S]* : (?!:) )? <span class="comment"># Is this a property name?</span> ///</span> NUMBER = <span class="regexp">/// ^ 0b[01]+ | <span class="comment"># binary</span> ^ 0o[0-7]+ | <span class="comment"># octal</span> ^ 0x[\da-f]+ | <span class="comment"># hex</span> ^ \d*\.?\d+ (?:e[+-]?\d+)? <span class="comment"># decimal</span> ///</span>i HEREDOC = <span class="regexp">/// ^ ("""|''') ([\s\S]*?) (?:\n[^\n\S]*)? \1 ///</span> OPERATOR = <span class="regexp">/// ^ ( ?: [-=]> <span class="comment"># function</span> | [-+*/%<>&|^!?=]= <span class="comment"># compound assign / compare</span> | >>>=? <span class="comment"># zero-fill right shift</span> | ([-+:])\1 <span class="comment"># doubles</span> | ([&|<>])\2=? <span class="comment"># logic / shift</span> | \?(\.|::) <span class="comment"># soak access</span> | \.{2,3} <span class="comment"># range or splat</span> ) ///</span> WHITESPACE = <span class="regexp">/^[^\n\S]+/</span> COMMENT = <span class="regexp">/^###([^#][\s\S]*?)(?:###[^\n\S]*|(?:###)$)|^(?:\s*#(?!##[^#]).*)+/</span> CODE = <span class="regexp">/^[-=]>/</span> MULTI_DENT = <span class="regexp">/^(?:\n[^\n\S]*)+/</span> SIMPLESTR = <span class="regexp">/^'[^\\']*(?:\\.[^\\']*)*'/</span> JSTOKEN = <span class="regexp">/^`[^\\`]*(?:\\.[^\\`]*)*`/</span></pre></div></div> </li> <li id="section-71"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-71">¶</a> </div> <p>Regex-matching-regexes. </p> </div> <div class="content"><div class='highlight'><pre>REGEX = <span class="regexp">/// ^ (/ (?! [\s=] ) <span class="comment"># disallow leading whitespace or equals signs</span> [^ [ / \n \\ ]* <span class="comment"># every other thing</span> (?: (?: \\[\s\S] <span class="comment"># anything escaped</span> | \[ <span class="comment"># character class</span> [^ \] \n \\ ]* (?: \\[\s\S] [^ \] \n \\ ]* )* ] ) [^ [ / \n \\ ]* )* /) ([imgy]{0,4}) (?!\w) ///</span> HEREGEX = <span class="regexp">/// ^ /{3} ([\s\S]+?) /{3} ([imgy]{0,4}) (?!\w) ///</span> HEREGEX_OMIT = <span class="regexp">/\s+(?:#.*)?/g</span></pre></div></div> </li> <li id="section-72"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-72">¶</a> </div> <p>Token cleaning regexes. </p> </div> <div class="content"><div class='highlight'><pre>MULTILINER = <span class="regexp">/\n/g</span> HEREDOC_INDENT = <span class="regexp">/\n+([^\n\S]*)/g</span> HEREDOC_ILLEGAL = <span class="regexp">/\*\//</span> LINE_CONTINUER = <span class="regexp">/// ^ \s* (?: , | \??\.(?![.\d]) | :: ) ///</span> TRAILING_SPACES = <span class="regexp">/\s+$/</span></pre></div></div> </li> <li id="section-73"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-73">¶</a> </div> <p>Compound assignment tokens. </p> </div> <div class="content"><div class='highlight'><pre>COMPOUND_ASSIGN = [ <span class="string">'-='</span>, <span class="string">'+='</span>, <span class="string">'/='</span>, <span class="string">'*='</span>, <span class="string">'%='</span>, <span class="string">'||='</span>, <span class="string">'&&='</span>, <span class="string">'?='</span>, <span class="string">'<<='</span>, <span class="string">'>>='</span>, <span class="string">'>>>='</span>, <span class="string">'&='</span>, <span class="string">'^='</span>, <span class="string">'|='</span> ]</pre></div></div> </li> <li id="section-74"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-74">¶</a> </div> <p>Unary tokens. </p> </div> <div class="content"><div class='highlight'><pre>UNARY = [<span class="string">'!'</span>, <span class="string">'~'</span>, <span class="string">'NEW'</span>, <span class="string">'TYPEOF'</span>, <span class="string">'DELETE'</span>, <span class="string">'DO'</span>]</pre></div></div> </li> <li id="section-75"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-75">¶</a> </div> <p>Logical tokens. </p> </div> <div class="content"><div class='highlight'><pre>LOGIC = [<span class="string">'&&'</span>, <span class="string">'||'</span>, <span class="string">'&'</span>, <span class="string">'|'</span>, <span class="string">'^'</span>]</pre></div></div> </li> <li id="section-76"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-76">¶</a> </div> <p>Bit-shifting tokens. </p> </div> <div class="content"><div class='highlight'><pre>SHIFT = [<span class="string">'<<'</span>, <span class="string">'>>'</span>, <span class="string">'>>>'</span>]</pre></div></div> </li> <li id="section-77"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-77">¶</a> </div> <p>Comparison tokens. </p> </div> <div class="content"><div class='highlight'><pre>COMPARE = [<span class="string">'=='</span>, <span class="string">'!='</span>, <span class="string">'<'</span>, <span class="string">'>'</span>, <span class="string">'<='</span>, <span class="string">'>='</span>]</pre></div></div> </li> <li id="section-78"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-78">¶</a> </div> <p>Mathematical tokens. </p> </div> <div class="content"><div class='highlight'><pre>MATH = [<span class="string">'*'</span>, <span class="string">'/'</span>, <span class="string">'%'</span>]</pre></div></div> </li> <li id="section-79"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-79">¶</a> </div> <p>Relational tokens that are negatable with <code>not</code> prefix. </p> </div> <div class="content"><div class='highlight'><pre>RELATION = [<span class="string">'IN'</span>, <span class="string">'OF'</span>, <span class="string">'INSTANCEOF'</span>]</pre></div></div> </li> <li id="section-80"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-80">¶</a> </div> <p>Boolean tokens. </p> </div> <div class="content"><div class='highlight'><pre>BOOL = [<span class="string">'TRUE'</span>, <span class="string">'FALSE'</span>]</pre></div></div> </li> <li id="section-81"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-81">¶</a> </div> <p>Tokens which a regular expression will never immediately follow, but which a division operator might. </p> <p>See: <a href="http://www.mozilla.org/js/language/js20-2002-04/rationale/syntax.html#regular-expressions">http://www.mozilla.org/js/language/js20-2002-04/rationale/syntax.html#regular-expressions</a> </p> <p>Our list is shorter, due to sans-parentheses method calls. </p> </div> <div class="content"><div class='highlight'><pre>NOT_REGEX = [<span class="string">'NUMBER'</span>, <span class="string">'REGEX'</span>, <span class="string">'BOOL'</span>, <span class="string">'NULL'</span>, <span class="string">'UNDEFINED'</span>, <span class="string">'++'</span>, <span class="string">'--'</span>]</pre></div></div> </li> <li id="section-82"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-82">¶</a> </div> <p>If the previous token is not spaced, there are more preceding tokens that force a division parse: </p> </div> <div class="content"><div class='highlight'><pre>NOT_SPACED_REGEX = NOT_REGEX.concat <span class="string">')'</span>, <span class="string">'}'</span>, <span class="string">'THIS'</span>, <span class="string">'IDENTIFIER'</span>, <span class="string">'STRING'</span>, <span class="string">']'</span></pre></div></div> </li> <li id="section-83"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-83">¶</a> </div> <p>Tokens which could legitimately be invoked or indexed. An opening parentheses or bracket following these tokens will be recorded as the start of a function invocation or indexing operation. </p> </div> <div class="content"><div class='highlight'><pre>CALLABLE = [<span class="string">'IDENTIFIER'</span>, <span class="string">'STRING'</span>, <span class="string">'REGEX'</span>, <span class="string">')'</span>, <span class="string">']'</span>, <span class="string">'}'</span>, <span class="string">'?'</span>, <span class="string">'::'</span>, <span class="string">'@'</span>, <span class="string">'THIS'</span>, <span class="string">'SUPER'</span>] INDEXABLE = CALLABLE.concat <span class="string">'NUMBER'</span>, <span class="string">'BOOL'</span>, <span class="string">'NULL'</span>, <span class="string">'UNDEFINED'</span></pre></div></div> </li> <li id="section-84"> <div class="annotation"> <div class="pilwrap "> <a class="pilcrow" href="#section-84">¶</a> </div> <p>Tokens that, when immediately preceding a <code>WHEN</code>, indicate that the <code>WHEN</code> occurs at the start of a line. We disambiguate these from trailing whens to avoid an ambiguity in the grammar. </p> </div> <div class="content"><div class='highlight'><pre>LINE_BREAK = [<span class="string">'INDENT'</span>, <span class="string">'OUTDENT'</span>, <span class="string">'TERMINATOR'</span>]</pre></div></div> </li> </ul> </div> </body> </html>