.. -*- rst -*- .. highlightlang:: none Tokenizers ========== TODO: Write me. Here are the list of built-in tokenizers: * TokenBigram * TokenBigramSplitSymbol * TokenBigramSplitSymbolAlpha * TokenBigramSplitSymbolAlphaDigit * TokenBigramIgnoreBlank * TokenBigramIgnoreBlankSplitSymbol * TokenBigramIgnoreBlankSplitAlpha * TokenBigramIgnoreBlankSplitAlphaDigit * TokenDelimit * TokenDelimitNull * TokenTrigram * TokenUnigram