.. -*- rst -*- .. highlightlang:: none Tokenizers ========== TODO: Write me. Here are the list of built-in tokenizers: * TokenBigram * TokenBigramSplitSymbol * TokenBigramSplitSymbolAlpha * TokenBigramSplitSymbolAlphaDigit * TokenBigramIgnoreBlank * TokenBigramIgnoreBlankSplitSymbol * TokenBigramIgnoreBlankSplitAlpha * TokenBigramIgnoreBlankSplitAlphaDigit * TokenDelimit * TokenDelimitNull * TokenTrigram * TokenUnigram
distrib
>
Fedora
>
18
>
x86_64
>
media
>
updates
>
by-pkgid
>
e450e7f3d6075c4a54de19e68d38177f
>
files
>
128
