$Id: markov.txt,v 1.00 2009/12/22 12:25:59 sbajic Exp $

To implement Markovian weighting, the following pieces must be configured:

1. The storage driver. Be sure and compile using Bill Yerazunis' CRM114
   Sparse Spectra driver (hash_drv). This is the only driver that is presently
   fast enough to handle the extra data generated by the tokenizer used.

   NOTE: If you plan on doing TEFT or TUM type training, you'll need a huge
     database. In dspam.conf, HashRecMax should be set to around 5000000 
     with a HashExtentSize of around 1000000. If you run into performance
     issues, you may consider increasing this or use csscompress after training

   NOTE: Bill has told me that TOE yields the best results on real-world
     email, however for initial training TEFT or a TUNE approach might
     be best.

2. The tokenizer. Bill Yerazunis' CRM114 uses OSB/Markovian. You'll want to
   set the tokenizer to 'osb', or for old-school CRM114, sbph.

3. The value computing algorithm. This should be set to 'markov' which uses
   Markovian weighting. Comment out graham.

4. The combination algorithm (Algorithm). This should be set to 'naive' to
   act like CRM114 or you may consider 'burton' or a combination of
   "graham burton", both which gave me better results than naive. 
   Comment out any existing algorithms.

This implements the "standard" CRM114ish Markovian type discrimination, but
you could also mix and match different tokenizers and combination algorithms
if you wanted to play around. It's quite possible you may get better results
from using a different combo. The only thing that is certain is the value
computing algorithm should always be 'markov'.