<?xml version="1.0" encoding="utf-8" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="Docutils 0.5: http://docutils.sourceforge.net/" />
<title>Xapian::PostingSource</title>
<style type="text/css">
/*
:Author: David Goodger (goodger@python.org)
:Id: $Id: html4css1.css 5196 2007-06-03 20:25:28Z wiemann $
:Copyright: This stylesheet has been placed in the public domain.
Default cascading style sheet for the HTML output of Docutils.
See http://docutils.sf.net/docs/howto/html-stylesheets.html for how to
customize this style sheet.
*/
/* used to remove borders from tables and images */
.borderless, table.borderless td, table.borderless th {
border: 0 }
table.borderless td, table.borderless th {
/* Override padding for "table.docutils td" with "! important".
The right padding separates the table cells. */
padding: 0 0.5em 0 0 ! important }
.first {
/* Override more specific margin styles with "! important". */
margin-top: 0 ! important }
.last, .with-subtitle {
margin-bottom: 0 ! important }
.hidden {
display: none }
a.toc-backref {
text-decoration: none ;
color: black }
blockquote.epigraph {
margin: 2em 5em ; }
dl.docutils dd {
margin-bottom: 0.5em }
/* Uncomment (and remove this text!) to get bold-faced definition list terms
dl.docutils dt {
font-weight: bold }
*/
div.abstract {
margin: 2em 5em }
div.abstract p.topic-title {
font-weight: bold ;
text-align: center }
div.admonition, div.attention, div.caution, div.danger, div.error,
div.hint, div.important, div.note, div.tip, div.warning {
margin: 2em ;
border: medium outset ;
padding: 1em }
div.admonition p.admonition-title, div.hint p.admonition-title,
div.important p.admonition-title, div.note p.admonition-title,
div.tip p.admonition-title {
font-weight: bold ;
font-family: sans-serif }
div.attention p.admonition-title, div.caution p.admonition-title,
div.danger p.admonition-title, div.error p.admonition-title,
div.warning p.admonition-title {
color: red ;
font-weight: bold ;
font-family: sans-serif }
/* Uncomment (and remove this text!) to get reduced vertical space in
compound paragraphs.
div.compound .compound-first, div.compound .compound-middle {
margin-bottom: 0.5em }
div.compound .compound-last, div.compound .compound-middle {
margin-top: 0.5em }
*/
div.dedication {
margin: 2em 5em ;
text-align: center ;
font-style: italic }
div.dedication p.topic-title {
font-weight: bold ;
font-style: normal }
div.figure {
margin-left: 2em ;
margin-right: 2em }
div.footer, div.header {
clear: both;
font-size: smaller }
div.line-block {
display: block ;
margin-top: 1em ;
margin-bottom: 1em }
div.line-block div.line-block {
margin-top: 0 ;
margin-bottom: 0 ;
margin-left: 1.5em }
div.sidebar {
margin: 0 0 0.5em 1em ;
border: medium outset ;
padding: 1em ;
background-color: #ffffee ;
width: 40% ;
float: right ;
clear: right }
div.sidebar p.rubric {
font-family: sans-serif ;
font-size: medium }
div.system-messages {
margin: 5em }
div.system-messages h1 {
color: red }
div.system-message {
border: medium outset ;
padding: 1em }
div.system-message p.system-message-title {
color: red ;
font-weight: bold }
div.topic {
margin: 2em }
h1.section-subtitle, h2.section-subtitle, h3.section-subtitle,
h4.section-subtitle, h5.section-subtitle, h6.section-subtitle {
margin-top: 0.4em }
h1.title {
text-align: center }
h2.subtitle {
text-align: center }
hr.docutils {
width: 75% }
img.align-left {
clear: left }
img.align-right {
clear: right }
ol.simple, ul.simple {
margin-bottom: 1em }
ol.arabic {
list-style: decimal }
ol.loweralpha {
list-style: lower-alpha }
ol.upperalpha {
list-style: upper-alpha }
ol.lowerroman {
list-style: lower-roman }
ol.upperroman {
list-style: upper-roman }
p.attribution {
text-align: right ;
margin-left: 50% }
p.caption {
font-style: italic }
p.credits {
font-style: italic ;
font-size: smaller }
p.label {
white-space: nowrap }
p.rubric {
font-weight: bold ;
font-size: larger ;
color: maroon ;
text-align: center }
p.sidebar-title {
font-family: sans-serif ;
font-weight: bold ;
font-size: larger }
p.sidebar-subtitle {
font-family: sans-serif ;
font-weight: bold }
p.topic-title {
font-weight: bold }
pre.address {
margin-bottom: 0 ;
margin-top: 0 ;
font-family: serif ;
font-size: 100% }
pre.literal-block, pre.doctest-block {
margin-left: 2em ;
margin-right: 2em }
span.classifier {
font-family: sans-serif ;
font-style: oblique }
span.classifier-delimiter {
font-family: sans-serif ;
font-weight: bold }
span.interpreted {
font-family: sans-serif }
span.option {
white-space: nowrap }
span.pre {
white-space: pre }
span.problematic {
color: red }
span.section-subtitle {
/* font-size relative to parent (h1..h6 element) */
font-size: 80% }
table.citation {
border-left: solid 1px gray;
margin-left: 1px }
table.docinfo {
margin: 2em 4em }
table.docutils {
margin-top: 0.5em ;
margin-bottom: 0.5em }
table.footnote {
border-left: solid 1px black;
margin-left: 1px }
table.docutils td, table.docutils th,
table.docinfo td, table.docinfo th {
padding-left: 0.5em ;
padding-right: 0.5em ;
vertical-align: top }
table.docutils th.field-name, table.docinfo th.docinfo-name {
font-weight: bold ;
text-align: left ;
white-space: nowrap ;
padding-left: 0 }
h1 tt.docutils, h2 tt.docutils, h3 tt.docutils,
h4 tt.docutils, h5 tt.docutils, h6 tt.docutils {
font-size: 100% }
ul.auto-toc {
list-style-type: none }
</style>
</head>
<body>
<div class="document" id="xapian-postingsource">
<h1 class="title">Xapian::PostingSource</h1>
<!-- Copyright (C) 2008,2009,2010,2011 Olly Betts -->
<!-- Copyright (C) 2008,2009 Lemur Consulting Ltd -->
<div class="contents topic" id="table-of-contents">
<p class="topic-title first">Table of contents</p>
<ul class="simple">
<li><a class="reference internal" href="#introduction" id="id1">Introduction</a></li>
<li><a class="reference internal" href="#anatomy" id="id2">Anatomy</a></li>
<li><a class="reference internal" href="#examples" id="id3">Examples</a></li>
<li><a class="reference internal" href="#multiple-databases-and-remote-databases" id="id4">Multiple databases, and remote databases</a></li>
</ul>
</div>
<div class="section" id="introduction">
<h1><a class="toc-backref" href="#id1">Introduction</a></h1>
<p>Xapian::PostingSource is an API class which you can subclass to feed data to
Xapian's matcher. This feature can be made use of in a number of ways - for
example:</p>
<p>As a filter - a subclass could return a stream of document ids to filter a
query against.</p>
<p>As a weight boost - a subclass could return every document, but with a
varying weight so that certain documents receive a weight boost. This could
be used to prefer documents based on some external factor, such as age,
price, proximity to a physical location, link analysis score, etc.</p>
<p>As an alternative way of ranking documents - if the weighting scheme is set
to Xapian::BoolWeight, then the ranking will be entirely by the weight
returned by Xapian::PostingSource.</p>
</div>
<div class="section" id="anatomy">
<h1><a class="toc-backref" href="#id2">Anatomy</a></h1>
<p>When first constructed, a PostingSource is not tied to a particular database.
Before Xapian can get any postings (or statistics) from the source, it needs to
be supplied with a database. This is performed by the init() method, which is
passed a single parameter holding the database to use. This method will always
be called before asking for any information about the postings in the list. If
a posting source is used for multiple searches, the init() method will be
called before each search; implementations must cope with init() being called
multiple times, and should always use the database provided in the most recent
call:</p>
<pre class="literal-block">
virtual void init(const Xapian::Database & db) = 0;
</pre>
<p>Three methods return statistics independent of the iteration position.
These are upper and lower bounds for the number of documents which can
be returned, and an estimate of this number:</p>
<pre class="literal-block">
virtual Xapian::doccount get_termfreq_min() const = 0;
virtual Xapian::doccount get_termfreq_max() const = 0;
virtual Xapian::doccount get_termfreq_est() const = 0;
</pre>
<p>These methods are pure-virtual in the base class, so you have to define
them when deriving your subclass.</p>
<p>It must always be true that:</p>
<pre class="literal-block">
get_termfreq_min() <= get_termfreq_est() <= get_termfreq_max()
</pre>
<p>PostingSources must always return documents in increasing document ID order.</p>
<p>After construction, a PostingSource points to a position <em>before</em> the first
document id - so before a docid can be read, the position must be advanced.</p>
<p>The <tt class="docutils literal"><span class="pre">get_weight()</span></tt> method returns the weight that you want to contribute
to the current document. This weight must always be >= 0:</p>
<pre class="literal-block">
virtual Xapian::weight get_weight() const;
</pre>
<p>The default implementation of <tt class="docutils literal"><span class="pre">get_weight()</span></tt> returns 0, for convenience when
deriving "weight-less" subclasses.</p>
<p>You also need to specify an upper bound on the value which <tt class="docutils literal"><span class="pre">get_weight()</span></tt> can
return, which is used by the matcher to perform various optimisations. You
should try hard to find a bound for efficiency, but if there really isn't one
then you can set <tt class="docutils literal"><span class="pre">DBL_MAX</span></tt>:</p>
<pre class="literal-block">
void get_maxweight(Xapian::weight max_weight);
</pre>
<p>This method specifies an upper bound on what <tt class="docutils literal"><span class="pre">get_weight()</span></tt> will return <em>from
now on</em> (until the next call to <tt class="docutils literal"><span class="pre">init()</span></tt>). So if you know that the upper
bound has decreased, you should call <tt class="docutils literal"><span class="pre">set_maxweight()</span></tt> with the new reduced
bound.</p>
<p>One thing to be aware of is that currently calling <tt class="docutils literal"><span class="pre">set_maxweight()</span></tt> during
the match triggers an recursion through the postlist tree to recalculate the
new overall maxweight, which takes a comparable amount of time to calculating
the weight for a matching document. If your maxweight reduces for nearly
every document, you may want to profile to see if it's beneficial to notify
every single change. Experiments with a modified <tt class="docutils literal"><span class="pre">FixedWeightPostingSource</span></tt>
which forces a pointless recalculation for every document suggest a worst case
overhead in search times of about 37%, but reports of profiling results for
real world examples are most welcome. In real cases, this overhead could
easily be offset by the extra scope for matcher optimisations which a tighter
maxweight bound allows.</p>
<p>A simple approach to reducing the number of calculations is only to do it every
N documents. If it's cheap to calculate the maxweight in your posting source,
a more sophisticated strategy might be to decide an absolute maximum number of
times to update the maxweight (say 100) and then to call it whenever:</p>
<pre class="literal-block">
last_notified_maxweight - new_maxweight >= original_maxweight / 100.0
</pre>
<p>This ensures that only reasonably significant drops result in a recalculation
of the maxweight.</p>
<p>Since <tt class="docutils literal"><span class="pre">get_weight()</span></tt> must always return >= 0, the upper bound must clearly
also always be >= 0 too. If you don't call <tt class="docutils literal"><span class="pre">get_maxweight()</span></tt> then the
bound defaults to 0, to match the default implementation of <tt class="docutils literal"><span class="pre">get_weight()</span></tt>.</p>
<p>If you want to read the currently set upper bound, you can call:</p>
<pre class="literal-block">
Xapian::weight get_maxweight() const;
</pre>
<p>This is just a getter method for a member variable in the
<tt class="docutils literal"><span class="pre">Xapian::PostingSource</span></tt> class, and is inlined from the API headers, so
there's no point storing this yourself in your subclass - it should be just as
efficient to call <tt class="docutils literal"><span class="pre">get_maxweight()</span></tt> whenever you want to use it.</p>
<p>The <tt class="docutils literal"><span class="pre">at_end()</span></tt> method checks if the current iteration position is past the
last entry:</p>
<pre class="literal-block">
virtual bool at_end() const = 0;
</pre>
<p>The <tt class="docutils literal"><span class="pre">get_docid()</span></tt> method returns the document id at the current iteration
position:</p>
<pre class="literal-block">
virtual Xapian::docid get_docid() const = 0;
</pre>
<p>There are three methods which advance the current position. All of these take
a Xapian::Weight parameter <tt class="docutils literal"><span class="pre">min_wt</span></tt>, which indicates the minimum weight
contribution which the matcher is interested in. The matcher still checks
the weight of documents so it's OK to ignore this parameter completely, or to
use it to discard only some documents. But it can be useful for optimising
in some cases.</p>
<p>The simplest of these three methods is <tt class="docutils literal"><span class="pre">next()</span></tt>, which simply advances the
iteration position to the next document (possibly skipping documents with
weight contribution < min_wt):</p>
<pre class="literal-block">
virtual void next(Xapian::weight min_wt) = 0;
</pre>
<p>Then there's <tt class="docutils literal"><span class="pre">skip_to()</span></tt>. This advances the iteration position to the next
document with document id >= that specified (possibly also skipping documents
with weight contribution < min_wt):</p>
<pre class="literal-block">
virtual void skip_to(Xapian::docid did, Xapian::weight min_wt);
</pre>
<p>A default implementation of <tt class="docutils literal"><span class="pre">skip_to()</span></tt> is provided which just calls
<tt class="docutils literal"><span class="pre">next()</span></tt> repeatedly. This works but <tt class="docutils literal"><span class="pre">skip_to()</span></tt> can often be implemented
much more efficiently.</p>
<p>The final method of this group is <tt class="docutils literal"><span class="pre">check()</span></tt>. In some cases, it's fairly
cheap to check if a given document matches, but the requirement that
<tt class="docutils literal"><span class="pre">skip_to()</span></tt> must leave the iteration position on the next document is
rather costly to implement (for example, it might require linear scanning
of document ids). To avoid this where possible, the <tt class="docutils literal"><span class="pre">check()</span></tt> method
allows the matcher to just check if a given document matches:</p>
<pre class="literal-block">
virtual bool check(Xapian::docid did, Xapian::weight min_wt);
</pre>
<p>The return value is <tt class="docutils literal"><span class="pre">true</span></tt> if the method leaves the iteration position valid,
and <tt class="docutils literal"><span class="pre">false</span></tt> if it doesn't. In the latter case, <tt class="docutils literal"><span class="pre">next()</span></tt> will advance to
the first matching position after document id <tt class="docutils literal"><span class="pre">did</span></tt>, and <tt class="docutils literal"><span class="pre">skip_to()</span></tt> will
act as it would if the iteration position was the first matching position
after <tt class="docutils literal"><span class="pre">did</span></tt>.</p>
<p>The default implementation of <tt class="docutils literal"><span class="pre">check()</span></tt> is just a thin wrapper around
<tt class="docutils literal"><span class="pre">skip_to()</span></tt> which returns true - you should use this if <tt class="docutils literal"><span class="pre">skip_to()</span></tt> incurs
only a small extra cost.</p>
<p>There's also a method to return a string describing this object:</p>
<pre class="literal-block">
virtual std::string get_description() const;
</pre>
<p>The default implementation returns a generic answer. This default is provided
to avoid forcing you to provide an implementation if you don't really care
what <tt class="docutils literal"><span class="pre">get_description()</span></tt> gives for your sub-class.</p>
</div>
<div class="section" id="examples">
<h1><a class="toc-backref" href="#id3">Examples</a></h1>
<p>Here is an example of a Python PostingSource which contributes additional
weight from some external source:</p>
<pre class="literal-block">
class ExternalWeightPostingSource(xapian.PostingSource):
"""
A Xapian posting source returning weights from an external source.
"""
def __init__(self, db, wtsource):
xapian.PostingSource.__init__(self)
self.db = db
self.wtsource = wtsource
def init(self, db):
self.alldocs = db.postlist('')
def get_termfreq_min(self): return 0
def get_termfreq_est(self): return self.db.get_doccount()
def get_termfreq_max(self): return self.db.get_doccount()
def next(self, minweight):
try:
self.current = self.alldocs.next()
except StopIteration:
self.current = None
def skip_to(self, docid, minweight):
try:
self.current = self.alldocs.skip_to(docid)
except StopIteration:
self.current = None
def at_end(self):
return self.current is None
def get_docid(self):
return self.current.docid
def get_maxweight(self):
return self.wtsource.get_maxweight()
def get_weight(self):
doc = self.db.get_document(self.current.docid)
return self.wtsource.get_weight(doc)
</pre>
<p>ExternalWeightPostingSource doesn't restrict which documents match - it's
intended to be combined with an existing query using <cite>OP_AND_MAYBE</cite> like so:</p>
<pre class="literal-block">
extwtps = xapian.ExternalWeightPostingSource(db, wtsource)
query = xapian.Query(query.OP_AND_MAYBE, query, xapian.Query(extwtps))
</pre>
<p>The wtsource would be a class like this one:</p>
<pre class="literal-block">
class WeightSource:
def get_maxweight(self):
return 12.34;
def get_weight(self, doc):
return some_func(doc.get_docid())
</pre>
<!-- FIXME: Provide some more examples! -->
<!-- FIXME "why you might want to do this" (e.g. scenario) too -->
</div>
<div class="section" id="multiple-databases-and-remote-databases">
<h1><a class="toc-backref" href="#id4">Multiple databases, and remote databases</a></h1>
<p>In order to work with searches across multiple databases, or in remote
databases, some additional methods need to be implemented in your
Xapian::PostingSource subclass. The first of these is <tt class="docutils literal"><span class="pre">clone()</span></tt>, which is
used for multi database searches. This method should just return a newly
allocated instance of the same posting source class, initialised in the same
way as the source that clone() was called on. The returned source will be
deallocated by the caller (using "delete" - so you should allocate it with
"new").</p>
<p>If you don't care about supporting searches across multiple databases, you can
simply return NULL from this method. In fact, the default implementation does
this, so you can just leave the default implementation in place. If
<tt class="docutils literal"><span class="pre">clone()</span></tt> returns NULL, an attempt to perform a search with multiple
databases will raise an exception:</p>
<pre class="literal-block">
virtual PostingSource * clone() const;
</pre>
<p>To work with searches across remote databases, you need to implement a few more
methods. Firstly, you need to implement the <tt class="docutils literal"><span class="pre">name()</span></tt> method. This simply
returns the name of your posting source (fully qualified with any namespace):</p>
<pre class="literal-block">
virtual std::string name() const;
</pre>
<p>Next, you need to implement the serialise and unserialise methods. The
<tt class="docutils literal"><span class="pre">serialise()</span></tt> method converts all the settings of the PostingSource to a
string, and the <tt class="docutils literal"><span class="pre">unserialise()</span></tt> method converts one of these strings back
into a PostingSource. Note that the serialised string doesn't need to include
any information about the current iteration position of the PostingSource:</p>
<pre class="literal-block">
virtual std::string serialise() const;
virtual PostingSource * unserialise(const std::string &s) const;
</pre>
<p>Finally, you need to make a remote server which knows about your PostingSource.
Currently, the only way to do this is to modify the source slightly, and
compile your own xapian-tcpsrv. To do this, you need to edit
<tt class="docutils literal"><span class="pre">xapian-core/bin/xapian-tcpsrv.cc</span></tt> and find the
<tt class="docutils literal"><span class="pre">register_user_weighting_schemes()</span></tt> function. If <tt class="docutils literal"><span class="pre">MyPostingSource</span></tt> is your
posting source, at the end of this function, add these lines:</p>
<pre class="literal-block">
Xapian::Registry registry;
registry.register_postingsource(MyPostingSource());
server.set_registry(registry);
</pre>
</div>
</div>
</body>
</html>
