<?xml version="1.0" encoding="utf-8" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="Docutils 0.5: http://docutils.sourceforge.net/" />
<title>Xapian Database Replication Protocol</title>
<style type="text/css">
/*
:Author: David Goodger (goodger@python.org)
:Id: $Id: html4css1.css 5196 2007-06-03 20:25:28Z wiemann $
:Copyright: This stylesheet has been placed in the public domain.
Default cascading style sheet for the HTML output of Docutils.
See http://docutils.sf.net/docs/howto/html-stylesheets.html for how to
customize this style sheet.
*/
/* used to remove borders from tables and images */
.borderless, table.borderless td, table.borderless th {
border: 0 }
table.borderless td, table.borderless th {
/* Override padding for "table.docutils td" with "! important".
The right padding separates the table cells. */
padding: 0 0.5em 0 0 ! important }
.first {
/* Override more specific margin styles with "! important". */
margin-top: 0 ! important }
.last, .with-subtitle {
margin-bottom: 0 ! important }
.hidden {
display: none }
a.toc-backref {
text-decoration: none ;
color: black }
blockquote.epigraph {
margin: 2em 5em ; }
dl.docutils dd {
margin-bottom: 0.5em }
/* Uncomment (and remove this text!) to get bold-faced definition list terms
dl.docutils dt {
font-weight: bold }
*/
div.abstract {
margin: 2em 5em }
div.abstract p.topic-title {
font-weight: bold ;
text-align: center }
div.admonition, div.attention, div.caution, div.danger, div.error,
div.hint, div.important, div.note, div.tip, div.warning {
margin: 2em ;
border: medium outset ;
padding: 1em }
div.admonition p.admonition-title, div.hint p.admonition-title,
div.important p.admonition-title, div.note p.admonition-title,
div.tip p.admonition-title {
font-weight: bold ;
font-family: sans-serif }
div.attention p.admonition-title, div.caution p.admonition-title,
div.danger p.admonition-title, div.error p.admonition-title,
div.warning p.admonition-title {
color: red ;
font-weight: bold ;
font-family: sans-serif }
/* Uncomment (and remove this text!) to get reduced vertical space in
compound paragraphs.
div.compound .compound-first, div.compound .compound-middle {
margin-bottom: 0.5em }
div.compound .compound-last, div.compound .compound-middle {
margin-top: 0.5em }
*/
div.dedication {
margin: 2em 5em ;
text-align: center ;
font-style: italic }
div.dedication p.topic-title {
font-weight: bold ;
font-style: normal }
div.figure {
margin-left: 2em ;
margin-right: 2em }
div.footer, div.header {
clear: both;
font-size: smaller }
div.line-block {
display: block ;
margin-top: 1em ;
margin-bottom: 1em }
div.line-block div.line-block {
margin-top: 0 ;
margin-bottom: 0 ;
margin-left: 1.5em }
div.sidebar {
margin: 0 0 0.5em 1em ;
border: medium outset ;
padding: 1em ;
background-color: #ffffee ;
width: 40% ;
float: right ;
clear: right }
div.sidebar p.rubric {
font-family: sans-serif ;
font-size: medium }
div.system-messages {
margin: 5em }
div.system-messages h1 {
color: red }
div.system-message {
border: medium outset ;
padding: 1em }
div.system-message p.system-message-title {
color: red ;
font-weight: bold }
div.topic {
margin: 2em }
h1.section-subtitle, h2.section-subtitle, h3.section-subtitle,
h4.section-subtitle, h5.section-subtitle, h6.section-subtitle {
margin-top: 0.4em }
h1.title {
text-align: center }
h2.subtitle {
text-align: center }
hr.docutils {
width: 75% }
img.align-left {
clear: left }
img.align-right {
clear: right }
ol.simple, ul.simple {
margin-bottom: 1em }
ol.arabic {
list-style: decimal }
ol.loweralpha {
list-style: lower-alpha }
ol.upperalpha {
list-style: upper-alpha }
ol.lowerroman {
list-style: lower-roman }
ol.upperroman {
list-style: upper-roman }
p.attribution {
text-align: right ;
margin-left: 50% }
p.caption {
font-style: italic }
p.credits {
font-style: italic ;
font-size: smaller }
p.label {
white-space: nowrap }
p.rubric {
font-weight: bold ;
font-size: larger ;
color: maroon ;
text-align: center }
p.sidebar-title {
font-family: sans-serif ;
font-weight: bold ;
font-size: larger }
p.sidebar-subtitle {
font-family: sans-serif ;
font-weight: bold }
p.topic-title {
font-weight: bold }
pre.address {
margin-bottom: 0 ;
margin-top: 0 ;
font-family: serif ;
font-size: 100% }
pre.literal-block, pre.doctest-block {
margin-left: 2em ;
margin-right: 2em }
span.classifier {
font-family: sans-serif ;
font-style: oblique }
span.classifier-delimiter {
font-family: sans-serif ;
font-weight: bold }
span.interpreted {
font-family: sans-serif }
span.option {
white-space: nowrap }
span.pre {
white-space: pre }
span.problematic {
color: red }
span.section-subtitle {
/* font-size relative to parent (h1..h6 element) */
font-size: 80% }
table.citation {
border-left: solid 1px gray;
margin-left: 1px }
table.docinfo {
margin: 2em 4em }
table.docutils {
margin-top: 0.5em ;
margin-bottom: 0.5em }
table.footnote {
border-left: solid 1px black;
margin-left: 1px }
table.docutils td, table.docutils th,
table.docinfo td, table.docinfo th {
padding-left: 0.5em ;
padding-right: 0.5em ;
vertical-align: top }
table.docutils th.field-name, table.docinfo th.docinfo-name {
font-weight: bold ;
text-align: left ;
white-space: nowrap ;
padding-left: 0 }
h1 tt.docutils, h2 tt.docutils, h3 tt.docutils,
h4 tt.docutils, h5 tt.docutils, h6 tt.docutils {
font-size: 100% }
ul.auto-toc {
list-style-type: none }
</style>
</head>
<body>
<div class="document" id="xapian-database-replication-protocol">
<h1 class="title">Xapian Database Replication Protocol</h1>
<!-- Copyright (C) 2008 Lemur Consulting Ltd -->
<!-- Copyright (C) 2010 Olly Betts -->
<div class="contents topic" id="table-of-contents">
<p class="topic-title first">Table of contents</p>
<ul class="simple">
<li><a class="reference internal" href="#protocol-description" id="id1">Protocol description</a><ul>
<li><a class="reference internal" href="#client-messages" id="id2">Client messages</a></li>
<li><a class="reference internal" href="#server-messages" id="id3">Server messages</a></li>
</ul>
</li>
<li><a class="reference internal" href="#changeset-files" id="id4">Changeset files</a></li>
</ul>
</div>
<p>This document contains details of the implementation of the replication
protocol, version 1. For details of how and why to use the replication
protocol, see the separate <a class="reference external" href="replication.html">Replication Users Guide</a>
document.</p>
<div class="section" id="protocol-description">
<h1><a class="toc-backref" href="#id1">Protocol description</a></h1>
<p>The protocol used to transfer the updates is based on the RemoteConnection
system (also used for remote Xapian databases). This provides a "message"
layer abstraction for the connections; so the communication is based on a set
of messages, each with a type, and some associated data.</p>
<p>Where the following description refers to "packed" strings or integers, this
means packed according to the same methods for packing these into databases.</p>
<div class="section" id="client-messages">
<h2><a class="toc-backref" href="#id2">Client messages</a></h2>
<p>The client sends a single message type to the server: this is a message of type
'R', and includes the name of a database to be replicated and a revision string
for that database. This message is sent whenever the client wants to receive
updates for a database.</p>
</div>
<div class="section" id="server-messages">
<h2><a class="toc-backref" href="#id3">Server messages</a></h2>
<p>The server can send a variety of messages. The message types are currently
defined in an enum in common/replicationprotocol.h (in which each type is
preceded by <tt class="docutils literal"><span class="pre">REPL_REPLY_</span></tt>):</p>
<blockquote>
<ul class="simple">
<li>END_OF_CHANGES: this indicates that no further changes are needed, and ends
the response to the original request. It contains no data.</li>
<li>FAIL: this indicates that a consistent set of changes couldn't be sent. It
may occur because the database is being changed too quickly at the senders
end, or for other reasons. It ends the response to the original request,
and may occur when any other messages are expected.</li>
<li>DB_HEADER: this indicates that an entire database copy is about to be sent.
It contains a string representing the UUID of the database which is about to
be sent, followed by a (packed) unsigned integer, representing the revision
number of the copy which is about to be sent.</li>
<li>DB_FILENAME: this contains the name of the next file to be sent in a DB copy
operation.</li>
<li>DB_FILEDATA: this contains the contents of a file in a DB copy operation.
The contents of the message are the details of the file.</li>
<li>DB_FOOTER: this indicates the end of a DB copy operation. The contents of
this message are a single (packed) unsigned integer, which represents a
revision number. The newly copied database is not safe to make live until
changesets up to the specified revision have been applied.</li>
<li>CHANGESET: this indicates that a changeset file (see below) is being sent.</li>
</ul>
</blockquote>
</div>
</div>
<div class="section" id="changeset-files">
<h1><a class="toc-backref" href="#id4">Changeset files</a></h1>
<p>Changes are represented by changeset files. When changeset logging is enabled
for a database, just before each commit a changeset file is created in
the database directory. This file contains a record of the changes made,
currently in the following format (but note that this format may change in
the future):</p>
<blockquote>
<ul class="simple">
<li>12 bytes holding the string "FlintChanges", "ChertChanges" or "BrassChanges"
(used to check that a file is a changeset file).</li>
<li>The format of the changeset (as a variable length unsigned integer).</li>
<li>The revision number of the database before the changes were applied (as a
variable length unsigned integer).</li>
<li>The revision number of the database after the changes were applied (as a
variable length unsigned integer).</li>
<li>A byte:<ul>
<li>0 if the changes can safely be applied to a live database</li>
<li>1 if the changes cannot be applied while searching is in progress. (This
will be set if the database was modified in "DANGEROUS" mode).</li>
</ul>
</li>
<li>A series of items:<ul>
<li>A byte: 0 if there are no more items in the changeset, 1 if the next item
is a base file, 2 if the next item is a list of blocks.</li>
<li>A (packed) string, holding a table name.</li>
<li>If a base file:<ul>
<li>The letter of the base file (currently 'A' or 'B').</li>
<li>The length of the file (as a variable length unsigned integer).</li>
<li>The contents of the base file.</li>
</ul>
</li>
<li>If a list of blocks:<ul>
<li>The blocksize in use.</li>
<li>A list of items:<ul>
<li>A variable length unsigned integer holding 0 if the list is at an end,
or holding (block number + 1) otherwise.</li>
<li>The contents of the block.</li>
</ul>
</li>
</ul>
</li>
</ul>
</li>
<li>A revision number that the database must be upgraded to, with more
changesets, before it is safe to be made live. This will normally be the
revision number of the database after the changes were applied, but if the
changeset was created by copying parts of the database without a read lock,
and modifications were made to the database while the copy was in progress,
parts of the copy may contain later revisions than intended - in this
situation further changesets will be needed to ensure that these parts of
the database are fully integrated.</li>
</ul>
</blockquote>
</div>
</div>
</body>
</html>
