<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="X-UA-Compatible" content="IE=Edge" />
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Fetching data — Bazaar 2.7.0 documentation</title>
<link rel="stylesheet" href="_static/classic.css" type="text/css" />
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<script type="text/javascript" id="documentation_options" data-url_root="./" src="_static/documentation_options.js"></script>
<script type="text/javascript" src="_static/jquery.js"></script>
<script type="text/javascript" src="_static/underscore.js"></script>
<script type="text/javascript" src="_static/doctools.js"></script>
<script type="text/javascript" src="_static/language_data.js"></script>
<link rel="shortcut icon" href="_static/bzr.ico"/>
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="Developer guide to bzrlib transports" href="transports.html" />
<link rel="prev" title="Configuring Bazaar" href="configuration.html" />
<link rel="stylesheet" href="_static/bzr-doc.css" type="text/css" />
</head><body>
<div class="related" role="navigation" aria-label="related navigation">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="transports.html" title="Developer guide to bzrlib transports"
accesskey="N">next</a></li>
<li class="right" >
<a href="configuration.html" title="Configuring Bazaar"
accesskey="P">previous</a> |</li>
<li><a href="http://bazaar.canonical.com/">
<img src="_static/bzr icon 16.png" /> Home</a> | </li>
<a href="http://doc.bazaar.canonical.com/en/">Documentation</a> | </li>
<li class="nav-item nav-item-0"><a href="index.html">Developer Document Catalog (2.7.0)</a> »</li>
</ul>
</div>
<div class="document">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body" role="main">
<div class="section" id="fetching-data">
<h1>Fetching data<a class="headerlink" href="#fetching-data" title="Permalink to this headline">¶</a></h1>
<div class="section" id="overview-of-a-fetch">
<h2>Overview of a fetch<a class="headerlink" href="#overview-of-a-fetch" title="Permalink to this headline">¶</a></h2>
<p>Inside bzr, a typical fetch happens like this:</p>
<ul class="simple">
<li>a user runs a command like <code class="docutils literal notranslate"><span class="pre">bzr</span> <span class="pre">branch</span></code> or <code class="docutils literal notranslate"><span class="pre">bzr</span> <span class="pre">pull</span></code></li>
<li><code class="docutils literal notranslate"><span class="pre">Repository.fetch</span></code> is called (by a higher-level method such as
<code class="docutils literal notranslate"><span class="pre">ControlDir.sprout</span></code>, <code class="docutils literal notranslate"><span class="pre">Branch.fetch</span></code>, etc).</li>
<li>An <code class="docutils literal notranslate"><span class="pre">InterRepository</span></code> object is created. The exact implementation of
<code class="docutils literal notranslate"><span class="pre">InterRepository</span></code> chosen depends on the format/capabilities of the
source and target repos.</li>
<li>The source and target repositories are compared to determine which data
needs to be transferred.</li>
<li>The repository data is copied. Often this is done by creating a
<code class="docutils literal notranslate"><span class="pre">StreamSource</span></code> and <code class="docutils literal notranslate"><span class="pre">StreamSink</span></code> from the source and target
repositories and feeding the stream from the source into the sink, but
some <code class="docutils literal notranslate"><span class="pre">InterRepository</span></code> implementations do differently.</li>
</ul>
</div>
<div class="section" id="how-objects-to-be-transferred-are-determined">
<h2>How objects to be transferred are determined<a class="headerlink" href="#how-objects-to-be-transferred-are-determined" title="Permalink to this headline">¶</a></h2>
<p>See <code class="docutils literal notranslate"><span class="pre">InterRepository._walk_to_common_revisions</span></code>. The basic idea is to
do a breadth-first search in the source repository’s revision graph
(starting from the head or heads the caller asked for), and look in the
target repository to see if those revisions are already present.
Eventually this will find the common ancestors in both graphs, and thus
the set of revisions to be copied has been identified.</p>
<p>All inventories for the copied revisions need to be present (and all
parent inventories at the stacking boundary too, to support stacking).</p>
<p>All texts versions introduced by those inventories need to be transferred
(but see also stacking constraints).</p>
</div>
<div class="section" id="fetch-specs">
<h2>Fetch specs<a class="headerlink" href="#fetch-specs" title="Permalink to this headline">¶</a></h2>
<p>The most <code class="docutils literal notranslate"><span class="pre">fetch</span></code> methods accept a <code class="docutils literal notranslate"><span class="pre">fetch_spec</span></code> parameter. This is how
the caller controls what is fetched: e.g. all revisions for a given head
(that aren’t already present in the target), the full ancestry for one or
more heads, or even the full contents of the source repository.</p>
<p>The <code class="docutils literal notranslate"><span class="pre">fetch_spec</span></code> parameter is an object that implements the interface
defined by <code class="docutils literal notranslate"><span class="pre">AbstractSearchResult</span></code> in <code class="docutils literal notranslate"><span class="pre">bzrlib.graph</span></code>. It describes
which keys should be fetched. Current implementations are
<code class="docutils literal notranslate"><span class="pre">SearchResult</span></code>, <code class="docutils literal notranslate"><span class="pre">PendingAncestryResult</span></code>, <code class="docutils literal notranslate"><span class="pre">EmptySearchResult</span></code>, and
<code class="docutils literal notranslate"><span class="pre">EverythingResult</span></code>. Some have options controlling if missing revisions
cause errors or not, etc.</p>
<p>There are also some “search” objects, which can be used to conveniently
construct a search result for common cases: <code class="docutils literal notranslate"><span class="pre">EverythingNotInOther</span></code> and
<code class="docutils literal notranslate"><span class="pre">NotInOtherForRevs</span></code>. They provide an <code class="docutils literal notranslate"><span class="pre">execute</span></code> method that performs
the search and returns a search result.</p>
<p>Also, <code class="docutils literal notranslate"><span class="pre">Graph._make_breadth_first_searcher</span></code> returns an object with a
<code class="docutils literal notranslate"><span class="pre">get_result</span></code> method that returns a search result.</p>
</div>
<div class="section" id="streams">
<h2>Streams<a class="headerlink" href="#streams" title="Permalink to this headline">¶</a></h2>
<p>A <strong>stream</strong> is an iterable of (substream type, substream) pairs.
The <strong>substream type</strong> is a <code class="docutils literal notranslate"><span class="pre">str</span></code> that will be one of <code class="docutils literal notranslate"><span class="pre">texts</span></code>,
<code class="docutils literal notranslate"><span class="pre">inventories</span></code>, <code class="docutils literal notranslate"><span class="pre">inventory-deltas</span></code>, <code class="docutils literal notranslate"><span class="pre">chk_bytes</span></code>, <code class="docutils literal notranslate"><span class="pre">revisions</span></code> or
<code class="docutils literal notranslate"><span class="pre">signatures</span></code>. A <strong>substream</strong> is a record stream. The format of those
records depends on the repository format being streamed, except for
<code class="docutils literal notranslate"><span class="pre">inventory-deltas</span></code> records which are format-independent.</p>
<p>A stream source can be constructed with <code class="docutils literal notranslate"><span class="pre">repo._get_source(to_format)</span></code>,
and it provides a <code class="docutils literal notranslate"><span class="pre">get_stream(search)</span></code> method (among others). A stream
sink can be constructed with <code class="docutils literal notranslate"><span class="pre">repo._get_sink()</span></code>, and provides an
<code class="docutils literal notranslate"><span class="pre">insert_stream(stream,</span> <span class="pre">src_format,</span> <span class="pre">resume_tokens)</span></code> method (among
others).</p>
</div>
<div class="section" id="stacking-constraints">
<h2>Stacking constraints<a class="headerlink" href="#stacking-constraints" title="Permalink to this headline">¶</a></h2>
<p><strong>In short the rule is:</strong> “repositories must hold revisions’ parent
inventories and their new texts (or else all texts for those revisions).”</p>
<p>This is sometimes called “the stacking invariant.”</p>
<div class="section" id="why-that-rule">
<h3>Why that rule?<a class="headerlink" href="#why-that-rule" title="Permalink to this headline">¶</a></h3>
<p>A stacked repository needs to be capable of generating a complete stream
for the revisions it does hold without access to its fallback
repositories <a class="footnote-reference" href="#id3" id="id1">[1]</a>. “Complete” here means that the stream for a revision (or
set of revisions) can be inserted into a repository that already contains
the parent(s) of that revision, and that repository will have a fully
usable copy of that revision: a working tree can be built for that
revision, etc.</p>
<p>Assuming for a moment the stream has the necessary inventory, signature
and CHK records to have a usable revision, what texts are required to have
a usable revision? The simple way to satisfy the requirement is to have
<em>every</em> text for every revision at the stacking boundary. Thus the
revisions at the stacking boundary and all their descendants have their
texts present and so can be fully reconstructed. But this is expensive:
it implies each stacked repository much contain <em>O(tree)</em> data even for a
single revision of a 1-line change, and also implies transferring
<em>O(tree)</em> data to fetch that revision.</p>
<p>Because the goal is a usable revision <em>when added to a repository with the
parent revision(s)</em> most of those texts will be redundant. The minimal
set that is needed is just those texts that are new in the revisions in
our repository. However, we need enough inventory data to be able to
determine that set of texts. So to make this possible every revision must
have its parent inventories present so that the inventory delta between
revisions can be calculated, and of course the CHK pages associated with
that delta. In fact the entire inventory does not need to be present,
just enough of it to find the delta (assuming a repository format, like
2a, that allows only part of an inventory to be stored). Thus the stacked
repository can contain only <em>O(changes)</em> data <a class="footnote-reference" href="#id4" id="id2">[2]</a> and still deliver
complete streams of that data.</p>
<p>What about revisions at the stacking boundary with more than one parent?
All of the parent inventories must be present, as a client may ask for a
stream up to any parent, not just the left-hand parent. If any parent is
absent then all texts must be present instead. Otherwise there will be
the strange situation where some fetches of a revision will succeed and
others fail depending the precise details of the fetch.</p>
</div>
<div class="section" id="implications-for-fetching">
<h3>Implications for fetching<a class="headerlink" href="#implications-for-fetching" title="Permalink to this headline">¶</a></h3>
<p>Fetches must retrieve the records necessary to satisfy that rule. The
stream source will attempt to send the necessary records, and the stream
sink will check for any missing records and make a second fetch for just
those missing records before committing the write group.</p>
<p>Our repository implementations check this constraint is satisfied before
committing a write group, to prevent a bad stream from creating a corrupt
repository. So a fetch from a bad source (e.g. a damaged repository, or a
buggy foreign-format import) may trigger <code class="docutils literal notranslate"><span class="pre">BzrCheckError</span></code> during
<code class="docutils literal notranslate"><span class="pre">commit_write_group</span></code>.</p>
<p>To fetch from a stacked repository via a smart server, the smart client:</p>
<ul class="simple">
<li>first fetches a stream of as many of the requested revisions as possible
from the initial repository,</li>
<li>then while there are still missing revisions and untried fallback
repositories fetches the outstanding revisions from the next fallback
until either all revisions have been found (success) or the list of
fallbacks has been exhausted (failure).</li>
</ul>
<table class="docutils footnote" frame="void" id="id3" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id1">[1]</a></td><td>This is not just a theoretical concern. The smart server always
opens repositories without opening fallbacks, as it cannot assume it
can access the fallbacks that the client can.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="id4" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id2">[2]</a></td><td>Actually <em>O(changes)</em> isn’t quite right in practice. In the
current implementation the fulltext of a changed file must be
transferred, not just a delta, so a 1-line change to a 10MB file will
still transfer 10MB of text data. This is because current formats
require records’ compression parents to be present in the same
repository.</td></tr>
</tbody>
</table>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="sphinxsidebar" role="navigation" aria-label="main navigation">
<div class="sphinxsidebarwrapper">
<h3><a href="index.html">Table of Contents</a></h3>
<ul>
<li><a class="reference internal" href="#">Fetching data</a><ul>
<li><a class="reference internal" href="#overview-of-a-fetch">Overview of a fetch</a></li>
<li><a class="reference internal" href="#how-objects-to-be-transferred-are-determined">How objects to be transferred are determined</a></li>
<li><a class="reference internal" href="#fetch-specs">Fetch specs</a></li>
<li><a class="reference internal" href="#streams">Streams</a></li>
<li><a class="reference internal" href="#stacking-constraints">Stacking constraints</a><ul>
<li><a class="reference internal" href="#why-that-rule">Why that rule?</a></li>
<li><a class="reference internal" href="#implications-for-fetching">Implications for fetching</a></li>
</ul>
</li>
</ul>
</li>
</ul>
<h4>Previous topic</h4>
<p class="topless"><a href="configuration.html"
title="previous chapter">Configuring Bazaar</a></p>
<h4>Next topic</h4>
<p class="topless"><a href="transports.html"
title="next chapter">Developer guide to bzrlib transports</a></p>
<div role="note" aria-label="source link">
<h3>This Page</h3>
<ul class="this-page-menu">
<li><a href="_sources/fetch.txt"
rel="nofollow">Show Source</a></li>
</ul>
</div>
<div id="searchbox" style="display: none" role="search">
<h3>Quick search</h3>
<div class="searchformwrapper">
<form class="search" action="search.html" method="get">
<input type="text" name="q" />
<input type="submit" value="Go" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div>
<script type="text/javascript">$('#searchbox').show(0);</script>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="related" role="navigation" aria-label="related navigation">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="transports.html" title="Developer guide to bzrlib transports"
>next</a></li>
<li class="right" >
<a href="configuration.html" title="Configuring Bazaar"
>previous</a> |</li>
<li><a href="http://bazaar.canonical.com/">
<img src="_static/bzr icon 16.png" /> Home</a> | </li>
<a href="http://doc.bazaar.canonical.com/en/">Documentation</a> | </li>
<li class="nav-item nav-item-0"><a href="index.html">Developer Document Catalog (2.7.0)</a> »</li>
</ul>
</div>
<div class="footer" role="contentinfo">
© Copyright 2009-2011 Canonical Ltd.
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 1.8.4.
</div>
</body>
</html>
