<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="X-UA-Compatible" content="IE=Edge" />
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Repository Streams — Bazaar 2.7.0 documentation</title>
<link rel="stylesheet" href="_static/classic.css" type="text/css" />
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<script type="text/javascript" id="documentation_options" data-url_root="./" src="_static/documentation_options.js"></script>
<script type="text/javascript" src="_static/jquery.js"></script>
<script type="text/javascript" src="_static/underscore.js"></script>
<script type="text/javascript" src="_static/doctools.js"></script>
<script type="text/javascript" src="_static/language_data.js"></script>
<link rel="shortcut icon" href="_static/bzr.ico"/>
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="Case Insensitive File Systems" href="case-insensitive-file-systems.html" />
<link rel="prev" title="Repositories" href="repository.html" />
<link rel="stylesheet" href="_static/bzr-doc.css" type="text/css" />
</head><body>
<div class="related" role="navigation" aria-label="related navigation">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="case-insensitive-file-systems.html" title="Case Insensitive File Systems"
accesskey="N">next</a></li>
<li class="right" >
<a href="repository.html" title="Repositories"
accesskey="P">previous</a> |</li>
<li><a href="http://bazaar.canonical.com/">
<img src="_static/bzr icon 16.png" /> Home</a> | </li>
<a href="http://doc.bazaar.canonical.com/en/">Documentation</a> | </li>
<li class="nav-item nav-item-0"><a href="index.html">Developer Document Catalog (2.7.0)</a> »</li>
<li class="nav-item nav-item-1"><a href="specifications.html" accesskey="U">Specifications</a> »</li>
</ul>
</div>
<div class="document">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body" role="main">
<div class="section" id="repository-streams">
<h1><a class="toc-backref" href="#id2">Repository Streams</a><a class="headerlink" href="#repository-streams" title="Permalink to this headline">¶</a></h1>
<div class="section" id="status">
<h2><a class="toc-backref" href="#id3">Status</a><a class="headerlink" href="#status" title="Permalink to this headline">¶</a></h2>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Date:</th><td class="field-body">2008-04-11</td>
</tr>
</tbody>
</table>
<p>This document describes the proposed programming interface for streaming
data from and into repositories. This programming interface should allow
a single interface for pulling data from and inserting data into a Bazaar
repository.</p>
<div class="contents topic" id="contents">
<p class="topic-title first">Contents</p>
<ul class="simple">
<li><a class="reference internal" href="#repository-streams" id="id2">Repository Streams</a><ul>
<li><a class="reference internal" href="#status" id="id3">Status</a></li>
<li><a class="reference internal" href="#motivation" id="id4">Motivation</a></li>
<li><a class="reference internal" href="#use-cases" id="id5">Use Cases</a><ul>
<li><a class="reference internal" href="#fetch-operations" id="id6">Fetch operations</a><ul>
<li><a class="reference internal" href="#smart-server-operations" id="id7">Smart server operations</a></li>
</ul>
</li>
<li><a class="reference internal" href="#bundles" id="id8">Bundles</a></li>
<li><a class="reference internal" href="#data-conversion" id="id9">Data conversion</a></li>
</ul>
</li>
<li><a class="reference internal" href="#characteristics" id="id10">Characteristics</a><ul>
<li><a class="reference internal" href="#single-round-trip" id="id11">Single round trip</a></li>
<li><a class="reference internal" href="#forward-only-reads" id="id12">Forward-only reads</a></li>
</ul>
</li>
<li><a class="reference internal" href="#serialisation" id="id13">Serialisation</a><ul>
<li><a class="reference internal" href="#weaves" id="id14">Weaves</a></li>
<li><a class="reference internal" href="#id1" id="id15">Bundles</a></li>
</ul>
</li>
<li><a class="reference internal" href="#specification" id="id16">Specification</a><ul>
<li><a class="reference internal" href="#requesting-a-stream" id="id17">Requesting a stream</a></li>
<li><a class="reference internal" href="#structure-of-a-stream" id="id18">Structure of a stream</a></li>
<li><a class="reference internal" href="#consuming-a-stream" id="id19">Consuming a stream</a><ul>
<li><a class="reference internal" href="#factory-metadata" id="id20">factory metadata</a></li>
</ul>
</li>
</ul>
</li>
</ul>
</li>
</ul>
</div>
</div>
<div class="section" id="motivation">
<h2><a class="toc-backref" href="#id4">Motivation</a><a class="headerlink" href="#motivation" title="Permalink to this headline">¶</a></h2>
<p>To eliminate the current requirement that extracting data from a
repository requires either using a slow format, or knowing the format of
both the source repository and the target repository.</p>
</div>
<div class="section" id="use-cases">
<h2><a class="toc-backref" href="#id5">Use Cases</a><a class="headerlink" href="#use-cases" title="Permalink to this headline">¶</a></h2>
<p>Here’s a brief description of use cases this interface is intended to
support.</p>
<div class="section" id="fetch-operations">
<h3><a class="toc-backref" href="#id6">Fetch operations</a><a class="headerlink" href="#fetch-operations" title="Permalink to this headline">¶</a></h3>
<p>We fetch data between repositories as part of push/pull/branch operations.
Fetching data is currently an very interactive process with lots of
requests. For performance having the data be supplied in a stream will
improve push and pull to remote servers. For purely local operations the
streaming logic should help reduce memory pressure. In fetch operations
we always know the formats of both the source and target.</p>
<div class="section" id="smart-server-operations">
<h4><a class="toc-backref" href="#id7">Smart server operations</a><a class="headerlink" href="#smart-server-operations" title="Permalink to this headline">¶</a></h4>
<p>With the smart server we support one streaming format, but this is only
usable when both the client and server have the same model of data, and
requires non-optimal IO ordering for pack to pack operations. Ideally we
can both provide optimal IO ordering the pack to pack case, and correct
ordering for pack to knits.</p>
</div>
</div>
<div class="section" id="bundles">
<h3><a class="toc-backref" href="#id8">Bundles</a><a class="headerlink" href="#bundles" title="Permalink to this headline">¶</a></h3>
<p>Bundles also create a stream of data for revisions from a repository.
Unlike fetch operations we do not know the format of the target at the
time the stream is created. It would be good to be able to treat bundles
as frozen branches and repositories, so a serialised stream should be
suitable for this.</p>
</div>
<div class="section" id="data-conversion">
<h3><a class="toc-backref" href="#id9">Data conversion</a><a class="headerlink" href="#data-conversion" title="Permalink to this headline">¶</a></h3>
<p>At this point we are not trying to integrate data conversion into this
interface, though it is likely possible.</p>
</div>
</div>
<div class="section" id="characteristics">
<h2><a class="toc-backref" href="#id10">Characteristics</a><a class="headerlink" href="#characteristics" title="Permalink to this headline">¶</a></h2>
<p>Some key aspects of the described interface are discussed in this section.</p>
<div class="section" id="single-round-trip">
<h3><a class="toc-backref" href="#id11">Single round trip</a><a class="headerlink" href="#single-round-trip" title="Permalink to this headline">¶</a></h3>
<p>All users of this should be able to create an appropriate stream from a
single round trip.</p>
</div>
<div class="section" id="forward-only-reads">
<h3><a class="toc-backref" href="#id12">Forward-only reads</a><a class="headerlink" href="#forward-only-reads" title="Permalink to this headline">¶</a></h3>
<p>There should be no need to seek in a stream when inserting data from it
into a repository. This places an ordering constraint on streams which
some repositories do not need.</p>
</div>
</div>
<div class="section" id="serialisation">
<h2><a class="toc-backref" href="#id13">Serialisation</a><a class="headerlink" href="#serialisation" title="Permalink to this headline">¶</a></h2>
<p>At this point serialisation of a repository stream has not been specified.
Some considerations to bear in mind about serialisation are worth noting
however.</p>
<div class="section" id="weaves">
<h3><a class="toc-backref" href="#id14">Weaves</a><a class="headerlink" href="#weaves" title="Permalink to this headline">¶</a></h3>
<p>While there shouldn’t be too many users of weave repositories anymore,
avoiding pathological behaviour when a weave is being read is a good idea.
Having the weave itself embedded in the stream is very straight forward
and does not need expensive on the fly extraction and re-diffing to take
place.</p>
</div>
<div class="section" id="id1">
<h3><a class="toc-backref" href="#id15">Bundles</a><a class="headerlink" href="#id1" title="Permalink to this headline">¶</a></h3>
<p>Being able to perform random reads from a repository stream which is a
bundle would allow stacking a bundle and a real repository together. This
will need the pack container format to be used in such a way that we can
avoid reading more data than needed within the pack container’s readv
interface.</p>
</div>
</div>
<div class="section" id="specification">
<h2><a class="toc-backref" href="#id16">Specification</a><a class="headerlink" href="#specification" title="Permalink to this headline">¶</a></h2>
<p>This describes the interface for requesting a stream, and the programming
interface a stream must provide. Streams that have been serialised should
expose the same interface.</p>
<div class="section" id="requesting-a-stream">
<h3><a class="toc-backref" href="#id17">Requesting a stream</a><a class="headerlink" href="#requesting-a-stream" title="Permalink to this headline">¶</a></h3>
<p>To request a stream, three parameters are needed:</p>
<blockquote>
<div><ul class="simple">
<li>A revision search to select the revisions to include.</li>
<li>A data ordering flag. There are two values for this - ‘unordered’ and
‘topological’. ‘unordered’ streams are useful when inserting into
repositories that have the ability to perform atomic insertions.
‘topological’ streams are useful when converting data, or when
inserting into repositories that cannot perform atomic insertions (such
as knit or weave based repositories).</li>
<li>A complete_inventory flag. When provided this flag signals the stream
generator to include all the data needed to construct the inventory of
each revision included in the stream, rather than just deltas. This is
useful when converting data from a repository with a different
inventory serialisation, as pure deltas would not be able to be
reconstructed.</li>
</ul>
</div></blockquote>
</div>
<div class="section" id="structure-of-a-stream">
<h3><a class="toc-backref" href="#id18">Structure of a stream</a><a class="headerlink" href="#structure-of-a-stream" title="Permalink to this headline">¶</a></h3>
<p>A stream is an object. It can be consistency checked via the <code class="docutils literal notranslate"><span class="pre">check</span></code>
method (which consumes the stream). The <code class="docutils literal notranslate"><span class="pre">iter_contents</span></code> method can be
used to iterate the contents of the stream. The contents of the stream are
a series of top level records, each of which contains one or more
bytestrings (potentially as a delta against another item in the
repository) and some optional metadata.</p>
</div>
<div class="section" id="consuming-a-stream">
<h3><a class="toc-backref" href="#id19">Consuming a stream</a><a class="headerlink" href="#consuming-a-stream" title="Permalink to this headline">¶</a></h3>
<p>To consume a stream, obtain an iterator from the streams
<code class="docutils literal notranslate"><span class="pre">iter_contents</span></code> method. This iterator will yield the top level records.
Each record has two attributes. One is <code class="docutils literal notranslate"><span class="pre">key_prefix</span></code> which is a tuple key
prefix for the names of each of the bytestrings in the record. The other
attribute is <code class="docutils literal notranslate"><span class="pre">entries</span></code>, an iterator of the individual items in the
record. Each item that the iterator yields is a factory which has metadata
about the entry and the ability to return the compressed bytes. This
factory can be decorated to allow obtaining different representations (for
example from a compressed knit fulltext to a plain fulltext).</p>
<p>In pseudocode:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">stream</span> <span class="o">=</span> <span class="n">repository</span><span class="o">.</span><span class="n">get_repository_stream</span><span class="p">(</span><span class="n">search</span><span class="p">,</span> <span class="n">UNORDERED</span><span class="p">,</span> <span class="kc">False</span><span class="p">)</span>
<span class="k">for</span> <span class="n">record</span> <span class="ow">in</span> <span class="n">stream</span><span class="o">.</span><span class="n">iter_contents</span><span class="p">():</span>
<span class="k">for</span> <span class="n">factory</span> <span class="ow">in</span> <span class="n">record</span><span class="o">.</span><span class="n">entries</span><span class="p">:</span>
<span class="n">compression</span> <span class="o">=</span> <span class="n">factory</span><span class="o">.</span><span class="n">storage_kind</span>
<span class="nb">print</span> <span class="s2">"Object </span><span class="si">%s</span><span class="s2">, compression type </span><span class="si">%s</span><span class="s2">, </span><span class="si">%d</span><span class="s2"> bytes long."</span> <span class="o">%</span> <span class="p">(</span>
<span class="n">record</span><span class="o">.</span><span class="n">key_prefix</span> <span class="o">+</span> <span class="n">factory</span><span class="o">.</span><span class="n">key</span><span class="p">,</span>
<span class="n">compression</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">factory</span><span class="o">.</span><span class="n">get_bytes_as</span><span class="p">(</span><span class="n">compression</span><span class="p">)))</span>
</pre></div>
</div>
<p>This structure should allow stream adapters to be written which can coerce
all records to the type of compression that a particular client needs. For
instance, inserting into weaves requires fulltexts, so a stream would be
adapted for weaves by an adapter that takes a stream, and the target
weave, and then uses the target weave to reconstruct full texts (which is
all that the weave inserter would ask for). In a similar approach, a
stream could internally delta compress many fulltexts and be able to
answer both fulltext and compressed record requests without extra IO.</p>
<div class="section" id="factory-metadata">
<h4><a class="toc-backref" href="#id20">factory metadata</a><a class="headerlink" href="#factory-metadata" title="Permalink to this headline">¶</a></h4>
<dl class="docutils">
<dt>Valid attributes on the factory are:</dt>
<dd><ul class="first last simple">
<li>sha1: Optional ascii representation of the sha1 of the bytestring (after
delta reconstruction).</li>
<li>storage_kind: Required kind of storage compression that has been used
on the bytestring. One of <code class="docutils literal notranslate"><span class="pre">mpdiff</span></code>, <code class="docutils literal notranslate"><span class="pre">knit-annotated-ft</span></code>,
<code class="docutils literal notranslate"><span class="pre">knit-annotated-delta</span></code>, <code class="docutils literal notranslate"><span class="pre">knit-ft</span></code>, <code class="docutils literal notranslate"><span class="pre">knit-delta</span></code>, <code class="docutils literal notranslate"><span class="pre">fulltext</span></code>.</li>
<li>parents: Required graph parents to associate with this bytestring.</li>
<li>compressor_data: Required opaque data relevant to the storage_kind.
(This is set to None when the compressor has no special state needed)</li>
<li>key: The key for this bytestring. Like each parent this is a tuple that
should have the key_prefix prepended to it to give the unified
repository key name.</li>
</ul>
</dd>
</dl>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="sphinxsidebar" role="navigation" aria-label="main navigation">
<div class="sphinxsidebarwrapper">
<h3><a href="index.html">Table of Contents</a></h3>
<ul>
<li><a class="reference internal" href="#">Repository Streams</a><ul>
<li><a class="reference internal" href="#status">Status</a></li>
<li><a class="reference internal" href="#motivation">Motivation</a></li>
<li><a class="reference internal" href="#use-cases">Use Cases</a><ul>
<li><a class="reference internal" href="#fetch-operations">Fetch operations</a><ul>
<li><a class="reference internal" href="#smart-server-operations">Smart server operations</a></li>
</ul>
</li>
<li><a class="reference internal" href="#bundles">Bundles</a></li>
<li><a class="reference internal" href="#data-conversion">Data conversion</a></li>
</ul>
</li>
<li><a class="reference internal" href="#characteristics">Characteristics</a><ul>
<li><a class="reference internal" href="#single-round-trip">Single round trip</a></li>
<li><a class="reference internal" href="#forward-only-reads">Forward-only reads</a></li>
</ul>
</li>
<li><a class="reference internal" href="#serialisation">Serialisation</a><ul>
<li><a class="reference internal" href="#weaves">Weaves</a></li>
<li><a class="reference internal" href="#id1">Bundles</a></li>
</ul>
</li>
<li><a class="reference internal" href="#specification">Specification</a><ul>
<li><a class="reference internal" href="#requesting-a-stream">Requesting a stream</a></li>
<li><a class="reference internal" href="#structure-of-a-stream">Structure of a stream</a></li>
<li><a class="reference internal" href="#consuming-a-stream">Consuming a stream</a><ul>
<li><a class="reference internal" href="#factory-metadata">factory metadata</a></li>
</ul>
</li>
</ul>
</li>
</ul>
</li>
</ul>
<h4>Previous topic</h4>
<p class="topless"><a href="repository.html"
title="previous chapter">Repositories</a></p>
<h4>Next topic</h4>
<p class="topless"><a href="case-insensitive-file-systems.html"
title="next chapter">Case Insensitive File Systems</a></p>
<div role="note" aria-label="source link">
<h3>This Page</h3>
<ul class="this-page-menu">
<li><a href="_sources/repository-stream.txt"
rel="nofollow">Show Source</a></li>
</ul>
</div>
<div id="searchbox" style="display: none" role="search">
<h3>Quick search</h3>
<div class="searchformwrapper">
<form class="search" action="search.html" method="get">
<input type="text" name="q" />
<input type="submit" value="Go" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div>
<script type="text/javascript">$('#searchbox').show(0);</script>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="related" role="navigation" aria-label="related navigation">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="case-insensitive-file-systems.html" title="Case Insensitive File Systems"
>next</a></li>
<li class="right" >
<a href="repository.html" title="Repositories"
>previous</a> |</li>
<li><a href="http://bazaar.canonical.com/">
<img src="_static/bzr icon 16.png" /> Home</a> | </li>
<a href="http://doc.bazaar.canonical.com/en/">Documentation</a> | </li>
<li class="nav-item nav-item-0"><a href="index.html">Developer Document Catalog (2.7.0)</a> »</li>
<li class="nav-item nav-item-1"><a href="specifications.html" >Specifications</a> »</li>
</ul>
</div>
<div class="footer" role="contentinfo">
© Copyright 2009-2011 Canonical Ltd.
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 1.8.4.
</div>
</body>
</html>
