<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>KnitPack repository format — Bazaar 2.6.0 documentation</title>
<link rel="stylesheet" href="_static/default.css" type="text/css" />
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT: './',
VERSION: '2.6.0',
COLLAPSE_INDEX: false,
FILE_SUFFIX: '.html',
HAS_SOURCE: true
};
</script>
<script type="text/javascript" src="_static/jquery.js"></script>
<script type="text/javascript" src="_static/underscore.js"></script>
<script type="text/javascript" src="_static/doctools.js"></script>
<link rel="shortcut icon" href="_static/bzr.ico"/>
<link rel="top" title="Bazaar 2.6.0 documentation" href="index.html" />
<link rel="up" title="Specifications" href="specifications.html" />
<link rel="next" title="Implementation notes" href="implementation-notes.html" />
<link rel="prev" title="Development repository formats" href="development-repo.html" />
<link rel="stylesheet" href="_static/bzr-doc.css" type="text/css" />
</head>
<body>
<div class="related">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="implementation-notes.html" title="Implementation notes"
accesskey="N">next</a></li>
<li class="right" >
<a href="development-repo.html" title="Development repository formats"
accesskey="P">previous</a> |</li>
<li><a href="http://bazaar.canonical.com/">
<img src="_static/bzr icon 16.png" /> Home</a> | </li>
<a href="http://doc.bazaar.canonical.com/en/">Documentation</a> | </li>
<li><a href="index.html">Developer Document Catalog (2.6.0)</a> »</li>
<li><a href="specifications.html" accesskey="U">Specifications</a> »</li>
</ul>
</div>
<div class="document">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body">
<div class="section" id="knitpack-repository-format">
<h1><a class="toc-backref" href="#id1">KnitPack repository format</a><a class="headerlink" href="#knitpack-repository-format" title="Permalink to this headline">¶</a></h1>
<div class="contents topic" id="contents">
<p class="topic-title first">Contents</p>
<ul class="simple">
<li><a class="reference internal" href="#knitpack-repository-format" id="id1">KnitPack repository format</a><ul>
<li><a class="reference internal" href="#using-knitpack-repositories" id="id2">Using KnitPack repositories</a><ul>
<li><a class="reference internal" href="#motivation" id="id3">Motivation</a></li>
<li><a class="reference internal" href="#preparation" id="id4">Preparation</a></li>
<li><a class="reference internal" href="#creating-a-new-knitpack-branch" id="id5">Creating a new knitpack branch</a></li>
<li><a class="reference internal" href="#creating-a-new-knitpack-repository" id="id6">Creating a new knitpack repository</a></li>
<li><a class="reference internal" href="#upgrading-an-existing-branch-or-repository-to-knitpack-format" id="id7">Upgrading an existing branch or repository to knitpack format</a></li>
<li><a class="reference internal" href="#starting-a-new-knitpack-branch-from-one-in-an-older-format" id="id8">Starting a new knitpack branch from one in an older format</a></li>
<li><a class="reference internal" href="#testing-packs-for-bzr-svn-users" id="id9">Testing packs for bzr-svn users</a></li>
<li><a class="reference internal" href="#reporting-problems" id="id10">Reporting problems</a></li>
</ul>
</li>
<li><a class="reference internal" href="#technical-notes" id="id11">Technical notes</a></li>
</ul>
</li>
</ul>
</div>
<div class="section" id="using-knitpack-repositories">
<h2><a class="toc-backref" href="#id2">Using KnitPack repositories</a><a class="headerlink" href="#using-knitpack-repositories" title="Permalink to this headline">¶</a></h2>
<div class="section" id="motivation">
<h3><a class="toc-backref" href="#id3">Motivation</a><a class="headerlink" href="#motivation" title="Permalink to this headline">¶</a></h3>
<p>KnitPack is a new repository format for Bazaar, which is expected to be
faster both locally and over the network, is usually more compact, and
will work with more FTP servers.</p>
<p>Our benchmarking results to date have been very promising. We fully expect
to make a pack-based format the default in the near future. We would
therefore like as many people as possible using KnitPack repositories,
benchmarking the results and telling us where improvements are still needed.</p>
</div>
<div class="section" id="preparation">
<h3><a class="toc-backref" href="#id4">Preparation</a><a class="headerlink" href="#preparation" title="Permalink to this headline">¶</a></h3>
<p>A small percentage of existing repositories may have some inconsistent
data within them. It’s is a good idea to check the integrity of your
repositories before migrating them to knitpack format. To do this, run:</p>
<div class="highlight-python"><div class="highlight"><pre>bzr check
</pre></div>
</div>
<p>If that reports a problem, run this command:</p>
<div class="highlight-python"><div class="highlight"><pre>bzr reconcile
</pre></div>
</div>
<p>Note that this can take many hours for repositories with deep history
so be sure to set aside some time for this if it is required.</p>
</div>
<div class="section" id="creating-a-new-knitpack-branch">
<h3><a class="toc-backref" href="#id5">Creating a new knitpack branch</a><a class="headerlink" href="#creating-a-new-knitpack-branch" title="Permalink to this headline">¶</a></h3>
<p>If you’re starting a project from scratch, it’s easy to make it a
<tt class="docutils literal"><span class="pre">knitpack</span></tt> one. Here’s how:</p>
<div class="highlight-python"><div class="highlight"><pre>cd my-stuff
bzr init --pack-0.92
bzr add
bzr commit -m "initial import"
</pre></div>
</div>
<p>In other words, use the normal sequence of commands but add the
<tt class="docutils literal"><span class="pre">--pack-0.92</span></tt> option to the <tt class="docutils literal"><span class="pre">init</span></tt> command.</p>
<p><strong>Note:</strong> In bzr 0.92, this format was called <tt class="docutils literal"><span class="pre">knitpack-experimental</span></tt>.</p>
</div>
<div class="section" id="creating-a-new-knitpack-repository">
<h3><a class="toc-backref" href="#id6">Creating a new knitpack repository</a><a class="headerlink" href="#creating-a-new-knitpack-repository" title="Permalink to this headline">¶</a></h3>
<p>If you’re starting a project from scratch and wish to use a shared repository
for branches, you can make it a <tt class="docutils literal"><span class="pre">knitpack</span></tt> repository like this:</p>
<div class="highlight-python"><div class="highlight"><pre>cd my-repo
bzr init-repo --pack-0.92 .
cd my-stuff
bzr init
bzr add
bzr commit -m "initial import"
</pre></div>
</div>
<p>In other words, use the normal sequence of commands but add the
<tt class="docutils literal"><span class="pre">--pack-0.92</span></tt> option to the <tt class="docutils literal"><span class="pre">init-repo</span></tt> command.</p>
</div>
<div class="section" id="upgrading-an-existing-branch-or-repository-to-knitpack-format">
<h3><a class="toc-backref" href="#id7">Upgrading an existing branch or repository to knitpack format</a><a class="headerlink" href="#upgrading-an-existing-branch-or-repository-to-knitpack-format" title="Permalink to this headline">¶</a></h3>
<p>If you have an existing branch and wish to migrate it to
a <tt class="docutils literal"><span class="pre">knitpack</span></tt> format, use the <tt class="docutils literal"><span class="pre">upgrade</span></tt> command like this:</p>
<div class="highlight-python"><div class="highlight"><pre>bzr upgrade --pack-0.92 path-to-my-branch
</pre></div>
</div>
<p>If you are using a shared repository, run:</p>
<div class="highlight-python"><div class="highlight"><pre>bzr upgrade --pack-0.92 ROOT_OF_REPOSITORY
</pre></div>
</div>
<p>to upgrade the history database. Note that this will not
alter the branch format of each branch, so
you will need to also upgrade each branch individually
if you are upgrading from an old (e.g. < 0.17) bzr.
More modern bzr’s will already have the branch format at
our latest branch format which adds support for tags.</p>
</div>
<div class="section" id="starting-a-new-knitpack-branch-from-one-in-an-older-format">
<h3><a class="toc-backref" href="#id8">Starting a new knitpack branch from one in an older format</a><a class="headerlink" href="#starting-a-new-knitpack-branch-from-one-in-an-older-format" title="Permalink to this headline">¶</a></h3>
<p>This can be done in one of several ways:</p>
<ol class="arabic simple">
<li>Create a new branch and pull into it</li>
<li>Create a standalone branch and upgrade its format</li>
<li>Create a knitpack shared repository and branch into it</li>
</ol>
<p>Here are the commands for using the <tt class="docutils literal"><span class="pre">pull</span></tt> approach:</p>
<div class="highlight-python"><div class="highlight"><pre>bzr init --pack-0.92 my-new-branch
cd my-new-branch
bzr pull my-source-branch
</pre></div>
</div>
<p>Here are the commands for using the <tt class="docutils literal"><span class="pre">upgrade</span></tt> approach:</p>
<div class="highlight-python"><div class="highlight"><pre>bzr branch my-source-branch my-new-branch
cd my-new-branch
bzr upgrade --pack-0.92 .
</pre></div>
</div>
<p>Here are the commands for the shared repository approach:</p>
<div class="highlight-python"><div class="highlight"><pre>cd my-repo
bzr init-repo --pack-0.92 .
bzr branch my-source-branch my-new-branch
cd my-new-branch
</pre></div>
</div>
<p>As a reminder, any of the above approaches can fail if the source branch
has inconsistent data within it and hasn’t been reconciled yet. Please
be sure to check that before reporting problems.</p>
</div>
<div class="section" id="testing-packs-for-bzr-svn-users">
<h3><a class="toc-backref" href="#id9">Testing packs for bzr-svn users</a><a class="headerlink" href="#testing-packs-for-bzr-svn-users" title="Permalink to this headline">¶</a></h3>
<p>If you are using <tt class="docutils literal"><span class="pre">bzr-svn</span></tt> or are testing the prototype subtree support,
you can still use and assist in testing KnitPacks. The commands to use
are identical to the ones given above except that the name of the format
to use is <tt class="docutils literal"><span class="pre">knitpack-subtree-experimental</span></tt>.</p>
<p>WARNING: Note that the subtree formats, <tt class="docutils literal"><span class="pre">dirstate-subtree</span></tt> and
<tt class="docutils literal"><span class="pre">knitpack-subtree-experimental</span></tt>, are <strong>not</strong> production strength yet and
may cause unexpected problems. They are required for the bzr-svn
plug-in but should otherwise only be used by people happy to live on the
bleeding edge. If you are using bzr-svn, you’re on the bleeding edge anyway.
:-)</p>
</div>
<div class="section" id="reporting-problems">
<h3><a class="toc-backref" href="#id10">Reporting problems</a><a class="headerlink" href="#reporting-problems" title="Permalink to this headline">¶</a></h3>
<p>If you need any help or encounter any problems, please contact the developers
via the usual ways, i.e. chat to us on IRC or send a message to our mailing
list. See <a class="reference external" href="http://wiki.bazaar.canonical.com/BzrSupport">http://wiki.bazaar.canonical.com/BzrSupport</a> for contact details.</p>
</div>
</div>
<div class="section" id="technical-notes">
<h2><a class="toc-backref" href="#id11">Technical notes</a><a class="headerlink" href="#technical-notes" title="Permalink to this headline">¶</a></h2>
<p>Bazaar 0.92 adds a new format (experimental at first) implemented in
<tt class="docutils literal"><span class="pre">bzrlib.repofmt.pack_repo.py</span></tt>.</p>
<p>This format provides a knit-like interface which is quite compatible
with knit format repositories: you can get a VersionedFile for a
particular file-id, or for revisions, or for the inventory, even though
these do not correspond to single files on disk.</p>
<p>The on-disk format is that the repository directory contains these
files and subdirectories:</p>
<table border="1" class="docutils">
<colgroup>
<col width="31%" />
<col width="69%" />
</colgroup>
<tbody valign="top">
<tr class="row-odd"><td>packs/</td>
<td>completed readonly packs</td>
</tr>
<tr class="row-even"><td>indices/</td>
<td>indices for completed packs</td>
</tr>
<tr class="row-odd"><td>upload/</td>
<td>temporary files for packs currently being
written</td>
</tr>
<tr class="row-even"><td>obsolete_packs/</td>
<td>packs that have been repacked and are no
longer normally needed</td>
</tr>
<tr class="row-odd"><td>pack-names</td>
<td>index of all live packs</td>
</tr>
<tr class="row-even"><td>lock/</td>
<td>lockdir</td>
</tr>
</tbody>
</table>
<p>Note that for consistency we always write “indices” not “indexes”.</p>
<p>This is implemented on top of pack files, which are written once from
start to end, then left alone. A pack consists of a body file, plus
several index files. There are four index files for each pack, which
have the same basename and an extension indicating the purpose of the
index:</p>
<table border="1" class="docutils">
<colgroup>
<col width="12%" />
<col width="15%" />
<col width="35%" />
<col width="38%" />
</colgroup>
<thead valign="bottom">
<tr class="row-odd"><th class="head">extn</th>
<th class="head">Purpose</th>
<th class="head">Key</th>
<th class="head">References</th>
</tr>
</thead>
<tbody valign="top">
<tr class="row-even"><td><tt class="docutils literal"><span class="pre">.tix</span></tt></td>
<td>File texts</td>
<td><tt class="docutils literal"><span class="pre">file_id,</span> <span class="pre">revision_id</span></tt></td>
<td>per-file parents,
compression basis
per-file parents</td>
</tr>
<tr class="row-odd"><td><tt class="docutils literal"><span class="pre">.six</span></tt></td>
<td>Signatures</td>
<td><tt class="docutils literal"><span class="pre">revision_id,</span></tt></td>
<td><ul class="first last simple">
<li></li>
</ul>
</td>
</tr>
<tr class="row-even"><td><tt class="docutils literal"><span class="pre">.rix</span></tt></td>
<td>Revisions</td>
<td><tt class="docutils literal"><span class="pre">revision_id,</span></tt></td>
<td>revision parents</td>
</tr>
<tr class="row-odd"><td><tt class="docutils literal"><span class="pre">.iix</span></tt></td>
<td>Inventory</td>
<td><tt class="docutils literal"><span class="pre">revision_id,</span></tt></td>
<td>revision parents,
compression base</td>
</tr>
</tbody>
</table>
<p>Indices are accessed through the <tt class="docutils literal"><span class="pre">bzrlib.index.GraphIndex</span></tt> class.
Indices are stored as sorted files on disk. Each line is one record,
and contains:</p>
<blockquote>
<div><ul class="simple">
<li>key fields</li>
<li>a value string - for all these indices, this is an ascii decimal pair
of “offset length” giving the position of the referenced data within
the pack body file</li>
<li>a list of zero or more reference lists</li>
</ul>
</div></blockquote>
<p>The reference lists let a graph be stored within the index. Each
reference list entry points to another entry in the same index. The
references are represented as a byte offset for the target within the
index file.</p>
<p>When a compression base is given, it indicates that the body of the text
or inventory is a forward delta from the referenced revision. The
compression base list must have length 0 or 1.</p>
<p>Like packs, indexes are written only once and then unmodified. A
GraphIndex builder is a mutable in-memory graph that can be sorted,
cross-referenced and written out when the write group completes.</p>
<p>There can also be index entries with a value of ‘a’ for absent. These
records exist just to be pointed to in a graph. This is used, for
example, to give the revision-parent pointer when the parent revision is
in a previous pack.</p>
<p>The data content for each record is a knit data chunk. The knits are
always unannotated - the annotations must be generated when needed.
(We’d like to cache/memoize the annotations.) The data hunks can be
moved between packs without needing to recompress them.</p>
<p>It is not possible to regenerate an index from the body file, because it
contains information stored in the knit index that’s not in the body.
(In particular, the per-file graph is only stored in the index.)
We would like to change this in a future format.</p>
<p>The lock is a regular LockDir lock. The lock is only held for a much
reduced scope, while updating the pack-names file. The bulk of the
insertion can be done without the repository locked. This is an
implementation detail; the repository user should still call
<tt class="docutils literal"><span class="pre">repository.lock_write</span></tt> at the regular time but be aware this does not
correspond to a physical mutex.</p>
<p>Read locks control caching but do not affect writers.</p>
<p>The newly-added repository write group concept is very important to
KnitPack repositories. When <tt class="docutils literal"><span class="pre">start_write_group</span></tt> is called, a new
temporary pack is created and all modifications to the repository will
go into it until either <tt class="docutils literal"><span class="pre">commit_write_group</span></tt> or <tt class="docutils literal"><span class="pre">abort_write_group</span></tt>
is called, at which time it is either finished and moved into place or
discarded respectively. Write groups cannot be nested, only one can be
underway at a time on a Repository instance and they must occur within a
write lock.</p>
<p>Normally the data for each revision will be entirely within a single
pack but this is not required.</p>
<p>When a pack is finished, it gets a final name based on the md5 of all
the data written into the pack body file.</p>
<p>The <tt class="docutils literal"><span class="pre">pack-names</span></tt> file gives the list of all finished non-obsolete
packs. (This should always be the same as the list of files in the
<tt class="docutils literal"><span class="pre">packs/</span></tt> directory, but the file is needed for read-only HTTP clients
that can’t easily list directories, and it includes other information.)
The constraint on the <tt class="docutils literal"><span class="pre">pack-names</span></tt> list is that every file mentioned
must exist in the <tt class="docutils literal"><span class="pre">packs/</span></tt> directory.</p>
<p>In rare cases, when a writer is interrupted, about-to-be-removed packs
may still be present in the directory but removed from the list.</p>
<p>As well as the list of names, the pack-names file also contains the
size, in bytes, of each of the four indices. This is used to bootstrap
bisection search within the indices.</p>
<p>In normal use, one pack will be created for each commit to a repository.
This would build up to an inefficient number of files over time, so a
<tt class="docutils literal"><span class="pre">repack</span></tt> operation is available to recombine them, by producing larger
files containing data on multiple revisions. This can be done manually
by running <tt class="docutils literal"><span class="pre">bzr</span> <span class="pre">pack</span></tt>, and it also may happen automatically when a
write group is committed.</p>
<p>The repacking strategy used at the moment tries to balance not doing too
much work during commit with not having too many small files left in the
repository. The algorithm is roughly this: the total number of
revisions in the repository is expressed as a decimal number, e.g.
“532”. Then we’ll repack until we have five packs containing a hundred
revisions each, three packs containing ten revisions each, and two packs
with single revisions. This means that each revision will normally
initially be created in a single-revision pack, then moved to a
ten-revision pack, then to a 100-pack, and so on.</p>
<p>As with other repositories, in normal use data is only inserted.
However, in some circumstances we may want to garbage-collect or prune
existing data, or reconcile indexes.</p>
</div>
</div>
</div>
</div>
</div>
<div class="sphinxsidebar">
<div class="sphinxsidebarwrapper">
<h3><a href="index.html">Table Of Contents</a></h3>
<ul>
<li><a class="reference internal" href="#">KnitPack repository format</a><ul>
<li><a class="reference internal" href="#using-knitpack-repositories">Using KnitPack repositories</a><ul>
<li><a class="reference internal" href="#motivation">Motivation</a></li>
<li><a class="reference internal" href="#preparation">Preparation</a></li>
<li><a class="reference internal" href="#creating-a-new-knitpack-branch">Creating a new knitpack branch</a></li>
<li><a class="reference internal" href="#creating-a-new-knitpack-repository">Creating a new knitpack repository</a></li>
<li><a class="reference internal" href="#upgrading-an-existing-branch-or-repository-to-knitpack-format">Upgrading an existing branch or repository to knitpack format</a></li>
<li><a class="reference internal" href="#starting-a-new-knitpack-branch-from-one-in-an-older-format">Starting a new knitpack branch from one in an older format</a></li>
<li><a class="reference internal" href="#testing-packs-for-bzr-svn-users">Testing packs for bzr-svn users</a></li>
<li><a class="reference internal" href="#reporting-problems">Reporting problems</a></li>
</ul>
</li>
<li><a class="reference internal" href="#technical-notes">Technical notes</a></li>
</ul>
</li>
</ul>
<h4>Previous topic</h4>
<p class="topless"><a href="development-repo.html"
title="previous chapter">Development repository formats</a></p>
<h4>Next topic</h4>
<p class="topless"><a href="implementation-notes.html"
title="next chapter">Implementation notes</a></p>
<h3>This Page</h3>
<ul class="this-page-menu">
<li><a href="_sources/packrepo.txt"
rel="nofollow">Show Source</a></li>
</ul>
<div id="searchbox" style="display: none">
<h3>Quick search</h3>
<form class="search" action="search.html" method="get">
<input type="text" name="q" />
<input type="submit" value="Go" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
<p class="searchtip" style="font-size: 90%">
Enter search terms or a module, class or function name.
</p>
</div>
<script type="text/javascript">$('#searchbox').show(0);</script>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="related">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="implementation-notes.html" title="Implementation notes"
>next</a></li>
<li class="right" >
<a href="development-repo.html" title="Development repository formats"
>previous</a> |</li>
<li><a href="http://bazaar.canonical.com/">
<img src="_static/bzr icon 16.png" /> Home</a> | </li>
<a href="http://doc.bazaar.canonical.com/en/">Documentation</a> | </li>
<li><a href="index.html">Developer Document Catalog (2.6.0)</a> »</li>
<li><a href="specifications.html" >Specifications</a> »</li>
</ul>
</div>
<div class="footer">
© Copyright 2009-2011 Canonical Ltd.
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 1.2.3.
</div>
</body>
</html>
