<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Interlocks — Buildbot 0.8.12 documentation</title>
<link rel="stylesheet" href="../_static/agogo.css" type="text/css" />
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT: '../',
VERSION: '0.8.12',
COLLAPSE_INDEX: false,
FILE_SUFFIX: '.html',
HAS_SOURCE: true
};
</script>
<script type="text/javascript" src="../_static/jquery.js"></script>
<script type="text/javascript" src="../_static/underscore.js"></script>
<script type="text/javascript" src="../_static/doctools.js"></script>
<link rel="shortcut icon" href="../_static/buildbot.ico"/>
<link rel="top" title="Buildbot 0.8.12 documentation" href="../index.html" />
<link rel="up" title="Configuration" href="configuration.html" />
<link rel="next" title="Status Targets" href="cfg-statustargets.html" />
<link rel="prev" title="Build Steps" href="cfg-buildsteps.html" />
</head>
<body role="document">
<div class="header-wrapper" role="banner">
<div class="header">
<p class="logo"><a href="../index.html">
<img class="logo" src="../_static/header-text-transparent.png" alt="Logo"/>
</a></p>
<div class="headertitle"><a
href="../index.html">Buildbot 0.8.12 documentation</a></div>
<div class="rel" role="navigation" aria-label="related navigation">
<a href="cfg-buildsteps.html" title="Build Steps"
accesskey="P">previous</a> |
<a href="cfg-statustargets.html" title="Status Targets"
accesskey="N">next</a> |
<a href="../py-modindex.html" title="Python Module Index"
>modules</a> |
<a href="../genindex.html" title="General Index"
accesskey="I">index</a>
</div>
</div>
</div>
<div class="content-wrapper">
<div class="content">
<div class="document">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body" role="main">
<div class="section" id="interlocks">
<span id="id1"></span><h1>Interlocks<a class="headerlink" href="#interlocks" title="Permalink to this headline">¶</a></h1>
<div class="contents local topic" id="contents">
<ul class="simple">
<li><a class="reference internal" href="#access-modes" id="id9">Access Modes</a></li>
<li><a class="reference internal" href="#count" id="id10">Count</a></li>
<li><a class="reference internal" href="#scope" id="id11">Scope</a></li>
<li><a class="reference internal" href="#examples" id="id12">Examples</a></li>
</ul>
</div>
<p>Until now, we assumed that a master can run builds at any slave whenever needed or desired.
Some times, you want to enforce additional constraints on builds.
For reasons like limited network bandwidth, old slave machines, or a self-willed data base server, you may want to limit the number of builds (or build steps) that can access a resource.</p>
<div class="section" id="access-modes">
<span id="id2"></span><h2><a class="toc-backref" href="#id9">Access Modes</a><a class="headerlink" href="#access-modes" title="Permalink to this headline">¶</a></h2>
<p>The mechanism used by Buildbot is known as the read/write lock <a class="footnote-reference" href="#id6" id="id3">[1]</a>.
It allows either many readers or a single writer but not a combination of readers and writers.
The general lock has been modified and extended for use in Buildbot.
Firstly, the general lock allows an infinite number of readers.
In Buildbot, we often want to put an upper limit on the number of readers, for example allowing two out of five possible builds at the same time.
To do this, the lock counts the number of active readers.
Secondly, the terms <em>read mode</em> and <em>write mode</em> are confusing in Buildbot context.
They have been replaced by <em>counting mode</em> (since the lock counts them) and <em>exclusive mode</em>.
As a result of these changes, locks in Buildbot allow a number of builds (up to some fixed number) in counting mode, or they allow one build in exclusive mode.</p>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">Access modes are specified when a lock is used.
That is, it is possible to have a single lock that is used by several slaves in counting mode, and several slaves in exclusive mode.
In fact, this is the strength of the modes: accessing a lock in exclusive mode will prevent all counting-mode accesses.</p>
</div>
</div>
<div class="section" id="count">
<h2><a class="toc-backref" href="#id10">Count</a><a class="headerlink" href="#count" title="Permalink to this headline">¶</a></h2>
<p>Often, not all slaves are equal.
To allow for this situation, Buildbot allows to have a separate upper limit on the count for each slave.
In this way, you can have at most 3 concurrent builds at a fast slave, 2 at a slightly older slave, and 1 at all other slaves.</p>
</div>
<div class="section" id="scope">
<h2><a class="toc-backref" href="#id11">Scope</a><a class="headerlink" href="#scope" title="Permalink to this headline">¶</a></h2>
<p>The final thing you can specify when you introduce a new lock is its scope.
Some constraints are global -- they must be enforced over all slaves.
Other constraints are local to each slave.
A <em>master lock</em> is used for the global constraints.
You can ensure for example that at most one build (of all builds running at all slaves) accesses the data base server.
With a <em>slave lock</em> you can add a limit local to each slave.
With such a lock, you can for example enforce an upper limit to the number of active builds at a slave, like above.</p>
</div>
<div class="section" id="examples">
<h2><a class="toc-backref" href="#id12">Examples</a><a class="headerlink" href="#examples" title="Permalink to this headline">¶</a></h2>
<p>Time for a few examples.
Below a master lock is defined to protect a data base, and a slave lock is created to limit the number of builds at each slave.</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">buildbot.plugins</span> <span class="k">import</span> <span class="n">util</span>
<span class="n">db_lock</span> <span class="o">=</span> <span class="n">util</span><span class="o">.</span><span class="n">MasterLock</span><span class="p">(</span><span class="s2">"database"</span><span class="p">)</span>
<span class="n">build_lock</span> <span class="o">=</span> <span class="n">util</span><span class="o">.</span><span class="n">SlaveLock</span><span class="p">(</span><span class="s2">"slave_builds"</span><span class="p">,</span>
<span class="n">maxCount</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span>
<span class="n">maxCountForSlave</span><span class="o">=</span><span class="p">{</span><span class="s1">'fast'</span><span class="p">:</span> <span class="mi">3</span><span class="p">,</span> <span class="s1">'new'</span><span class="p">:</span> <span class="mi">2</span><span class="p">})</span>
</pre></div>
</div>
<p>After importing locks from buildbot, <code class="xref py py-data docutils literal"><span class="pre">db_lock</span></code> is defined to be a master lock.
The <code class="docutils literal"><span class="pre">database</span></code> string is used for uniquely identifying the lock.
At the next line, a slave lock called <code class="xref py py-data docutils literal"><span class="pre">build_lock</span></code> is created.
It is identified by the <code class="docutils literal"><span class="pre">slave_builds</span></code> string.
Since the requirements of the lock are a bit more complicated, two optional arguments are also specified.
The <code class="docutils literal"><span class="pre">maxCount</span></code> parameter sets the default limit for builds in counting mode to <code class="docutils literal"><span class="pre">1</span></code>.
For the slave called <code class="docutils literal"><span class="pre">'fast'</span></code> however, we want to have at most three builds, and for the slave called <code class="docutils literal"><span class="pre">'new'</span></code> the upper limit is two builds running at the same time.</p>
<p>The next step is accessing the locks in builds.
Buildbot allows a lock to be used during an entire build (from beginning to end), or only during a single build step.
In the latter case, the lock is claimed for use just before the step starts, and released again when the step ends.
To prevent deadlocks, <a class="footnote-reference" href="#id7" id="id4">[2]</a> it is not possible to claim or release locks at other times.</p>
<p>To use locks, you add them with a <code class="docutils literal"><span class="pre">locks</span></code> argument to a build or a step.
Each use of a lock is either in counting mode (that is, possibly shared with other builds) or in exclusive mode, and this is indicated with the syntax <code class="docutils literal"><span class="pre">lock.access(mode)</span></code>, where <code class="xref py py-data docutils literal"><span class="pre">mode</span></code> is one of <code class="docutils literal"><span class="pre">"counting"</span></code> or <code class="docutils literal"><span class="pre">"exclusive"</span></code>.</p>
<p>A build or build step proceeds only when it has acquired all locks.
If a build or step needs a lot of locks, it may be starved <a class="footnote-reference" href="#id8" id="id5">[3]</a> by other builds that need fewer locks.</p>
<p>To illustrate use of locks, a few examples.</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">buildbot.plugins</span> <span class="k">import</span> <span class="n">util</span><span class="p">,</span> <span class="n">steps</span>
<span class="n">db_lock</span> <span class="o">=</span> <span class="n">util</span><span class="o">.</span><span class="n">MasterLock</span><span class="p">(</span><span class="s2">"database"</span><span class="p">)</span>
<span class="n">build_lock</span> <span class="o">=</span> <span class="n">util</span><span class="o">.</span><span class="n">SlaveLock</span><span class="p">(</span><span class="s2">"slave_builds"</span><span class="p">,</span>
<span class="n">maxCount</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span>
<span class="n">maxCountForSlave</span><span class="o">=</span><span class="p">{</span><span class="s1">'fast'</span><span class="p">:</span> <span class="mi">3</span><span class="p">,</span> <span class="s1">'new'</span><span class="p">:</span> <span class="mi">2</span><span class="p">})</span>
<span class="n">f</span> <span class="o">=</span> <span class="n">util</span><span class="o">.</span><span class="n">BuildFactory</span><span class="p">()</span>
<span class="n">f</span><span class="o">.</span><span class="n">addStep</span><span class="p">(</span><span class="n">steps</span><span class="o">.</span><span class="n">SVN</span><span class="p">(</span><span class="n">svnurl</span><span class="o">=</span><span class="s2">"http://example.org/svn/Trunk"</span><span class="p">))</span>
<span class="n">f</span><span class="o">.</span><span class="n">addStep</span><span class="p">(</span><span class="n">steps</span><span class="o">.</span><span class="n">ShellCommand</span><span class="p">(</span><span class="n">command</span><span class="o">=</span><span class="s2">"make all"</span><span class="p">))</span>
<span class="n">f</span><span class="o">.</span><span class="n">addStep</span><span class="p">(</span><span class="n">steps</span><span class="o">.</span><span class="n">ShellCommand</span><span class="p">(</span><span class="n">command</span><span class="o">=</span><span class="s2">"make test"</span><span class="p">,</span>
<span class="n">locks</span><span class="o">=</span><span class="p">[</span><span class="n">db_lock</span><span class="o">.</span><span class="n">access</span><span class="p">(</span><span class="s1">'exclusive'</span><span class="p">)]))</span>
<span class="n">b1</span> <span class="o">=</span> <span class="p">{</span>
<span class="s1">'name'</span><span class="p">:</span> <span class="s1">'full1'</span><span class="p">,</span>
<span class="s1">'slavename'</span><span class="p">:</span> <span class="s1">'fast'</span><span class="p">,</span>
<span class="s1">'builddir'</span><span class="p">:</span> <span class="s1">'f1'</span><span class="p">,</span>
<span class="s1">'factory'</span><span class="p">:</span> <span class="n">f</span><span class="p">,</span>
<span class="s1">'locks'</span><span class="p">:</span> <span class="p">[</span><span class="n">build_lock</span><span class="o">.</span><span class="n">access</span><span class="p">(</span><span class="s1">'counting'</span><span class="p">)]</span>
<span class="p">}</span>
<span class="n">b2</span> <span class="o">=</span> <span class="p">{</span>
<span class="s1">'name'</span><span class="p">:</span> <span class="s1">'full2'</span><span class="p">,</span>
<span class="s1">'slavename'</span><span class="p">:</span> <span class="s1">'new'</span><span class="p">,</span>
<span class="s1">'builddir'</span><span class="p">:</span> <span class="s1">'f2'</span><span class="p">,</span>
<span class="s1">'factory'</span><span class="p">:</span> <span class="n">f</span><span class="p">,</span>
<span class="s1">'locks'</span><span class="p">:</span> <span class="p">[</span><span class="n">build_lock</span><span class="o">.</span><span class="n">access</span><span class="p">(</span><span class="s1">'counting'</span><span class="p">)]</span>
<span class="p">}</span>
<span class="n">b3</span> <span class="o">=</span> <span class="p">{</span>
<span class="s1">'name'</span><span class="p">:</span> <span class="s1">'full3'</span><span class="p">,</span>
<span class="s1">'slavename'</span><span class="p">:</span> <span class="s1">'old'</span><span class="p">,</span>
<span class="s1">'builddir'</span><span class="p">:</span> <span class="s1">'f3'</span><span class="p">,</span>
<span class="s1">'factory'</span><span class="p">:</span> <span class="n">f</span><span class="p">,</span>
<span class="s1">'locks'</span><span class="p">:</span> <span class="p">[</span><span class="n">build_lock</span><span class="o">.</span><span class="n">access</span><span class="p">(</span><span class="s1">'counting'</span><span class="p">)]</span>
<span class="p">}</span>
<span class="n">b4</span> <span class="o">=</span> <span class="p">{</span>
<span class="s1">'name'</span><span class="p">:</span> <span class="s1">'full4'</span><span class="p">,</span>
<span class="s1">'slavename'</span><span class="p">:</span> <span class="s1">'other'</span><span class="p">,</span>
<span class="s1">'builddir'</span><span class="p">:</span> <span class="s1">'f4'</span><span class="p">,</span>
<span class="s1">'factory'</span><span class="p">:</span> <span class="n">f</span><span class="p">,</span>
<span class="s1">'locks'</span><span class="p">:</span> <span class="p">[</span><span class="n">build_lock</span><span class="o">.</span><span class="n">access</span><span class="p">(</span><span class="s1">'counting'</span><span class="p">)]</span>
<span class="p">}</span>
<span class="n">c</span><span class="p">[</span><span class="s1">'builders'</span><span class="p">]</span> <span class="o">=</span> <span class="p">[</span><span class="n">b1</span><span class="p">,</span> <span class="n">b2</span><span class="p">,</span> <span class="n">b3</span><span class="p">,</span> <span class="n">b4</span><span class="p">]</span>
</pre></div>
</div>
<p>Here we have four slaves <code class="xref py py-data docutils literal"><span class="pre">b1</span></code>, <code class="xref py py-data docutils literal"><span class="pre">b2</span></code>, <code class="xref py py-data docutils literal"><span class="pre">b3</span></code>, and <code class="xref py py-data docutils literal"><span class="pre">b4</span></code>.
Each slave performs the same checkout, make, and test build step sequence.
We want to enforce that at most one test step is executed between all slaves due to restrictions with the data base server.
This is done by adding the <code class="docutils literal"><span class="pre">locks=</span></code> parameter with the third step.
It takes a list of locks with their access mode.
In this case only the <code class="xref py py-data docutils literal"><span class="pre">db_lock</span></code> is needed.
The exclusive access mode is used to ensure there is at most one slave that executes the test step.</p>
<p>In addition to exclusive accessing the data base, we also want slaves to stay responsive even under the load of a large number of builds being triggered.
For this purpose, the slave lock called <code class="xref py py-data docutils literal"><span class="pre">build_lock</span></code> is defined.
Since the restraint holds for entire builds, the lock is specified in the builder with <code class="docutils literal"><span class="pre">'locks':</span> <span class="pre">[build_lock.access('counting')]</span></code>.</p>
<p>Note that you will occasionally see <code class="docutils literal"><span class="pre">lock.access(mode)</span></code> written as <code class="docutils literal"><span class="pre">LockAccess(lock,</span> <span class="pre">mode)</span></code>.
The two are equivalent, but the former is preferred.</p>
<table class="docutils footnote" frame="void" id="id6" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id3">[1]</a></td><td>See <a class="reference external" href="http://en.wikipedia.org/wiki/Read/write_lock_pattern">http://en.wikipedia.org/wiki/Read/write_lock_pattern</a> for more information.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="id7" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id4">[2]</a></td><td>Deadlock is the situation where two or more slaves each hold a lock in exclusive mode, and in addition want to claim the lock held by the other slave exclusively as well.
Since locks allow at most one exclusive user, both slaves will wait forever.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="id8" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id5">[3]</a></td><td>Starving is the situation that only a few locks are available, and they are immediately grabbed by another build.
As a result, it may take a long time before all locks needed by the starved build are free at the same time.</td></tr>
</tbody>
</table>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="sidebar">
<h3>Table Of Contents</h3>
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../tutorial/index.html">Buildbot Tutorial</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="index.html">Buildbot Manual</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="introduction.html">Introduction</a></li>
<li class="toctree-l2"><a class="reference internal" href="installation.html">Installation</a></li>
<li class="toctree-l2"><a class="reference internal" href="concepts.html">Concepts</a></li>
<li class="toctree-l2 current"><a class="reference internal" href="configuration.html">Configuration</a><ul class="current">
<li class="toctree-l3"><a class="reference internal" href="cfg-intro.html">Configuring Buildbot</a></li>
<li class="toctree-l3"><a class="reference internal" href="cfg-global.html">Global Configuration</a></li>
<li class="toctree-l3"><a class="reference internal" href="cfg-changesources.html">Change Sources</a></li>
<li class="toctree-l3"><a class="reference internal" href="cfg-schedulers.html">Schedulers</a></li>
<li class="toctree-l3"><a class="reference internal" href="cfg-buildslaves.html">Buildslaves</a></li>
<li class="toctree-l3"><a class="reference internal" href="cfg-builders.html">Builder Configuration</a></li>
<li class="toctree-l3"><a class="reference internal" href="cfg-buildfactories.html">Build Factories</a></li>
<li class="toctree-l3"><a class="reference internal" href="cfg-properties.html">Properties</a></li>
<li class="toctree-l3"><a class="reference internal" href="cfg-buildsteps.html">Build Steps</a></li>
<li class="toctree-l3 current"><a class="current reference internal" href="#">Interlocks</a><ul class="simple">
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="cfg-statustargets.html">Status Targets</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="customization.html">Customization</a></li>
<li class="toctree-l2"><a class="reference internal" href="new-style-steps.html">New-Style Build Steps</a></li>
<li class="toctree-l2"><a class="reference internal" href="cmdline.html">Command-line Tool</a></li>
<li class="toctree-l2"><a class="reference internal" href="resources.html">Resources</a></li>
<li class="toctree-l2"><a class="reference internal" href="optimization.html">Optimization</a></li>
<li class="toctree-l2"><a class="reference internal" href="plugins.html">Plugin Infrastructure in Buildbot</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../developer/index.html">Buildbot Development</a></li>
<li class="toctree-l1"><a class="reference internal" href="../relnotes/index.html">Release Notes for Buildbot 0.8.12</a></li>
</ul>
<div role="search">
<h3 style="margin-top: 1.5em;">Search</h3>
<form class="search" action="../search.html" method="get">
<input type="text" name="q" />
<input type="submit" value="Go" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div>
<div class="clearer"></div>
</div>
</div>
<div class="footer-wrapper">
<div class="footer">
<div class="left">
<div role="navigation" aria-label="related navigaton">
<a href="cfg-buildsteps.html" title="Build Steps"
>previous</a> |
<a href="cfg-statustargets.html" title="Status Targets"
>next</a> |
<a href="../py-modindex.html" title="Python Module Index"
>modules</a> |
<a href="../genindex.html" title="General Index"
>index</a>
</div>
<div role="note" aria-label="source link">
<br/>
<a href="../_sources/manual/cfg-interlocks.txt"
rel="nofollow">Show Source</a>
</div>
</div>
<div class="right">
<div class="footer" role="contentinfo">
© Copyright Buildbot Team Members.
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 1.4.1.
</div>
</div>
<div class="clearer"></div>
</div>
</div>
</body>
</html>
