<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Customization — Buildbot 0.8.8 documentation</title>
<link rel="stylesheet" href="../_static/agogo.css" type="text/css" />
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT: '../',
VERSION: '0.8.8',
COLLAPSE_INDEX: false,
FILE_SUFFIX: '.html',
HAS_SOURCE: true
};
</script>
<script type="text/javascript" src="../_static/jquery.js"></script>
<script type="text/javascript" src="../_static/underscore.js"></script>
<script type="text/javascript" src="../_static/doctools.js"></script>
<link rel="shortcut icon" href="../_static/buildbot.ico"/>
<link rel="top" title="Buildbot 0.8.8 documentation" href="../index.html" />
<link rel="up" title="Buildbot Manual" href="index.html" />
<link rel="next" title="Command-line Tool" href="cmdline.html" />
<link rel="prev" title="Status Targets" href="cfg-statustargets.html" />
</head>
<body>
<div class="header-wrapper">
<div class="header">
<p class="logo"><a href="../index.html">
<img class="logo" src="../_static/header-text-transparent.png" alt="Logo"/>
</a></p>
<div class="headertitle"><a
href="../index.html">Buildbot 0.8.8 documentation</a></div>
<div class="rel">
<a href="cfg-statustargets.html" title="Status Targets"
accesskey="P">previous</a> |
<a href="cmdline.html" title="Command-line Tool"
accesskey="N">next</a> |
<a href="../py-modindex.html" title="Python Module Index"
>modules</a> |
<a href="../genindex.html" title="General Index"
accesskey="I">index</a>
</div>
</div>
</div>
<div class="content-wrapper">
<div class="content">
<div class="document">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body">
<div class="section" id="customization">
<span id="id1"></span><h1>Customization<a class="headerlink" href="#customization" title="Permalink to this headline">¶</a></h1>
<p>For advanced users, Buildbot acts as a framework supporting a customized build
application. For the most part, such configurations consist of subclasses set
up for use in a regular Buildbot configuration file.</p>
<p>This chapter describes some of the more common idioms in advanced Buildbot
configurations.</p>
<p>At the moment, this chapter is an unordered set of suggestions; if you'd like
to clean it up, fork the project on GitHub and get started!</p>
<div class="section" id="programmatic-configuration-generation">
<h2>Programmatic Configuration Generation<a class="headerlink" href="#programmatic-configuration-generation" title="Permalink to this headline">¶</a></h2>
<p>Bearing in mind that <tt class="docutils literal"><span class="pre">master.cfg</span></tt> is a Python file, large configurations can
be shortened considerably by judicious use of Python loops. For example, the
following will generate a builder for each of a range of supported versions of
Python:</p>
<div class="highlight-python"><pre>pythons = [ 'python2.4', 'python2.5', 'python2.6', 'python2.7',
'python3.2', python3.3' ]
pytest_slaves = [ "slave%s" % n for n in range(10) ]
for python in pythons:
f = BuildFactory()
f.addStep(SVN(..))
f.addStep(ShellCommand(command=[ python, 'test.py' ]))
c['builders'].append(BuilderConfig(
name="test-%s" % python,
factory=f,
slavenames=pytest_slaves))</pre>
</div>
</div>
<div class="section" id="merge-request-functions">
<span id="id2"></span><h2>Merge Request Functions<a class="headerlink" href="#merge-request-functions" title="Permalink to this headline">¶</a></h2>
<p id="index-0">The logic Buildbot uses to decide which build request can be merged can be
customized by providing a Python function (a callable) instead of <tt class="docutils literal"><span class="pre">True</span></tt> or
<tt class="docutils literal"><span class="pre">False</span></tt> described in <a class="reference internal" href="cfg-builders.html#merging-build-requests"><em>Merging Build Requests</em></a>.</p>
<p>The callable will be invoked with three positional arguments: a
<tt class="xref py py-class docutils literal"><span class="pre">Builder</span></tt> object and two <tt class="xref py py-class docutils literal"><span class="pre">BuildRequest</span></tt> objects. It should return
true if the requests can be merged, and False otherwise. For example:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="k">def</span> <span class="nf">mergeRequests</span><span class="p">(</span><span class="n">builder</span><span class="p">,</span> <span class="n">req1</span><span class="p">,</span> <span class="n">req2</span><span class="p">):</span>
<span class="s">"any requests with the same branch can be merged"</span>
<span class="k">return</span> <span class="n">req1</span><span class="o">.</span><span class="n">source</span><span class="o">.</span><span class="n">branch</span> <span class="o">==</span> <span class="n">req2</span><span class="o">.</span><span class="n">source</span><span class="o">.</span><span class="n">branch</span>
<span class="n">c</span><span class="p">[</span><span class="s">'mergeRequests'</span><span class="p">]</span> <span class="o">=</span> <span class="n">mergeRequests</span>
</pre></div>
</div>
<p>In many cases, the details of the <tt class="xref py py-class docutils literal"><span class="pre">SourceStamp</span></tt>s and <tt class="xref py py-class docutils literal"><span class="pre">BuildRequest</span></tt>s are important.
In this example, only <tt class="xref py py-class docutils literal"><span class="pre">BuildRequest</span></tt>s with the same "reason" are merged; thus
developers forcing builds for different reasons will see distinct builds. Note
the use of the <tt class="xref py py-func docutils literal"><span class="pre">canBeMergedWith</span></tt> method to access the source stamp
compatibility algorithm.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="k">def</span> <span class="nf">mergeRequests</span><span class="p">(</span><span class="n">builder</span><span class="p">,</span> <span class="n">req1</span><span class="p">,</span> <span class="n">req2</span><span class="p">):</span>
<span class="k">if</span> <span class="n">req1</span><span class="o">.</span><span class="n">source</span><span class="o">.</span><span class="n">canBeMergedWith</span><span class="p">(</span><span class="n">req2</span><span class="o">.</span><span class="n">source</span><span class="p">)</span> <span class="ow">and</span> <span class="n">req1</span><span class="o">.</span><span class="n">reason</span> <span class="o">==</span> <span class="n">req2</span><span class="o">.</span><span class="n">reason</span><span class="p">:</span>
<span class="k">return</span> <span class="bp">True</span>
<span class="k">return</span> <span class="bp">False</span>
<span class="n">c</span><span class="p">[</span><span class="s">'mergeRequests'</span><span class="p">]</span> <span class="o">=</span> <span class="n">mergeRequests</span>
</pre></div>
</div>
<p>If it's necessary to perform some extended operation to determine whether two
requests can be merged, then the <tt class="docutils literal"><span class="pre">mergeRequests</span></tt> callable may return its
result via Deferred. Note, however, that the number of invocations of the
callable is proportional to the square of the request queue length, so a
long-running callable may cause undesirable delays when the queue length
grows. For example:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="k">def</span> <span class="nf">mergeRequests</span><span class="p">(</span><span class="n">builder</span><span class="p">,</span> <span class="n">req1</span><span class="p">,</span> <span class="n">req2</span><span class="p">):</span>
<span class="n">d</span> <span class="o">=</span> <span class="n">defer</span><span class="o">.</span><span class="n">gatherResults</span><span class="p">([</span>
<span class="n">getMergeInfo</span><span class="p">(</span><span class="n">req1</span><span class="o">.</span><span class="n">source</span><span class="o">.</span><span class="n">revision</span><span class="p">),</span>
<span class="n">getMergeInfo</span><span class="p">(</span><span class="n">req2</span><span class="o">.</span><span class="n">source</span><span class="o">.</span><span class="n">revision</span><span class="p">),</span>
<span class="p">])</span>
<span class="k">def</span> <span class="nf">process</span><span class="p">(</span><span class="n">info1</span><span class="p">,</span> <span class="n">info2</span><span class="p">):</span>
<span class="k">return</span> <span class="n">info1</span> <span class="o">==</span> <span class="n">info2</span>
<span class="n">d</span><span class="o">.</span><span class="n">addCallback</span><span class="p">(</span><span class="n">process</span><span class="p">)</span>
<span class="k">return</span> <span class="n">d</span>
<span class="n">c</span><span class="p">[</span><span class="s">'mergeRequests'</span><span class="p">]</span> <span class="o">=</span> <span class="n">mergeRequests</span>
</pre></div>
</div>
</div>
<div class="section" id="builder-priority-functions">
<span id="id3"></span><h2>Builder Priority Functions<a class="headerlink" href="#builder-priority-functions" title="Permalink to this headline">¶</a></h2>
<p id="index-1">The <a class="reference internal" href="cfg-global.html#cfg-prioritizeBuilders" title="prioritizeBuilders"><tt class="xref bb bb-cfg docutils literal"><span class="pre">prioritizeBuilders</span></tt></a> configuration key specifies a function which
is called with two arguments: a <tt class="xref py py-class docutils literal"><span class="pre">BuildMaster</span></tt> and a list of
<tt class="xref py py-class docutils literal"><span class="pre">Builder</span></tt> objects. It should return a list of the same <tt class="xref py py-class docutils literal"><span class="pre">Builder</span></tt>
objects, in the desired order. It may also remove items from the list if
builds should not be started on those builders. If necessary, this function can
return its results via a Deferred (it is called with <tt class="docutils literal"><span class="pre">maybeDeferred</span></tt>).</p>
<p>A simple <tt class="docutils literal"><span class="pre">prioritizeBuilders</span></tt> implementation might look like this:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="k">def</span> <span class="nf">prioritizeBuilders</span><span class="p">(</span><span class="n">buildmaster</span><span class="p">,</span> <span class="n">builders</span><span class="p">):</span>
<span class="sd">"""Prioritize builders. 'finalRelease' builds have the highest</span>
<span class="sd"> priority, so they should be built before running tests, or</span>
<span class="sd"> creating builds."""</span>
<span class="n">builderPriorities</span> <span class="o">=</span> <span class="p">{</span>
<span class="s">"finalRelease"</span><span class="p">:</span> <span class="mi">0</span><span class="p">,</span>
<span class="s">"test"</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="s">"build"</span><span class="p">:</span> <span class="mi">2</span><span class="p">,</span>
<span class="p">}</span>
<span class="n">builders</span><span class="o">.</span><span class="n">sort</span><span class="p">(</span><span class="n">key</span><span class="o">=</span><span class="k">lambda</span> <span class="n">b</span><span class="p">:</span> <span class="n">builderPriorities</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">b</span><span class="o">.</span><span class="n">name</span><span class="p">,</span> <span class="mi">0</span><span class="p">))</span>
<span class="k">return</span> <span class="n">builders</span>
<span class="n">c</span><span class="p">[</span><span class="s">'prioritizeBuilders'</span><span class="p">]</span> <span class="o">=</span> <span class="n">prioritizeBuilders</span>
</pre></div>
</div>
</div>
<div class="section" id="build-priority-functions">
<span id="index-2"></span><span id="id4"></span><h2>Build Priority Functions<a class="headerlink" href="#build-priority-functions" title="Permalink to this headline">¶</a></h2>
<p>When a builder has multiple pending build requests, it uses a <tt class="docutils literal"><span class="pre">nextBuild</span></tt>
function to decide which build it should start first. This function is given
two parameters: the <tt class="xref py py-class docutils literal"><span class="pre">Builder</span></tt>, and a list of <tt class="xref py py-class docutils literal"><span class="pre">BuildRequest</span></tt>
objects representing pending build requests.</p>
<p>A simple function to prioritize release builds over other builds might look
like this:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="k">def</span> <span class="nf">nextBuild</span><span class="p">(</span><span class="n">bldr</span><span class="p">,</span> <span class="n">requests</span><span class="p">):</span>
<span class="k">for</span> <span class="n">r</span> <span class="ow">in</span> <span class="n">requests</span><span class="p">:</span>
<span class="k">if</span> <span class="n">r</span><span class="o">.</span><span class="n">source</span><span class="o">.</span><span class="n">branch</span> <span class="o">==</span> <span class="s">'release'</span><span class="p">:</span>
<span class="k">return</span> <span class="n">r</span>
<span class="k">return</span> <span class="n">requests</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
</pre></div>
</div>
<p>If some non-immediate result must be calculated, the <tt class="docutils literal"><span class="pre">nextBuild</span></tt> function can
also return a Deferred:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="k">def</span> <span class="nf">nextBuild</span><span class="p">(</span><span class="n">bldr</span><span class="p">,</span> <span class="n">requests</span><span class="p">):</span>
<span class="n">d</span> <span class="o">=</span> <span class="n">get_request_priorities</span><span class="p">(</span><span class="n">requests</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">pick</span><span class="p">(</span><span class="n">priorities</span><span class="p">):</span>
<span class="k">if</span> <span class="n">requests</span><span class="p">:</span>
<span class="k">return</span> <span class="nb">sorted</span><span class="p">(</span><span class="nb">zip</span><span class="p">(</span><span class="n">priorities</span><span class="p">,</span> <span class="n">requests</span><span class="p">))[</span><span class="mi">0</span><span class="p">][</span><span class="mi">1</span><span class="p">]</span>
<span class="n">d</span><span class="o">.</span><span class="n">addCallback</span><span class="p">(</span><span class="n">pick</span><span class="p">)</span>
<span class="k">return</span> <span class="n">d</span>
</pre></div>
</div>
</div>
<div class="section" id="customizing-svnpoller">
<span id="id5"></span><h2>Customizing SVNPoller<a class="headerlink" href="#customizing-svnpoller" title="Permalink to this headline">¶</a></h2>
<p>Each source file that is tracked by a Subversion repository has a
fully-qualified SVN URL in the following form:
<tt class="docutils literal"><span class="pre">({REPOURL})({PROJECT-plus-BRANCH})({FILEPATH})</span></tt>. When you create the
<a class="reference internal" href="cfg-changesources.html#chsrc-SVNPoller" title="SVNPoller"><tt class="xref bb bb-chsrc docutils literal"><span class="pre">SVNPoller</span></tt></a>, you give it a <tt class="docutils literal"><span class="pre">svnurl</span></tt> value that includes all of the
<tt class="docutils literal"><span class="pre">{REPOURL}</span></tt> and possibly some portion of the
<tt class="docutils literal"><span class="pre">{PROJECT-plus-BRANCH}</span></tt> string. The <a class="reference internal" href="cfg-changesources.html#chsrc-SVNPoller" title="SVNPoller"><tt class="xref bb bb-chsrc docutils literal"><span class="pre">SVNPoller</span></tt></a> is responsible
for producing Changes that contain a branch name and a <tt class="docutils literal"><span class="pre">{FILEPATH}</span></tt>
(which is relative to the top of a checked-out tree). The details of how these
strings are split up depend upon how your repository names its branches.</p>
<div class="section" id="project-branchname-filepath-repositories">
<h3>PROJECT/BRANCHNAME/FILEPATH repositories<a class="headerlink" href="#project-branchname-filepath-repositories" title="Permalink to this headline">¶</a></h3>
<p>One common layout is to have all the various projects that share a repository
get a single top-level directory each, with <tt class="docutils literal"><span class="pre">branches</span></tt>, <tt class="docutils literal"><span class="pre">tags</span></tt>, and
<tt class="docutils literal"><span class="pre">trunk</span></tt> subdirectories:</p>
<div class="highlight-none"><div class="highlight"><pre>amanda/trunk
/branches/3_2
/3_3
/tags/3_2_1
/3_2_2
/3_3_0
</pre></div>
</div>
<p>To set up a <a class="reference internal" href="cfg-changesources.html#chsrc-SVNPoller" title="SVNPoller"><tt class="xref bb bb-chsrc docutils literal"><span class="pre">SVNPoller</span></tt></a> that watches the Amanda trunk (and nothing
else), we would use the following, using the default <tt class="docutils literal"><span class="pre">split_file</span></tt>:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="kn">from</span> <span class="nn">buildbot.changes.svnpoller</span> <span class="kn">import</span> <span class="n">SVNPoller</span>
<span class="n">c</span><span class="p">[</span><span class="s">'change_source'</span><span class="p">]</span> <span class="o">=</span> <span class="n">SVNPoller</span><span class="p">(</span>
<span class="n">svnurl</span><span class="o">=</span><span class="s">"https://svn.amanda.sourceforge.net/svnroot/amanda/amanda/trunk"</span><span class="p">)</span>
</pre></div>
</div>
<p>In this case, every Change that our <a class="reference internal" href="cfg-changesources.html#chsrc-SVNPoller" title="SVNPoller"><tt class="xref bb bb-chsrc docutils literal"><span class="pre">SVNPoller</span></tt></a> produces will have
its branch attribute set to <tt class="docutils literal"><span class="pre">None</span></tt>, to indicate that the Change is on the
trunk. No other sub-projects or branches will be tracked.</p>
<p>If we want our ChangeSource to follow multiple branches, we have to do
two things. First we have to change our <tt class="docutils literal"><span class="pre">svnurl=</span></tt> argument to
watch more than just <tt class="docutils literal"><span class="pre">amanda/trunk</span></tt>. We will set it to
<tt class="docutils literal"><span class="pre">amanda</span></tt> so that we'll see both the trunk and all the branches.
Second, we have to tell <a class="reference internal" href="cfg-changesources.html#chsrc-SVNPoller" title="SVNPoller"><tt class="xref bb bb-chsrc docutils literal"><span class="pre">SVNPoller</span></tt></a> how to split the
<tt class="docutils literal"><span class="pre">({PROJECT-plus-BRANCH})({FILEPATH})</span></tt> strings it gets from the repository
out into <tt class="docutils literal"><span class="pre">({BRANCH})</span></tt> and <tt class="docutils literal"><span class="pre">({FILEPATH})`</span></tt>.</p>
<p>We do the latter by providing a <tt class="docutils literal"><span class="pre">split_file</span></tt> function. This function is
responsible for splitting something like <tt class="docutils literal"><span class="pre">branches/3_3/common-src/amanda.h</span></tt>
into <tt class="docutils literal"><span class="pre">branch='branches/3_3'</span></tt> and <tt class="docutils literal"><span class="pre">filepath='common-src/amanda.h'</span></tt>. The
function is always given a string that names a file relative to the
subdirectory pointed to by the <a class="reference internal" href="cfg-changesources.html#chsrc-SVNPoller" title="SVNPoller"><tt class="xref bb bb-chsrc docutils literal"><span class="pre">SVNPoller</span></tt></a>'s <tt class="docutils literal"><span class="pre">svnurl=</span></tt> argument.
It is expected to return a dictionary with at least the <tt class="docutils literal"><span class="pre">path</span></tt> key. The
splitter may optionally set <tt class="docutils literal"><span class="pre">branch</span></tt>, <tt class="docutils literal"><span class="pre">project</span></tt> and <tt class="docutils literal"><span class="pre">repository</span></tt>.
For backwards compatibility it may return a tuple of <tt class="docutils literal"><span class="pre">(branchname,</span> <span class="pre">path)</span></tt>.
It may also return <tt class="docutils literal"><span class="pre">None</span></tt> to indicate that the file is of no interest.</p>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">the function should return <tt class="docutils literal"><span class="pre">branches/3_3</span></tt> rather than just <tt class="docutils literal"><span class="pre">3_3</span></tt>
because the SVN checkout step, will append the branch name to the
<tt class="docutils literal"><span class="pre">baseURL</span></tt>, which requires that we keep the <tt class="docutils literal"><span class="pre">branches</span></tt> component in
there. Other VC schemes use a different approach towards branches and may
not require this artifact.</p>
</div>
<p>If your repository uses this same <tt class="docutils literal"><span class="pre">{PROJECT}/{BRANCH}/{FILEPATH}</span></tt> naming
scheme, the following function will work:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="k">def</span> <span class="nf">split_file_branches</span><span class="p">(</span><span class="n">path</span><span class="p">):</span>
<span class="n">pieces</span> <span class="o">=</span> <span class="n">path</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s">'/'</span><span class="p">)</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">pieces</span><span class="p">)</span> <span class="o">></span> <span class="mi">1</span> <span class="ow">and</span> <span class="n">pieces</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">==</span> <span class="s">'trunk'</span><span class="p">:</span>
<span class="k">return</span> <span class="p">(</span><span class="bp">None</span><span class="p">,</span> <span class="s">'/'</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">pieces</span><span class="p">[</span><span class="mi">1</span><span class="p">:]))</span>
<span class="k">elif</span> <span class="nb">len</span><span class="p">(</span><span class="n">pieces</span><span class="p">)</span> <span class="o">></span> <span class="mi">2</span> <span class="ow">and</span> <span class="n">pieces</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">==</span> <span class="s">'branches'</span><span class="p">:</span>
<span class="k">return</span> <span class="p">(</span><span class="s">'/'</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">pieces</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="mi">2</span><span class="p">]),</span>
<span class="s">'/'</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">pieces</span><span class="p">[</span><span class="mi">2</span><span class="p">:]))</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">return</span> <span class="bp">None</span>
</pre></div>
</div>
<p>In fact, this is the definition of the provided <tt class="docutils literal"><span class="pre">split_file_branches</span></tt>
function. So to have our Twisted-watching <a class="reference internal" href="cfg-changesources.html#chsrc-SVNPoller" title="SVNPoller"><tt class="xref bb bb-chsrc docutils literal"><span class="pre">SVNPoller</span></tt></a> follow
multiple branches, we would use this:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="kn">from</span> <span class="nn">buildbot.changes.svnpoller</span> <span class="kn">import</span> <span class="n">SVNPoller</span><span class="p">,</span> <span class="n">split_file_branches</span>
<span class="n">c</span><span class="p">[</span><span class="s">'change_source'</span><span class="p">]</span> <span class="o">=</span> <span class="n">SVNPoller</span><span class="p">(</span><span class="s">"svn://svn.twistedmatrix.com/svn/Twisted"</span><span class="p">,</span>
<span class="n">split_file</span><span class="o">=</span><span class="n">split_file_branches</span><span class="p">)</span>
</pre></div>
</div>
<p>Changes for all sorts of branches (with names like <tt class="docutils literal"><span class="pre">"branches/1.5.x"</span></tt>, and
<tt class="docutils literal"><span class="pre">None</span></tt> to indicate the trunk) will be delivered to the Schedulers. Each
Scheduler is then free to use or ignore each branch as it sees fit.</p>
<p>If you have multiple projects in the same repository your split function can
attach a project name to the Change to help the Scheduler filter out unwanted
changes:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="kn">from</span> <span class="nn">buildbot.changes.svnpoller</span> <span class="kn">import</span> <span class="n">split_file_branches</span>
<span class="k">def</span> <span class="nf">split_file_projects_branches</span><span class="p">(</span><span class="n">path</span><span class="p">):</span>
<span class="k">if</span> <span class="ow">not</span> <span class="s">"/"</span> <span class="ow">in</span> <span class="n">path</span><span class="p">:</span>
<span class="k">return</span> <span class="bp">None</span>
<span class="n">project</span><span class="p">,</span> <span class="n">path</span> <span class="o">=</span> <span class="n">path</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s">"/"</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
<span class="n">f</span> <span class="o">=</span> <span class="n">split_file_branches</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
<span class="k">if</span> <span class="n">f</span><span class="p">:</span>
<span class="n">info</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">(</span><span class="n">project</span><span class="o">=</span><span class="n">project</span><span class="p">,</span> <span class="n">path</span><span class="o">=</span><span class="n">f</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span>
<span class="k">if</span> <span class="n">f</span><span class="p">[</span><span class="mi">0</span><span class="p">]:</span>
<span class="n">info</span><span class="p">[</span><span class="s">'branch'</span><span class="p">]</span> <span class="o">=</span> <span class="n">f</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="k">return</span> <span class="n">info</span>
<span class="k">return</span> <span class="n">f</span>
</pre></div>
</div>
<p>Again, this is provided by default. To use it you would do this:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="kn">from</span> <span class="nn">buildbot.changes.svnpoller</span> <span class="kn">import</span> <span class="n">SVNPoller</span><span class="p">,</span> <span class="n">split_file_projects_branches</span>
<span class="n">c</span><span class="p">[</span><span class="s">'change_source'</span><span class="p">]</span> <span class="o">=</span> <span class="n">SVNPoller</span><span class="p">(</span>
<span class="n">svnurl</span><span class="o">=</span><span class="s">"https://svn.amanda.sourceforge.net/svnroot/amanda/"</span><span class="p">,</span>
<span class="n">split_file</span><span class="o">=</span><span class="n">split_file_projects_branches</span><span class="p">)</span>
</pre></div>
</div>
<p>Note here that we are monitoring at the root of the repository, and that within
that repository is a <tt class="docutils literal"><span class="pre">amanda</span></tt> subdirectory which in turn has <tt class="docutils literal"><span class="pre">trunk</span></tt> and
<tt class="docutils literal"><span class="pre">branches</span></tt>. It is that <tt class="docutils literal"><span class="pre">amanda</span></tt> subdirectory whose name becomes the
<tt class="docutils literal"><span class="pre">project</span></tt> field of the Change.</p>
</div>
<div class="section" id="branchname-project-filepath-repositories">
<h3>BRANCHNAME/PROJECT/FILEPATH repositories<a class="headerlink" href="#branchname-project-filepath-repositories" title="Permalink to this headline">¶</a></h3>
<p>Another common way to organize a Subversion repository is to put the branch
name at the top, and the projects underneath. This is especially frequent when
there are a number of related sub-projects that all get released in a group.</p>
<p>For example, <a class="reference external" href="http://Divmod.org">Divmod.org</a> hosts a project named <cite>Nevow</cite> as
well as one named <cite>Quotient</cite>. In a checked-out Nevow tree there is a directory
named <cite>formless</cite> that contains a Python source file named <tt class="file docutils literal"><span class="pre">webform.py</span></tt>.
This repository is accessible via webdav (and thus uses an <cite>http:</cite> scheme)
through the divmod.org hostname. There are many branches in this repository,
and they use a <tt class="docutils literal"><span class="pre">({BRANCHNAME})/({PROJECT})</span></tt> naming policy.</p>
<p>The fully-qualified SVN URL for the trunk version of <tt class="file docutils literal"><span class="pre">webform.py</span></tt> is
<tt class="docutils literal"><span class="pre">http://divmod.org/svn/Divmod/trunk/Nevow/formless/webform.py</span></tt>.
The 1.5.x branch version of this file would have a URL of
<tt class="docutils literal"><span class="pre">http://divmod.org/svn/Divmod/branches/1.5.x/Nevow/formless/webform.py</span></tt>.
The whole Nevow trunk would be checked out with
<tt class="docutils literal"><span class="pre">http://divmod.org/svn/Divmod/trunk/Nevow</span></tt>, while the Quotient
trunk would be checked out using
<tt class="docutils literal"><span class="pre">http://divmod.org/svn/Divmod/trunk/Quotient</span></tt>.</p>
<p>Now suppose we want to have an <a class="reference internal" href="cfg-changesources.html#chsrc-SVNPoller" title="SVNPoller"><tt class="xref bb bb-chsrc docutils literal"><span class="pre">SVNPoller</span></tt></a> that only cares about the
Nevow trunk. This case looks just like the <tt class="docutils literal"><span class="pre">{PROJECT}/{BRANCH}</span></tt> layout
described earlier:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="kn">from</span> <span class="nn">buildbot.changes.svnpoller</span> <span class="kn">import</span> <span class="n">SVNPoller</span>
<span class="n">c</span><span class="p">[</span><span class="s">'change_source'</span><span class="p">]</span> <span class="o">=</span> <span class="n">SVNPoller</span><span class="p">(</span><span class="s">"http://divmod.org/svn/Divmod/trunk/Nevow"</span><span class="p">)</span>
</pre></div>
</div>
<p>But what happens when we want to track multiple Nevow branches? We
have to point our <tt class="docutils literal"><span class="pre">svnurl=</span></tt> high enough to see all those
branches, but we also don't want to include Quotient changes (since
we're only building Nevow). To accomplish this, we must rely upon the
<tt class="docutils literal"><span class="pre">split_file</span></tt> function to help us tell the difference between
files that belong to Nevow and those that belong to Quotient, as well
as figuring out which branch each one is on.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="kn">from</span> <span class="nn">buildbot.changes.svnpoller</span> <span class="kn">import</span> <span class="n">SVNPoller</span>
<span class="n">c</span><span class="p">[</span><span class="s">'change_source'</span><span class="p">]</span> <span class="o">=</span> <span class="n">SVNPoller</span><span class="p">(</span><span class="s">"http://divmod.org/svn/Divmod"</span><span class="p">,</span>
<span class="n">split_file</span><span class="o">=</span><span class="n">my_file_splitter</span><span class="p">)</span>
</pre></div>
</div>
<p>The <tt class="docutils literal"><span class="pre">my_file_splitter</span></tt> function will be called with repository-relative
pathnames like:</p>
<dl class="docutils">
<dt><tt class="file docutils literal"><span class="pre">trunk/Nevow/formless/webform.py</span></tt></dt>
<dd>This is a Nevow file, on the trunk. We want the Change that includes this
to see a filename of <tt class="file docutils literal"><span class="pre">formless/webform.py</span></tt>, and a branch of
<tt class="docutils literal"><span class="pre">None</span></tt></dd>
<dt><tt class="file docutils literal"><span class="pre">branches/1.5.x/Nevow/formless/webform.py</span></tt></dt>
<dd>This is a Nevow file, on a branch. We want to get
<tt class="docutils literal"><span class="pre">branch='branches/1.5.x'</span></tt> and <tt class="docutils literal"><span class="pre">filename='formless/webform.py'</span></tt>.</dd>
<dt><tt class="file docutils literal"><span class="pre">trunk/Quotient/setup.py</span></tt></dt>
<dd>This is a Quotient file, so we want to ignore it by having
<tt class="xref py py-meth docutils literal"><span class="pre">my_file_splitter</span></tt> return <tt class="docutils literal"><span class="pre">None</span></tt>.</dd>
<dt><tt class="file docutils literal"><span class="pre">branches/1.5.x/Quotient/setup.py</span></tt></dt>
<dd>This is also a Quotient file, which should be ignored.</dd>
</dl>
<p>The following definition for <tt class="xref py py-meth docutils literal"><span class="pre">my_file_splitter</span></tt> will do the job:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="k">def</span> <span class="nf">my_file_splitter</span><span class="p">(</span><span class="n">path</span><span class="p">):</span>
<span class="n">pieces</span> <span class="o">=</span> <span class="n">path</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s">'/'</span><span class="p">)</span>
<span class="k">if</span> <span class="n">pieces</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">==</span> <span class="s">'trunk'</span><span class="p">:</span>
<span class="n">branch</span> <span class="o">=</span> <span class="bp">None</span>
<span class="n">pieces</span><span class="o">.</span><span class="n">pop</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="c"># remove 'trunk'</span>
<span class="k">elif</span> <span class="n">pieces</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">==</span> <span class="s">'branches'</span><span class="p">:</span>
<span class="n">pieces</span><span class="o">.</span><span class="n">pop</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="c"># remove 'branches'</span>
<span class="c"># grab branch name</span>
<span class="n">branch</span> <span class="o">=</span> <span class="s">'branches/'</span> <span class="o">+</span> <span class="n">pieces</span><span class="o">.</span><span class="n">pop</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">return</span> <span class="bp">None</span> <span class="c"># something weird</span>
<span class="n">projectname</span> <span class="o">=</span> <span class="n">pieces</span><span class="o">.</span><span class="n">pop</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
<span class="k">if</span> <span class="n">projectname</span> <span class="o">!=</span> <span class="s">'Nevow'</span><span class="p">:</span>
<span class="k">return</span> <span class="bp">None</span> <span class="c"># wrong project</span>
<span class="k">return</span> <span class="nb">dict</span><span class="p">(</span><span class="n">branch</span><span class="o">=</span><span class="n">branch</span><span class="p">,</span> <span class="n">path</span><span class="o">=</span><span class="s">'/'</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">pieces</span><span class="p">))</span>
</pre></div>
</div>
<p>If you later decide you want to get changes for Quotient as well you could
replace the last 3 lines with simply:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="k">return</span> <span class="nb">dict</span><span class="p">(</span><span class="n">project</span><span class="o">=</span><span class="n">projectname</span><span class="p">,</span> <span class="n">branch</span><span class="o">=</span><span class="n">branch</span><span class="p">,</span> <span class="n">path</span><span class="o">=</span><span class="s">'/'</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">pieces</span><span class="p">))</span>
</pre></div>
</div>
</div>
</div>
<div class="section" id="writing-change-sources">
<span id="id6"></span><h2>Writing Change Sources<a class="headerlink" href="#writing-change-sources" title="Permalink to this headline">¶</a></h2>
<p>For some version-control systems, making Buildbot aware of new changes can be a
challenge. If the pre-supplied classes in <a class="reference internal" href="cfg-changesources.html#change-sources"><em>Change Sources</em></a> are not
sufficient, then you will need to write your own.</p>
<p>There are three approaches, one of which is not even a change source.
The first option is to write a change source that exposes some service to
which the version control system can "push" changes. This can be more
complicated, since it requires implementing a new service, but delivers changes
to Buildbot immediately on commit.</p>
<p>The second option is often preferable to the first: implement a notification
service in an external process (perhaps one that is started directly by the
version control system, or by an email server) and delivers changes to Buildbot
via <a class="reference internal" href="cfg-changesources.html#pbchangesource"><em>PBChangeSource</em></a>. This section does not describe this particular
approach, since it requires no customization within the buildmaster process.</p>
<p>The third option is to write a change source which polls for changes -
repeatedly connecting to an external service to check for new changes. This
works well in many cases, but can produce a high load on the version control
system if polling is too frequent, and can take too long to notice changes if
the polling is not frequent enough.</p>
<div class="section" id="writing-a-notification-based-change-source">
<h3>Writing a Notification-based Change Source<a class="headerlink" href="#writing-a-notification-based-change-source" title="Permalink to this headline">¶</a></h3>
<dl class="class">
<dt id="buildbot.changes.base.ChangeSource">
<em class="property">class </em><tt class="descclassname">buildbot.changes.base.</tt><tt class="descname">ChangeSource</tt><a class="headerlink" href="#buildbot.changes.base.ChangeSource" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<p>A custom change source must implement
<tt class="xref py py-class docutils literal"><span class="pre">buildbot.interfaces.IChangeSource</span></tt>.</p>
<p>The easiest way to do this is to subclass
<a class="reference internal" href="#buildbot.changes.base.ChangeSource" title="buildbot.changes.base.ChangeSource"><tt class="xref py py-class docutils literal"><span class="pre">buildbot.changes.base.ChangeSource</span></tt></a>, implementing the <tt class="xref py py-meth docutils literal"><span class="pre">describe</span></tt>
method to describe the instance. <tt class="xref py py-class docutils literal"><span class="pre">ChangeSource</span></tt> is a Twisted service, so
you will need to implement the <tt class="xref py py-meth docutils literal"><span class="pre">startService</span></tt> and <tt class="xref py py-meth docutils literal"><span class="pre">stopService</span></tt>
methods to control the means by which your change source receives
notifications.</p>
<p>When the class does receive a change, it should call
<tt class="docutils literal"><span class="pre">self.master.addChange(..)</span></tt> to submit it to the buildmaster. This method
shares the same parameters as <tt class="docutils literal"><span class="pre">master.db.changes.addChange</span></tt>, so consult the
API documentation for that function for details on the available arguments.</p>
<p>You will probably also want to set <tt class="docutils literal"><span class="pre">compare_attrs</span></tt> to the list of object
attributes which Buildbot will use to compare one change source to another when
reconfiguring. During reconfiguration, if the new change source is different
from the old, then the old will be stopped and the new started.</p>
</div>
<div class="section" id="writing-a-change-poller">
<h3>Writing a Change Poller<a class="headerlink" href="#writing-a-change-poller" title="Permalink to this headline">¶</a></h3>
<dl class="class">
<dt id="buildbot.changes.base.PollingChangeSource">
<em class="property">class </em><tt class="descclassname">buildbot.changes.base.</tt><tt class="descname">PollingChangeSource</tt><a class="headerlink" href="#buildbot.changes.base.PollingChangeSource" title="Permalink to this definition">¶</a></dt>
<dd></dd></dl>
<p>Polling is a very common means of seeking changes, so Buildbot supplies a
utility parent class to make it easier. A poller should subclass
<a class="reference internal" href="#buildbot.changes.base.PollingChangeSource" title="buildbot.changes.base.PollingChangeSource"><tt class="xref py py-class docutils literal"><span class="pre">buildbot.changes.base.PollingChangeSource</span></tt></a>, which is a subclass of
<tt class="xref py py-class docutils literal"><span class="pre">ChangeSource</span></tt>. This subclass implements the <tt class="xref py py-meth docutils literal"><span class="pre">Service</span></tt> methods,
and causes the <tt class="xref py py-meth docutils literal"><span class="pre">poll</span></tt> method to be called every <tt class="docutils literal"><span class="pre">self.pollInterval</span></tt>
seconds. This method should return a Deferred to signal its completion.</p>
<p>Aside from the service methods, the other concerns in the previous section
apply here, too.</p>
</div>
</div>
<div class="section" id="writing-a-new-latent-buildslave-implementation">
<h2>Writing a New Latent Buildslave Implementation<a class="headerlink" href="#writing-a-new-latent-buildslave-implementation" title="Permalink to this headline">¶</a></h2>
<p>Writing a new latent buildslave should only require subclassing
<tt class="xref py py-class docutils literal"><span class="pre">buildbot.buildslave.AbstractLatentBuildSlave</span></tt> and implementing
<tt class="xref py py-meth docutils literal"><span class="pre">start_instance</span></tt> and <tt class="xref py py-meth docutils literal"><span class="pre">stop_instance</span></tt>.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="k">def</span> <span class="nf">start_instance</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="c"># responsible for starting instance that will try to connect with this</span>
<span class="c"># master. Should return deferred. Problems should use an errback. The</span>
<span class="c"># callback value can be None, or can be an iterable of short strings to</span>
<span class="c"># include in the "substantiate success" status message, such as</span>
<span class="c"># identifying the instance that started.</span>
<span class="k">raise</span> <span class="ne">NotImplementedError</span>
<span class="k">def</span> <span class="nf">stop_instance</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">fast</span><span class="o">=</span><span class="bp">False</span><span class="p">):</span>
<span class="c"># responsible for shutting down instance. Return a deferred. If `fast`,</span>
<span class="c"># we're trying to shut the master down, so callback as soon as is safe.</span>
<span class="c"># Callback value is ignored.</span>
<span class="k">raise</span> <span class="ne">NotImplementedError</span>
</pre></div>
</div>
<p>See <tt class="xref py py-class docutils literal"><span class="pre">buildbot.ec2buildslave.EC2LatentBuildSlave</span></tt> for an example, or see
the test example <tt class="xref py py-class docutils literal"><span class="pre">buildbot.test_slaves.FakeLatentBuildSlave</span></tt>.</p>
</div>
<div class="section" id="custom-build-classes">
<h2>Custom Build Classes<a class="headerlink" href="#custom-build-classes" title="Permalink to this headline">¶</a></h2>
<p>The standard <tt class="xref py py-class docutils literal"><span class="pre">BuildFactory</span></tt> object creates <tt class="xref py py-class docutils literal"><span class="pre">Build</span></tt> objects
by default. These Builds will each execute a collection of <tt class="xref py py-class docutils literal"><span class="pre">BuildStep</span></tt>s
in a fixed sequence. Each step can affect the results of the build,
but in general there is little intelligence to tie the different steps
together.</p>
<p>By setting the factory's <tt class="docutils literal"><span class="pre">buildClass</span></tt> attribute to a different class, you can
instantiate a different build class. This might be useful, for example, to
create a build class that dynamically determines which steps to run. The
skeleton of such a project would look like:</p>
<div class="highlight-python"><pre>class DynamicBuild(Build):
# .. override some methods
f = factory.BuildFactory()
f.buildClass = DynamicBuild
f.addStep(...)</pre>
</div>
</div>
<div class="section" id="factory-workdir-functions">
<span id="id7"></span><h2>Factory Workdir Functions<a class="headerlink" href="#factory-workdir-functions" title="Permalink to this headline">¶</a></h2>
<p>It is sometimes helpful to have a build's workdir determined at runtime based
on the parameters of the build. To accomplish this, set the <tt class="docutils literal"><span class="pre">workdir</span></tt>
attribute of the build factory to a callable. That callable will be invoked
with the <tt class="xref py py-class docutils literal"><span class="pre">SourceStamp</span></tt> for the build, and should return the appropriate
workdir. Note that the value must be returned immediately - Deferreds are not
supported.</p>
<p>This can be useful, for example, in scenarios with multiple repositories
submitting changes to BuildBot. In this case you likely will want to have a
dedicated workdir per repository, since otherwise a sourcing step with mode =
"update" will fail as a workdir with a working copy of repository A can't be
"updated" for changes from a repository B. Here is an example how you can
achieve workdir-per-repo:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="k">def</span> <span class="nf">workdir</span><span class="p">(</span><span class="n">source_stamp</span><span class="p">):</span>
<span class="k">return</span> <span class="n">hashlib</span><span class="o">.</span><span class="n">md5</span> <span class="p">(</span><span class="n">source_stamp</span><span class="o">.</span><span class="n">repository</span><span class="p">)</span><span class="o">.</span><span class="n">hexdigest</span><span class="p">()[:</span><span class="mi">8</span><span class="p">]</span>
<span class="n">build_factory</span> <span class="o">=</span> <span class="n">factory</span><span class="o">.</span><span class="n">BuildFactory</span><span class="p">()</span>
<span class="n">build_factory</span><span class="o">.</span><span class="n">workdir</span> <span class="o">=</span> <span class="n">workdir</span>
<span class="n">build_factory</span><span class="o">.</span><span class="n">addStep</span><span class="p">(</span><span class="n">Git</span><span class="p">(</span><span class="n">mode</span><span class="o">=</span><span class="s">"update"</span><span class="p">))</span>
<span class="c"># ...</span>
<span class="n">builders</span><span class="o">.</span><span class="n">append</span> <span class="p">({</span><span class="s">'name'</span><span class="p">:</span> <span class="s">'mybuilder'</span><span class="p">,</span>
<span class="s">'slavename'</span><span class="p">:</span> <span class="s">'myslave'</span><span class="p">,</span>
<span class="s">'builddir'</span><span class="p">:</span> <span class="s">'mybuilder'</span><span class="p">,</span>
<span class="s">'factory'</span><span class="p">:</span> <span class="n">build_factory</span><span class="p">})</span>
</pre></div>
</div>
<p>The end result is a set of workdirs like</p>
<div class="highlight-none"><div class="highlight"><pre>Repo1 => <buildslave-base>/mybuilder/a78890ba
Repo2 => <buildslave-base>/mybuilder/0823ba88
</pre></div>
</div>
<p>You could make the <tt class="xref py py-func docutils literal"><span class="pre">workdir</span></tt> function compute other paths, based on
parts of the repo URL in the sourcestamp, or lookup in a lookup table
based on repo URL. As long as there is a permanent 1:1 mapping between
repos and workdir, this will work.</p>
</div>
<div class="section" id="writing-new-buildsteps">
<h2>Writing New BuildSteps<a class="headerlink" href="#writing-new-buildsteps" title="Permalink to this headline">¶</a></h2>
<p>While it is a good idea to keep your build process self-contained in the source code tree, sometimes it is convenient to put more intelligence into your Buildbot configuration.
One way to do this is to write a custom <tt class="xref py py-class docutils literal"><span class="pre">BuildStep</span></tt>.
Once written, this Step can be used in the <tt class="file docutils literal"><span class="pre">master.cfg</span></tt> file.</p>
<p>The best reason for writing a custom <tt class="xref py py-class docutils literal"><span class="pre">BuildStep</span></tt> is to better parse the results of the command being run.
For example, a <tt class="xref py py-class docutils literal"><span class="pre">BuildStep</span></tt> that knows about JUnit could look at the logfiles to determine which tests had been run, how many passed and how many failed, and then report more detailed information than a simple <tt class="docutils literal"><span class="pre">rc==0</span></tt> -based <cite>good/bad</cite> decision.</p>
<p>Buildbot has acquired a large fleet of build steps, and sports a number of knobs and hooks to make steps easier to write.
This section may seem a bit overwhelming, but most custom steps will only need to apply one or two of the techniques outlined here.</p>
<p>For complete documentation of the build step interfaces, see <a class="reference internal" href="../developer/cls-buildsteps.html"><em>BuildSteps</em></a>.</p>
<div class="section" id="writing-buildstep-constructors">
<span id="id8"></span><h3>Writing BuildStep Constructors<a class="headerlink" href="#writing-buildstep-constructors" title="Permalink to this headline">¶</a></h3>
<p>Build steps act as their own factories, so their constructors are a bit more complex than necessary.
In the configuration file, a <a class="reference internal" href="../developer/cls-buildsteps.html#buildbot.process.buildstep.BuildStep" title="buildbot.process.buildstep.BuildStep"><tt class="xref py py-class docutils literal"><span class="pre">BuildStep</span></tt></a> object is instantiated, but because steps store state locally while executing, this object cannot be used during builds.</p>
<p>Consider the use of a <tt class="xref py py-class docutils literal"><span class="pre">BuildStep</span></tt> in <tt class="file docutils literal"><span class="pre">master.cfg</span></tt>:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="n">f</span><span class="o">.</span><span class="n">addStep</span><span class="p">(</span><span class="n">MyStep</span><span class="p">(</span><span class="n">someopt</span><span class="o">=</span><span class="s">"stuff"</span><span class="p">,</span> <span class="n">anotheropt</span><span class="o">=</span><span class="mi">1</span><span class="p">))</span>
</pre></div>
</div>
<p>This creates a single instance of class <tt class="docutils literal"><span class="pre">MyStep</span></tt>.
However, Buildbot needs a new object each time the step is executed.
An instance of <a class="reference internal" href="../developer/cls-buildsteps.html#buildbot.process.buildstep.BuildStep" title="buildbot.process.buildstep.BuildStep"><tt class="xref py py-class docutils literal"><span class="pre">BuildStep</span></tt></a> remembers how it was constructed, and can create copies of itself.
When writing a new step class, then, keep in mind are that you cannot do anything "interesting" in the constructor -- limit yourself to checking and storing arguments.</p>
<p>It is customary to call the parent class's constructor with all otherwise-unspecified keyword arguments.
Keep a <tt class="docutils literal"><span class="pre">**kwargs</span></tt> argument on the end of your options, and pass that up to the parent class's constructor.</p>
<p>The whole thing looks like this:</p>
<div class="highlight-python"><pre>class Frobnify(LoggingBuildStep):
def __init__(self,
frob_what="frobee",
frob_how_many=None,
frob_how=None,
**kwargs):
# check
if frob_how_many is None:
raise TypeError("Frobnify argument how_many is required")
# override a parent option
kwargs['parentOpt'] = 'xyz'
# call parent
LoggingBuildStep.__init__(self, **kwargs)
# set Frobnify attributes
self.frob_what = frob_what
self.frob_how_many = how_many
self.frob_how = frob_how
class FastFrobnify(Frobnify):
def __init__(self,
speed=5,
**kwargs)
Frobnify.__init__(self, **kwargs)
self.speed = speed</pre>
</div>
</div>
<div class="section" id="running-commands">
<h3>Running Commands<a class="headerlink" href="#running-commands" title="Permalink to this headline">¶</a></h3>
<p>To spawn a command in the buildslave, create a <a class="reference internal" href="../developer/cls-remotecommands.html#buildbot.process.buildstep.RemoteCommand" title="buildbot.process.buildstep.RemoteCommand"><tt class="xref py py-class docutils literal"><span class="pre">RemoteCommand</span></tt></a> instance in your step's <tt class="docutils literal"><span class="pre">start</span></tt> method and run it with <a class="reference internal" href="../developer/cls-buildsteps.html#buildbot.process.buildstep.BuildStep.runCommand" title="buildbot.process.buildstep.BuildStep.runCommand"><tt class="xref py py-meth docutils literal"><span class="pre">runCommand</span></tt></a>:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="n">cmd</span> <span class="o">=</span> <span class="n">RemoteCommand</span><span class="p">(</span><span class="n">args</span><span class="p">)</span>
<span class="n">d</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">runCommand</span><span class="p">(</span><span class="n">cmd</span><span class="p">)</span>
</pre></div>
</div>
<p>To add a LogFile, use <a class="reference internal" href="../developer/cls-buildsteps.html#buildbot.process.buildstep.BuildStep.addLog" title="buildbot.process.buildstep.BuildStep.addLog"><tt class="xref py py-meth docutils literal"><span class="pre">addLog</span></tt></a>.
Make sure the log gets closed when it finishes.
When giving a Logfile to a <a class="reference internal" href="../developer/cls-remotecommands.html#buildbot.process.buildstep.RemoteShellCommand" title="buildbot.process.buildstep.RemoteShellCommand"><tt class="xref py py-class docutils literal"><span class="pre">RemoteShellCommand</span></tt></a>, just ask it to close the log when the command completes:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="n">log</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">addLog</span><span class="p">(</span><span class="s">'output'</span><span class="p">)</span>
<span class="n">cmd</span><span class="o">.</span><span class="n">useLog</span><span class="p">(</span><span class="n">log</span><span class="p">,</span> <span class="n">closeWhenFinished</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="section" id="updating-status">
<h3>Updating Status<a class="headerlink" href="#updating-status" title="Permalink to this headline">¶</a></h3>
<p>TBD</p>
</div>
<div class="section" id="capturing-logfiles">
<h3>Capturing Logfiles<a class="headerlink" href="#capturing-logfiles" title="Permalink to this headline">¶</a></h3>
<p>Each BuildStep has a collection of <cite>logfiles</cite>. Each one has a short
name, like <cite>stdio</cite> or <cite>warnings</cite>. Each <tt class="xref py py-class docutils literal"><span class="pre">LogFile</span></tt> contains an
arbitrary amount of text, usually the contents of some output file
generated during a build or test step, or a record of everything that
was printed to <tt class="file docutils literal"><span class="pre">stdout</span></tt>/<tt class="file docutils literal"><span class="pre">stderr</span></tt> during the execution of some command.</p>
<p>These <tt class="xref py py-class docutils literal"><span class="pre">LogFile</span></tt>s are stored to disk, so they can be retrieved later.</p>
<p>Each can contain multiple <cite>channels</cite>, generally limited to three
basic ones: stdout, stderr, and <cite>headers</cite>. For example, when a
ShellCommand runs, it writes a few lines to the <cite>headers</cite> channel to
indicate the exact argv strings being run, which directory the command
is being executed in, and the contents of the current environment
variables. Then, as the command runs, it adds a lot of <tt class="file docutils literal"><span class="pre">stdout</span></tt> and
<tt class="file docutils literal"><span class="pre">stderr</span></tt> messages. When the command finishes, a final <cite>header</cite>
line is added with the exit code of the process.</p>
<p>Status display plugins can format these different channels in
different ways. For example, the web page shows LogFiles as text/html,
with header lines in blue text, stdout in black, and stderr in red. A
different URL is available which provides a text/plain format, in
which stdout and stderr are collapsed together, and header lines are
stripped completely. This latter option makes it easy to save the
results to a file and run <strong class="command">grep</strong> or whatever against the
output.</p>
<p>Each <tt class="xref py py-class docutils literal"><span class="pre">BuildStep</span></tt> contains a mapping (implemented in a Python dictionary)
from <tt class="xref py py-class docutils literal"><span class="pre">LogFile</span></tt> name to the actual <tt class="xref py py-class docutils literal"><span class="pre">LogFile</span></tt> objects. Status plugins can
get a list of LogFiles to display, for example, a list of HREF links
that, when clicked, provide the full contents of the <tt class="xref py py-class docutils literal"><span class="pre">LogFile</span></tt>.</p>
<div class="section" id="using-logfiles-in-custom-buildsteps">
<h4>Using LogFiles in custom BuildSteps<a class="headerlink" href="#using-logfiles-in-custom-buildsteps" title="Permalink to this headline">¶</a></h4>
<p>The most common way for a custom <tt class="xref py py-class docutils literal"><span class="pre">BuildStep</span></tt> to use a <tt class="xref py py-class docutils literal"><span class="pre">LogFile</span></tt> is to
summarize the results of a <a class="reference internal" href="cfg-buildsteps.html#step-ShellCommand" title="ShellCommand"><tt class="xref bb bb-step docutils literal"><span class="pre">ShellCommand</span></tt></a> (after the command has
finished running). For example, a compile step with thousands of lines
of output might want to create a summary of just the warning messages.
If you were doing this from a shell, you would use something like:</p>
<div class="highlight-bash"><div class="highlight"><pre>grep <span class="s2">"warning:"</span> output.log >warnings.log
</pre></div>
</div>
<p>In a custom BuildStep, you could instead create a <tt class="docutils literal"><span class="pre">warnings</span></tt> <tt class="xref py py-class docutils literal"><span class="pre">LogFile</span></tt>
that contained the same text. To do this, you would add code to your
<tt class="xref py py-meth docutils literal"><span class="pre">createSummary</span></tt> method that pulls lines from the main output log
and creates a new <tt class="xref py py-class docutils literal"><span class="pre">LogFile</span></tt> with the results:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="k">def</span> <span class="nf">createSummary</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">log</span><span class="p">):</span>
<span class="n">warnings</span> <span class="o">=</span> <span class="p">[]</span>
<span class="n">sio</span> <span class="o">=</span> <span class="n">StringIO</span><span class="o">.</span><span class="n">StringIO</span><span class="p">(</span><span class="n">log</span><span class="o">.</span><span class="n">getText</span><span class="p">())</span>
<span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">sio</span><span class="o">.</span><span class="n">readlines</span><span class="p">():</span>
<span class="k">if</span> <span class="s">"warning:"</span> <span class="ow">in</span> <span class="n">line</span><span class="p">:</span>
<span class="n">warnings</span><span class="o">.</span><span class="n">append</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">addCompleteLog</span><span class="p">(</span><span class="s">'warnings'</span><span class="p">,</span> <span class="s">""</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">warnings</span><span class="p">))</span>
</pre></div>
</div>
<p>This example uses the <tt class="xref py py-meth docutils literal"><span class="pre">addCompleteLog</span></tt> method, which creates a
new <tt class="xref py py-class docutils literal"><span class="pre">LogFile</span></tt>, puts some text in it, and then <cite>closes</cite> it, meaning
that no further contents will be added. This <tt class="xref py py-class docutils literal"><span class="pre">LogFile</span></tt> will appear in
the HTML display under an HREF with the name <cite>warnings</cite>, since that
is the name of the <tt class="xref py py-class docutils literal"><span class="pre">LogFile</span></tt>.</p>
<p>You can also use <tt class="xref py py-meth docutils literal"><span class="pre">addHTMLLog</span></tt> to create a complete (closed)
<tt class="xref py py-class docutils literal"><span class="pre">LogFile</span></tt> that contains HTML instead of plain text. The normal <tt class="xref py py-class docutils literal"><span class="pre">LogFile</span></tt>
will be HTML-escaped if presented through a web page, but the HTML
<tt class="xref py py-class docutils literal"><span class="pre">LogFile</span></tt> will not. At the moment this is only used to present a pretty
HTML representation of an otherwise ugly exception traceback when
something goes badly wrong during the <tt class="xref py py-class docutils literal"><span class="pre">BuildStep</span></tt>.</p>
<p>In contrast, you might want to create a new <tt class="xref py py-class docutils literal"><span class="pre">LogFile</span></tt> at the beginning
of the step, and add text to it as the command runs. You can create
the <tt class="xref py py-class docutils literal"><span class="pre">LogFile</span></tt> and attach it to the build by calling <tt class="xref py py-meth docutils literal"><span class="pre">addLog</span></tt>, which
returns the <tt class="xref py py-class docutils literal"><span class="pre">LogFile</span></tt> object. You then add text to this <tt class="xref py py-class docutils literal"><span class="pre">LogFile</span></tt> by
calling methods like <tt class="xref py py-meth docutils literal"><span class="pre">addStdout</span></tt> and <tt class="xref py py-meth docutils literal"><span class="pre">addHeader</span></tt>. When you
are done, you must call the <tt class="xref py py-meth docutils literal"><span class="pre">finish</span></tt> method so the <tt class="xref py py-class docutils literal"><span class="pre">LogFile</span></tt> can be
closed. It may be useful to create and populate a <tt class="xref py py-class docutils literal"><span class="pre">LogFile</span></tt> like this
from a <tt class="xref py py-class docutils literal"><span class="pre">LogObserver</span></tt> method - see <a class="reference internal" href="#adding-logobservers"><em>Adding LogObservers</em></a>.</p>
<p>The <tt class="docutils literal"><span class="pre">logfiles=</span></tt> argument to <a class="reference internal" href="cfg-buildsteps.html#step-ShellCommand" title="ShellCommand"><tt class="xref bb bb-step docutils literal"><span class="pre">ShellCommand</span></tt></a> (see
<a class="reference internal" href="cfg-buildsteps.html#step-ShellCommand" title="ShellCommand"><tt class="xref bb bb-step docutils literal"><span class="pre">ShellCommand</span></tt></a>) creates new <tt class="xref py py-class docutils literal"><span class="pre">LogFile</span></tt>s and fills them in realtime
by asking the buildslave to watch a actual file on disk. The
buildslave will look for additions in the target file and report them
back to the <tt class="xref py py-class docutils literal"><span class="pre">BuildStep</span></tt>. These additions will be added to the <tt class="xref py py-class docutils literal"><span class="pre">LogFile</span></tt> by
calling <tt class="xref py py-meth docutils literal"><span class="pre">addStdout</span></tt>. These secondary LogFiles can be used as the
source of a LogObserver just like the normal <tt class="file docutils literal"><span class="pre">stdio</span></tt> <tt class="xref py py-class docutils literal"><span class="pre">LogFile</span></tt>.</p>
</div>
</div>
<div class="section" id="reading-logfiles">
<h3>Reading Logfiles<a class="headerlink" href="#reading-logfiles" title="Permalink to this headline">¶</a></h3>
<p>Once a <a class="reference internal" href="../developer/formats.html#buildbot.status.logfile.LogFile" title="buildbot.status.logfile.LogFile"><tt class="xref py py-class docutils literal"><span class="pre">LogFile</span></tt></a> has been added to a
<a class="reference internal" href="../developer/cls-buildsteps.html#buildbot.process.buildstep.BuildStep" title="buildbot.process.buildstep.BuildStep"><tt class="xref py py-class docutils literal"><span class="pre">BuildStep</span></tt></a> with
<a class="reference internal" href="../developer/cls-buildsteps.html#buildbot.process.buildstep.BuildStep.addLog" title="buildbot.process.buildstep.BuildStep.addLog"><tt class="xref py py-meth docutils literal"><span class="pre">addLog</span></tt></a>,
<a class="reference internal" href="../developer/cls-buildsteps.html#buildbot.process.buildstep.BuildStep.addCompleteLog" title="buildbot.process.buildstep.BuildStep.addCompleteLog"><tt class="xref py py-meth docutils literal"><span class="pre">addCompleteLog</span></tt></a>,
<a class="reference internal" href="../developer/cls-buildsteps.html#buildbot.process.buildstep.BuildStep.addHTMLLog" title="buildbot.process.buildstep.BuildStep.addHTMLLog"><tt class="xref py py-meth docutils literal"><span class="pre">addHTMLLog</span></tt></a>, or <tt class="docutils literal"><span class="pre">logfiles={}</span></tt>,
your <tt class="xref py py-class docutils literal"><span class="pre">BuildStep</span></tt> can retrieve it
by using <a class="reference internal" href="../developer/cls-buildsteps.html#buildbot.process.buildstep.BuildStep.getLog" title="buildbot.process.buildstep.BuildStep.getLog"><tt class="xref py py-meth docutils literal"><span class="pre">getLog</span></tt></a>:</p>
<div class="highlight-python"><pre>class MyBuildStep(ShellCommand):
logfiles = @{ "nodelog": "_test/node.log" @}
def evaluateCommand(self, cmd):
nodelog = self.getLog("nodelog")
if "STARTED" in nodelog.getText():
return SUCCESS
else:
return FAILURE</pre>
</div>
</div>
<div class="section" id="adding-logobservers">
<span id="id9"></span><h3>Adding LogObservers<a class="headerlink" href="#adding-logobservers" title="Permalink to this headline">¶</a></h3>
<p>Most shell commands emit messages to stdout or stderr as they operate,
especially if you ask them nicely with a <em class="xref std std-option">--verbose</em> flag of some
sort. They may also write text to a log file while they run. Your
<tt class="xref py py-class docutils literal"><span class="pre">BuildStep</span></tt> can watch this output as it arrives, to keep track of how
much progress the command has made. You can get a better measure of
progress by counting the number of source files compiled or test cases
run than by merely tracking the number of bytes that have been written
to stdout. This improves the accuracy and the smoothness of the ETA
display.</p>
<p>To accomplish this, you will need to attach a <tt class="xref py py-class docutils literal"><span class="pre">LogObserver</span></tt> to
one of the log channels, most commonly to the <tt class="file docutils literal"><span class="pre">stdio</span></tt> channel but
perhaps to another one which tracks a log file. This observer is given
all text as it is emitted from the command, and has the opportunity to
parse that output incrementally. Once the observer has decided that
some event has occurred (like a source file being compiled), it can
use the <tt class="xref py py-meth docutils literal"><span class="pre">setProgress</span></tt> method to tell the <tt class="xref py py-class docutils literal"><span class="pre">BuildStep</span></tt> about the
progress that this event represents.</p>
<p>There are a number of pre-built <tt class="xref py py-class docutils literal"><span class="pre">LogObserver</span></tt> classes that you
can choose from (defined in <a class="reference internal" href="../developer/cls-buildsteps.html#module-buildbot.process.buildstep" title="buildbot.process.buildstep"><tt class="xref py py-mod docutils literal"><span class="pre">buildbot.process.buildstep</span></tt></a>, and of
course you can subclass them to add further customization. The
<tt class="xref py py-class docutils literal"><span class="pre">LogLineObserver</span></tt> class handles the grunt work of buffering and
scanning for end-of-line delimiters, allowing your parser to operate
on complete <tt class="file docutils literal"><span class="pre">stdout</span></tt>/<tt class="file docutils literal"><span class="pre">stderr</span></tt> lines. (Lines longer than a set maximum
length are dropped; the maximum defaults to 16384 bytes, but you can
change it by calling <tt class="xref py py-meth docutils literal"><span class="pre">setMaxLineLength</span></tt> on your
<tt class="xref py py-class docutils literal"><span class="pre">LogLineObserver</span></tt> instance. Use <tt class="docutils literal"><span class="pre">sys.maxint</span></tt> for effective
infinity.)</p>
<p>For example, let's take a look at the <tt class="xref py py-class docutils literal"><span class="pre">TrialTestCaseCounter</span></tt>,
which is used by the <a class="reference internal" href="cfg-buildsteps.html#step-Trial" title="Trial"><tt class="xref bb bb-step docutils literal"><span class="pre">Trial</span></tt></a> step to count test cases as they are run.
As Trial executes, it emits lines like the following:</p>
<div class="highlight-none"><div class="highlight"><pre>buildbot.test.test_config.ConfigTest.testDebugPassword ... [OK]
buildbot.test.test_config.ConfigTest.testEmpty ... [OK]
buildbot.test.test_config.ConfigTest.testIRC ... [FAIL]
buildbot.test.test_config.ConfigTest.testLocks ... [OK]
</pre></div>
</div>
<p>When the tests are finished, trial emits a long line of <cite>======</cite> and
then some lines which summarize the tests that failed. We want to
avoid parsing these trailing lines, because their format is less
well-defined than the <cite>[OK]</cite> lines.</p>
<p>The parser class looks like this:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="kn">from</span> <span class="nn">buildbot.process.buildstep</span> <span class="kn">import</span> <span class="n">LogLineObserver</span>
<span class="k">class</span> <span class="nc">TrialTestCaseCounter</span><span class="p">(</span><span class="n">LogLineObserver</span><span class="p">):</span>
<span class="n">_line_re</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">compile</span><span class="p">(</span><span class="s">r'^([\w\.]+) \.\.\. \[([^\]]+)\]$'</span><span class="p">)</span>
<span class="n">numTests</span> <span class="o">=</span> <span class="mi">0</span>
<span class="n">finished</span> <span class="o">=</span> <span class="bp">False</span>
<span class="k">def</span> <span class="nf">outLineReceived</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">line</span><span class="p">):</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">finished</span><span class="p">:</span>
<span class="k">return</span>
<span class="k">if</span> <span class="n">line</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s">"="</span> <span class="o">*</span> <span class="mi">40</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">finished</span> <span class="o">=</span> <span class="bp">True</span>
<span class="k">return</span>
<span class="n">m</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_line_re</span><span class="o">.</span><span class="n">search</span><span class="p">(</span><span class="n">line</span><span class="o">.</span><span class="n">strip</span><span class="p">())</span>
<span class="k">if</span> <span class="n">m</span><span class="p">:</span>
<span class="n">testname</span><span class="p">,</span> <span class="n">result</span> <span class="o">=</span> <span class="n">m</span><span class="o">.</span><span class="n">groups</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">numTests</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="bp">self</span><span class="o">.</span><span class="n">step</span><span class="o">.</span><span class="n">setProgress</span><span class="p">(</span><span class="s">'tests'</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">numTests</span><span class="p">)</span>
</pre></div>
</div>
<p>This parser only pays attention to stdout, since that's where trial
writes the progress lines. It has a mode flag named <tt class="docutils literal"><span class="pre">finished</span></tt> to
ignore everything after the <tt class="docutils literal"><span class="pre">====</span></tt> marker, and a scary-looking
regular expression to match each line while hopefully ignoring other
messages that might get displayed as the test runs.</p>
<p>Each time it identifies a test has been completed, it increments its
counter and delivers the new progress value to the step with
@code{self.step.setProgress}. This class is specifically measuring
progress along the <cite>tests</cite> metric, in units of test cases (as
opposed to other kinds of progress like the <cite>output</cite> metric, which
measures in units of bytes). The Progress-tracking code uses each
progress metric separately to come up with an overall completion
percentage and an ETA value.</p>
<p>To connect this parser into the <a class="reference internal" href="cfg-buildsteps.html#step-Trial" title="Trial"><tt class="xref bb bb-step docutils literal"><span class="pre">Trial</span></tt></a> build step,
<tt class="docutils literal"><span class="pre">Trial.__init__</span></tt> ends with the following clause:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="c"># this counter will feed Progress along the 'test cases' metric</span>
<span class="n">counter</span> <span class="o">=</span> <span class="n">TrialTestCaseCounter</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">addLogObserver</span><span class="p">(</span><span class="s">'stdio'</span><span class="p">,</span> <span class="n">counter</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">progressMetrics</span> <span class="o">+=</span> <span class="p">(</span><span class="s">'tests'</span><span class="p">,)</span>
</pre></div>
</div>
<p>This creates a <tt class="xref py py-class docutils literal"><span class="pre">TrialTestCaseCounter</span></tt> and tells the step that the
counter wants to watch the <tt class="file docutils literal"><span class="pre">stdio</span></tt> log. The observer is
automatically given a reference to the step in its <tt class="xref py py-attr docutils literal"><span class="pre">step</span></tt>
attribute.</p>
</div>
<div class="section" id="using-properties">
<h3>Using Properties<a class="headerlink" href="#using-properties" title="Permalink to this headline">¶</a></h3>
<p>In custom <tt class="xref py py-class docutils literal"><span class="pre">BuildSteps</span></tt>, you can get and set the build properties with
the <a class="reference internal" href="../developer/cls-iproperties.html#getProperty" title="getProperty"><tt class="xref py py-meth docutils literal"><span class="pre">getProperty</span></tt></a>/<a class="reference internal" href="../developer/cls-iproperties.html#setProperty" title="setProperty"><tt class="xref py py-meth docutils literal"><span class="pre">setProperty</span></tt></a> methods. Each takes a string
for the name of the property, and returns or accepts an
arbitrary object. For example:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="k">class</span> <span class="nc">MakeTarball</span><span class="p">(</span><span class="n">ShellCommand</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">start</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">getProperty</span><span class="p">(</span><span class="s">"os"</span><span class="p">)</span> <span class="o">==</span> <span class="s">"win"</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">setCommand</span><span class="p">([</span> <span class="o">...</span> <span class="p">])</span> <span class="c"># windows-only command</span>
<span class="k">else</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">setCommand</span><span class="p">([</span> <span class="o">...</span> <span class="p">])</span> <span class="c"># equivalent for other systems</span>
<span class="n">ShellCommand</span><span class="o">.</span><span class="n">start</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span>
</pre></div>
</div>
<p>Remember that properties set in a step may not be available until the next step
begins. In particular, any <tt class="xref py py-class docutils literal"><span class="pre">Property</span></tt> or <tt class="xref py py-class docutils literal"><span class="pre">Interpolate</span></tt>
instances for the current step are interpolated before the <tt class="docutils literal"><span class="pre">start</span></tt> method
begins.</p>
</div>
<div class="section" id="buildstep-urls">
<span id="index-4"></span><h3>BuildStep URLs<a class="headerlink" href="#buildstep-urls" title="Permalink to this headline">¶</a></h3>
<p>Each BuildStep has a collection of <cite>links</cite>. Like its collection of
LogFiles, each link has a name and a target URL. The web status page
creates HREFs for each link in the same box as it does for LogFiles,
except that the target of the link is the external URL instead of an
internal link to a page that shows the contents of the LogFile.</p>
<p>These external links can be used to point at build information hosted
on other servers. For example, the test process might produce an
intricate description of which tests passed and failed, or some sort
of code coverage data in HTML form, or a PNG or GIF image with a graph
of memory usage over time. The external link can provide an easy way
for users to navigate from the buildbot's status page to these
external web sites or file servers. Note that the step itself is
responsible for insuring that there will be a document available at
the given URL (perhaps by using <strong class="command">scp</strong> to copy the HTML output
to a <tt class="file docutils literal"><span class="pre">~/public_html/</span></tt> directory on a remote web server). Calling
<tt class="xref py py-meth docutils literal"><span class="pre">addURL</span></tt> does not magically populate a web server.</p>
<p>To set one of these links, the <tt class="xref py py-class docutils literal"><span class="pre">BuildStep</span></tt> should call the <tt class="xref py py-meth docutils literal"><span class="pre">addURL</span></tt>
method with the name of the link and the target URL. Multiple URLs can
be set.</p>
<p>In this example, we assume that the <tt class="docutils literal"><span class="pre">make</span> <span class="pre">test</span></tt> command causes
a collection of HTML files to be created and put somewhere on the
coverage.example.org web server, in a filename that incorporates the
build number.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="k">class</span> <span class="nc">TestWithCodeCoverage</span><span class="p">(</span><span class="n">BuildStep</span><span class="p">):</span>
<span class="n">command</span> <span class="o">=</span> <span class="p">[</span><span class="s">"make"</span><span class="p">,</span> <span class="s">"test"</span><span class="p">,</span>
<span class="n">Interpolate</span><span class="p">(</span><span class="s">"buildnum=%(prop:buildnumber)s"</span><span class="p">)]</span>
<span class="k">def</span> <span class="nf">createSummary</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">log</span><span class="p">):</span>
<span class="n">buildnumber</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">getProperty</span><span class="p">(</span><span class="s">"buildnumber"</span><span class="p">)</span>
<span class="n">url</span> <span class="o">=</span> <span class="s">"http://coverage.example.org/builds/</span><span class="si">%s</span><span class="s">.html"</span> <span class="o">%</span> <span class="n">buildnumber</span>
<span class="bp">self</span><span class="o">.</span><span class="n">addURL</span><span class="p">(</span><span class="s">"coverage"</span><span class="p">,</span> <span class="n">url</span><span class="p">)</span>
</pre></div>
</div>
<p>You might also want to extract the URL from some special message
output by the build process itself:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="k">class</span> <span class="nc">TestWithCodeCoverage</span><span class="p">(</span><span class="n">BuildStep</span><span class="p">):</span>
<span class="n">command</span> <span class="o">=</span> <span class="p">[</span><span class="s">"make"</span><span class="p">,</span> <span class="s">"test"</span><span class="p">,</span>
<span class="n">Interpolate</span><span class="p">(</span><span class="s">"buildnum=%(prop:buildnumber)s"</span><span class="p">)]</span>
<span class="k">def</span> <span class="nf">createSummary</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">log</span><span class="p">):</span>
<span class="n">output</span> <span class="o">=</span> <span class="n">StringIO</span><span class="p">(</span><span class="n">log</span><span class="o">.</span><span class="n">getText</span><span class="p">())</span>
<span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">output</span><span class="o">.</span><span class="n">readlines</span><span class="p">():</span>
<span class="k">if</span> <span class="n">line</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s">"coverage-url:"</span><span class="p">):</span>
<span class="n">url</span> <span class="o">=</span> <span class="n">line</span><span class="p">[</span><span class="nb">len</span><span class="p">(</span><span class="s">"coverage-url:"</span><span class="p">):]</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">addURL</span><span class="p">(</span><span class="s">"coverage"</span><span class="p">,</span> <span class="n">url</span><span class="p">)</span>
<span class="k">return</span>
</pre></div>
</div>
<p>Note that a build process which emits both <tt class="file docutils literal"><span class="pre">stdout</span></tt> and <tt class="file docutils literal"><span class="pre">stderr</span></tt> might
cause this line to be split or interleaved between other lines. It
might be necessary to restrict the <tt class="xref py py-meth docutils literal"><span class="pre">getText</span></tt> call to only stdout with
something like this:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="n">output</span> <span class="o">=</span> <span class="n">StringIO</span><span class="p">(</span><span class="s">""</span><span class="o">.</span><span class="n">join</span><span class="p">([</span><span class="n">c</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
<span class="k">for</span> <span class="n">c</span> <span class="ow">in</span> <span class="n">log</span><span class="o">.</span><span class="n">getChunks</span><span class="p">()</span>
<span class="k">if</span> <span class="n">c</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">==</span> <span class="n">LOG_CHANNEL_STDOUT</span><span class="p">]))</span>
</pre></div>
</div>
<p>Of course if the build is run under a PTY, then stdout and stderr will
be merged before the buildbot ever sees them, so such interleaving
will be unavoidable.</p>
</div>
<div class="section" id="a-somewhat-whimsical-example">
<h3>A Somewhat Whimsical Example<a class="headerlink" href="#a-somewhat-whimsical-example" title="Permalink to this headline">¶</a></h3>
<p>Let's say that we've got some snazzy new unit-test framework called
Framboozle. It's the hottest thing since sliced bread. It slices, it
dices, it runs unit tests like there's no tomorrow. Plus if your unit
tests fail, you can use its name for a Web 2.1 startup company, make
millions of dollars, and hire engineers to fix the bugs for you, while
you spend your afternoons lazily hang-gliding along a scenic pacific
beach, blissfully unconcerned about the state of your
tests. <a class="footnote-reference" href="#framboozle-reg" id="id10">[1]</a></p>
<p>To run a Framboozle-enabled test suite, you just run the 'framboozler'
command from the top of your source code tree. The 'framboozler'
command emits a bunch of stuff to stdout, but the most interesting bit
is that it emits the line "FNURRRGH!" every time it finishes running a
test case You'd like to have a test-case counting LogObserver that
watches for these lines and counts them, because counting them will
help the buildbot more accurately calculate how long the build will
take, and this will let you know exactly how long you can sneak out of
the office for your hang-gliding lessons without anyone noticing that
you're gone.</p>
<p>This will involve writing a new <tt class="xref py py-class docutils literal"><span class="pre">BuildStep</span></tt> (probably named
"Framboozle") which inherits from <a class="reference internal" href="cfg-buildsteps.html#step-ShellCommand" title="ShellCommand"><tt class="xref bb bb-step docutils literal"><span class="pre">ShellCommand</span></tt></a>. The <tt class="xref py py-class docutils literal"><span class="pre">BuildStep</span></tt> class
definition itself will look something like this:</p>
<div class="highlight-python"><pre>from buildbot.steps.shell import ShellCommand
from buildbot.process.buildstep import LogLineObserver
class FNURRRGHCounter(LogLineObserver):
numTests = 0
def outLineReceived(self, line):
if "FNURRRGH!" in line:
self.numTests += 1
self.step.setProgress('tests', self.numTests)
class Framboozle(ShellCommand):
command = ["framboozler"]
def __init__(self, **kwargs):
ShellCommand.__init__(self, **kwargs) # always upcall!
counter = FNURRRGHCounter())
self.addLogObserver('stdio', counter)
self.progressMetrics += ('tests',)</pre>
</div>
<p>So that's the code that we want to wind up using. How do we actually
deploy it?</p>
<p>You have a couple of different options.</p>
<p>Option 1: The simplest technique is to simply put this text
(everything from START to FINISH) in your <tt class="FILE docutils literal"><span class="pre">master.cfg</span></tt> file, somewhere
before the <tt class="xref py py-class docutils literal"><span class="pre">BuildFactory</span></tt> definition where you actually use it in a
clause like:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="n">f</span> <span class="o">=</span> <span class="n">BuildFactory</span><span class="p">()</span>
<span class="n">f</span><span class="o">.</span><span class="n">addStep</span><span class="p">(</span><span class="n">SVN</span><span class="p">(</span><span class="n">svnurl</span><span class="o">=</span><span class="s">"stuff"</span><span class="p">))</span>
<span class="n">f</span><span class="o">.</span><span class="n">addStep</span><span class="p">(</span><span class="n">Framboozle</span><span class="p">())</span>
</pre></div>
</div>
<p>Remember that <tt class="file docutils literal"><span class="pre">master.cfg</span></tt> is secretly just a Python program with one
job: populating the <tt class="file docutils literal"><span class="pre">BuildmasterConfig</span></tt> dictionary. And Python programs
are allowed to define as many classes as they like. So you can define
classes and use them in the same file, just as long as the class is
defined before some other code tries to use it.</p>
<p>This is easy, and it keeps the point of definition very close to the
point of use, and whoever replaces you after that unfortunate
hang-gliding accident will appreciate being able to easily figure out
what the heck this stupid "Framboozle" step is doing anyways. The
downside is that every time you reload the config file, the Framboozle
class will get redefined, which means that the buildmaster will think
that you've reconfigured all the Builders that use it, even though
nothing changed. Bleh.</p>
<p>Option 2: Instead, we can put this code in a separate file, and import
it into the master.cfg file just like we would the normal buildsteps
like <a class="reference internal" href="cfg-buildsteps.html#step-ShellCommand" title="ShellCommand"><tt class="xref bb bb-step docutils literal"><span class="pre">ShellCommand</span></tt></a> and <a class="reference internal" href="cfg-buildsteps.html#step-SVN" title="SVN"><tt class="xref bb bb-step docutils literal"><span class="pre">SVN</span></tt></a>.</p>
<p>Create a directory named ~/lib/python, put everything from START to
FINISH in <tt class="file docutils literal"><span class="pre">~/lib/python/framboozle.py</span></tt>, and run your buildmaster using:</p>
<div class="highlight-bash"><div class="highlight"><pre><span class="nv">PYTHONPATH</span><span class="o">=</span>~/lib/python buildbot start MASTERDIR
</pre></div>
</div>
<p>or use the <tt class="file docutils literal"><span class="pre">Makefile.buildbot</span></tt> to control the way
<tt class="docutils literal"><span class="pre">buildbot</span> <span class="pre">start</span></tt> works. Or add something like this to
something like your <tt class="file docutils literal"><span class="pre">~/.bashrc</span></tt> or <tt class="file docutils literal"><span class="pre">~/.bash_profile</span></tt> or <tt class="file docutils literal"><span class="pre">~/.cshrc</span></tt>:</p>
<div class="highlight-bash"><div class="highlight"><pre><span class="nb">export </span><span class="nv">PYTHONPATH</span><span class="o">=</span>~/lib/python
</pre></div>
</div>
<p>Once we've done this, our <tt class="file docutils literal"><span class="pre">master.cfg</span></tt> can look like:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="kn">from</span> <span class="nn">framboozle</span> <span class="kn">import</span> <span class="n">Framboozle</span>
<span class="n">f</span> <span class="o">=</span> <span class="n">BuildFactory</span><span class="p">()</span>
<span class="n">f</span><span class="o">.</span><span class="n">addStep</span><span class="p">(</span><span class="n">SVN</span><span class="p">(</span><span class="n">svnurl</span><span class="o">=</span><span class="s">"stuff"</span><span class="p">))</span>
<span class="n">f</span><span class="o">.</span><span class="n">addStep</span><span class="p">(</span><span class="n">Framboozle</span><span class="p">())</span>
</pre></div>
</div>
<p>or:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="kn">import</span> <span class="nn">framboozle</span>
<span class="n">f</span> <span class="o">=</span> <span class="n">BuildFactory</span><span class="p">()</span>
<span class="n">f</span><span class="o">.</span><span class="n">addStep</span><span class="p">(</span><span class="n">SVN</span><span class="p">(</span><span class="n">svnurl</span><span class="o">=</span><span class="s">"stuff"</span><span class="p">))</span>
<span class="n">f</span><span class="o">.</span><span class="n">addStep</span><span class="p">(</span><span class="n">framboozle</span><span class="o">.</span><span class="n">Framboozle</span><span class="p">())</span>
</pre></div>
</div>
<p>(check out the Python docs for details about how "import" and "from A
import B" work).</p>
<p>What we've done here is to tell Python that every time it handles an
"import" statement for some named module, it should look in our
<tt class="file docutils literal"><span class="pre">~/lib/python/</span></tt> for that module before it looks anywhere else. After our
directories, it will try in a bunch of standard directories too
(including the one where buildbot is installed). By setting the
<span class="target" id="index-6"></span><tt class="xref std std-envvar docutils literal"><span class="pre">PYTHONPATH</span></tt> environment variable, you can add directories to the front
of this search list.</p>
<p>Python knows that once it "import"s a file, it doesn't need to
re-import it again. This means that reconfiguring the buildmaster
(with <tt class="docutils literal"><span class="pre">buildbot</span> <span class="pre">reconfig</span></tt>, for example) won't make it think the
Framboozle class has changed every time, so the Builders that use it
will not be spuriously restarted. On the other hand, you either have
to start your buildmaster in a slightly weird way, or you have to
modify your environment to set the <span class="target" id="index-7"></span><tt class="xref std std-envvar docutils literal"><span class="pre">PYTHONPATH</span></tt> variable.</p>
<p>Option 3: Install this code into a standard Python library directory</p>
<p>Find out what your Python's standard include path is by asking it:</p>
<div class="highlight-none"><div class="highlight"><pre>80:warner@luther% python
Python 2.4.4c0 (#2, Oct 2 2006, 00:57:46)
[GCC 4.1.2 20060928 (prerelease) (Debian 4.1.1-15)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> import pprint
>>> pprint.pprint(sys.path)
['',
'/usr/lib/python24.zip',
'/usr/lib/python2.4',
'/usr/lib/python2.4/plat-linux2',
'/usr/lib/python2.4/lib-tk',
'/usr/lib/python2.4/lib-dynload',
'/usr/local/lib/python2.4/site-packages',
'/usr/lib/python2.4/site-packages',
'/usr/lib/python2.4/site-packages/Numeric',
'/var/lib/python-support/python2.4',
'/usr/lib/site-python']
</pre></div>
</div>
<p>In this case, putting the code into
/usr/local/lib/python2.4/site-packages/framboozle.py would work just
fine. We can use the same <tt class="file docutils literal"><span class="pre">master.cfg</span></tt> <tt class="docutils literal"><span class="pre">import</span> <span class="pre">framboozle</span></tt> statement as
in Option 2. By putting it in a standard include directory (instead of
the decidedly non-standard <tt class="file docutils literal"><span class="pre">~/lib/python</span></tt>), we don't even have to set
<span class="target" id="index-8"></span><tt class="xref std std-envvar docutils literal"><span class="pre">PYTHONPATH</span></tt> to anything special. The downside is that you probably have
to be root to write to one of those standard include directories.</p>
<p>Option 4: Submit the code for inclusion in the Buildbot distribution</p>
<p>Make a fork of buildbot on <a class="reference external" href="http://github.com/djmitche/buildbot">http://github.com/djmitche/buildbot</a> or post a patch
in a bug at <a class="reference external" href="http://buildbot.net">http://buildbot.net</a>. In either case, post a note about your patch
to the mailing list, so others can provide feedback and, eventually, commit it.</p>
<blockquote>
<div>from buildbot.steps import framboozle
f = BuildFactory()
f.addStep(SVN(svnurl="stuff"))
f.addStep(framboozle.Framboozle())</div></blockquote>
<p>And then you don't even have to install framboozle.py anywhere on your system,
since it will ship with Buildbot. You don't have to be root, you don't have to
set <span class="target" id="index-9"></span><tt class="xref std std-envvar docutils literal"><span class="pre">PYTHONPATH</span></tt>. But you do have to make a good case for Framboozle
being worth going into the main distribution, you'll probably have to provide
docs and some unit test cases, you'll need to figure out what kind of beer the
author likes (IPA's and Stouts for Dustin), and then you'll have to wait until
the next release. But in some environments, all this is easier than getting
root on your buildmaster box, so the tradeoffs may actually be worth it.</p>
<p>Putting the code in master.cfg (1) makes it available to that
buildmaster instance. Putting it in a file in a personal library
directory (2) makes it available for any buildmasters you might be
running. Putting it in a file in a system-wide shared library
directory (3) makes it available for any buildmasters that anyone on
that system might be running. Getting it into the buildbot's upstream
repository (4) makes it available for any buildmasters that anyone in
the world might be running. It's all a matter of how widely you want
to deploy that new class.</p>
</div>
</div>
<div class="section" id="writing-new-status-plugins">
<h2>Writing New Status Plugins<a class="headerlink" href="#writing-new-status-plugins" title="Permalink to this headline">¶</a></h2>
<p>Each status plugin is an object which provides the
<tt class="xref py py-class docutils literal"><span class="pre">twisted.application.service.IService</span></tt> interface, which creates a
tree of Services with the buildmaster at the top [not strictly true].
The status plugins are all children of an object which implements
<tt class="xref py py-class docutils literal"><span class="pre">buildbot.interfaces.IStatus</span></tt>, the main status object. From this
object, the plugin can retrieve anything it wants about current and
past builds. It can also subscribe to hear about new and upcoming
builds.</p>
<p>Status plugins which only react to human queries (like the Waterfall
display) never need to subscribe to anything: they are idle until
someone asks a question, then wake up and extract the information they
need to answer it, then they go back to sleep. Plugins which need to
act spontaneously when builds complete (like the <tt class="xref py py-class docutils literal"><span class="pre">MailNotifier</span></tt> plugin)
need to subscribe to hear about new builds.</p>
<p>If the status plugin needs to run network services (like the HTTP
server used by the Waterfall plugin), they can be attached as Service
children of the plugin itself, using the <tt class="xref py py-class docutils literal"><span class="pre">IServiceCollection</span></tt>
interface.</p>
<table class="docutils footnote" frame="void" id="framboozle-reg" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id10">[1]</a></td><td>framboozle.com is still available. Remember, I get 10% :).</td></tr>
</tbody>
</table>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="sidebar">
<h3>Table Of Contents</h3>
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../tutorial/index.html">Buildbot Tutorial</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="index.html">Buildbot Manual</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="introduction.html">Introduction</a></li>
<li class="toctree-l2"><a class="reference internal" href="installation.html">Installation</a></li>
<li class="toctree-l2"><a class="reference internal" href="concepts.html">Concepts</a></li>
<li class="toctree-l2"><a class="reference internal" href="configuration.html">Configuration</a></li>
<li class="toctree-l2 current"><a class="current reference internal" href="">Customization</a><ul class="simple">
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="cmdline.html">Command-line Tool</a></li>
<li class="toctree-l2"><a class="reference internal" href="resources.html">Resources</a></li>
<li class="toctree-l2"><a class="reference internal" href="optimization.html">Optimization</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../developer/index.html">Buildbot Development</a></li>
<li class="toctree-l1"><a class="reference internal" href="../relnotes/index.html">Release Notes for Buildbot v0.8.8</a></li>
</ul>
<h3 style="margin-top: 1.5em;">Search</h3>
<form class="search" action="../search.html" method="get">
<input type="text" name="q" />
<input type="submit" value="Go" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
<p class="searchtip" style="font-size: 90%">
Enter search terms or a module, class or function name.
</p>
</div>
<div class="clearer"></div>
</div>
</div>
<div class="footer-wrapper">
<div class="footer">
<div class="left">
<a href="cfg-statustargets.html" title="Status Targets"
>previous</a> |
<a href="cmdline.html" title="Command-line Tool"
>next</a> |
<a href="../py-modindex.html" title="Python Module Index"
>modules</a> |
<a href="../genindex.html" title="General Index"
>index</a>
<br/>
<a href="../_sources/manual/customization.txt"
rel="nofollow">Show Source</a>
</div>
<div class="right">
<div class="footer">
© Copyright Buildbot Team Members.
Created using <a href="http://sphinx.pocoo.org/">Sphinx</a> 1.1.3.
</div>
</div>
<div class="clearer"></div>
</div>
</div>
</body>
</html>
