This is buildbot.info, produced by makeinfo version 4.13 from buildbot.texinfo. This is the BuildBot manual. Copyright (C) 2005, 2006, 2009, 2010 Brian Warner Copying and distribution of this file, with or without modification, are permitted in any medium without royalty provided the copyright notice and this notice are preserved. File: buildbot.info, Node: Top, Next: Introduction, Prev: (dir), Up: (dir) BuildBot ******** This is the BuildBot manual. Copyright (C) 2005, 2006, 2009, 2010 Brian Warner Copying and distribution of this file, with or without modification, are permitted in any medium without royalty provided the copyright notice and this notice are preserved. * Menu: * Introduction:: What the BuildBot does. * Installation:: Creating a buildmaster and buildslaves, running them. * Concepts:: What goes on in the buildbot's little mind. * Configuration:: Controlling the buildbot. * Getting Source Code Changes:: Discovering when to run a build. * Build Process:: Controlling how each build is run. * Status Delivery:: Telling the world about the build's results. * Command-line tool:: * Resources:: Getting help. * Developer's Appendix:: * Index of Useful Classes:: * Index of master.cfg keys:: * Index:: Complete index. --- The Detailed Node Listing --- Introduction * History and Philosophy:: * System Architecture:: * Control Flow:: System Architecture * BuildSlave Connections:: * Buildmaster Architecture:: * Status Delivery Architecture:: Installation * Requirements:: * Installing the code:: * Creating a buildmaster:: * Upgrading an Existing Buildmaster:: * Creating a buildslave:: * Launching the daemons:: * Logfiles:: * Shutdown:: * Maintenance:: * Troubleshooting:: Creating a buildslave * Buildslave Options:: Troubleshooting * Starting the buildslave:: * Connecting to the buildmaster:: * Forcing Builds:: Concepts * Version Control Systems:: * Schedulers:: * BuildSet:: * BuildRequest:: * Builder:: * Users:: * Build Properties:: Version Control Systems * Generalizing VC Systems:: * Source Tree Specifications:: * How Different VC Systems Specify Sources:: * Attributes of Changes:: Users * Doing Things With Users:: * Email Addresses:: * IRC Nicknames:: * Live Status Clients:: Configuration * Config File Format:: * Loading the Config File:: * Testing the Config File:: * Defining the Project:: * Change Sources and Schedulers:: * Setting the slaveport:: * Buildslave Specifiers:: * On-Demand ("Latent") Buildslaves:: * Defining Global Properties:: * Defining Builders:: * Defining Status Targets:: * Debug options:: Change Sources and Schedulers * Scheduler Scheduler:: * AnyBranchScheduler:: * Dependent Scheduler:: * Periodic Scheduler:: * Nightly Scheduler:: * Try Schedulers:: * Triggerable Scheduler:: Buildslave Specifiers * When Buildslaves Go Missing:: On-Demand ("Latent") Buildslaves * Amazon Web Services Elastic Compute Cloud ("AWS EC2"):: * Dangers with Latent Buildslaves:: * Writing New Latent Buildslaves:: Getting Source Code Changes * Change Sources:: * Choosing ChangeSources:: * CVSToys - PBService:: * Mail-parsing ChangeSources:: * PBChangeSource:: * P4Source:: * BonsaiPoller:: * SVNPoller:: * MercurialHook:: * Bzr Hook:: * Bzr Poller:: Mail-parsing ChangeSources * Subscribing the Buildmaster:: * Using Maildirs:: * Parsing Email Change Messages:: Parsing Email Change Messages * FCMaildirSource:: * SyncmailMaildirSource:: * BonsaiMaildirSource:: * SVNCommitEmailMaildirSource:: Build Process * Build Steps:: * Interlocks:: * Build Factories:: Build Steps * Common Parameters:: * Using Build Properties:: * Source Checkout:: * ShellCommand:: * Simple ShellCommand Subclasses:: * Python BuildSteps:: * Transferring Files:: * Steps That Run on the Master:: * Triggering Schedulers:: * Writing New BuildSteps:: Source Checkout * CVS:: * SVN:: * Darcs:: * Mercurial:: * Arch:: * Bazaar:: * Bzr:: * P4:: * Git:: Simple ShellCommand Subclasses * Configure:: * Compile:: * Test:: * TreeSize:: * PerlModuleTest:: * Testing with mysql-test-run:: * SetProperty:: * SubunitShellCommand:: Python BuildSteps * BuildEPYDoc:: * PyFlakes:: * PyLint:: Writing New BuildSteps * BuildStep LogFiles:: * Reading Logfiles:: * Adding LogObservers:: * BuildStep URLs:: Build Factories * BuildStep Objects:: * BuildFactory:: * Process-Specific build factories:: BuildStep Objects * BuildFactory Attributes:: * Quick builds:: BuildFactory * BuildFactory Attributes:: * Quick builds:: Process-Specific build factories * GNUAutoconf:: * CPAN:: * Python distutils:: * Python/Twisted/trial projects:: Status Delivery * WebStatus:: * MailNotifier:: * IRC Bot:: * PBListener:: * Writing New Status Plugins:: WebStatus * WebStatus Configuration Parameters:: * Buildbot Web Resources:: * XMLRPC server:: * HTML Waterfall:: Command-line tool * Administrator Tools:: * Developer Tools:: * Other Tools:: * .buildbot config directory:: Developer Tools * statuslog:: * statusgui:: * try:: waiting for results * try --diff:: Other Tools * sendchange:: * debugclient:: File: buildbot.info, Node: Introduction, Next: Installation, Prev: Top, Up: Top 1 Introduction ************** The BuildBot is a system to automate the compile/test cycle required by most software projects to validate code changes. By automatically rebuilding and testing the tree each time something has changed, build problems are pinpointed quickly, before other developers are inconvenienced by the failure. The guilty developer can be identified and harassed without human intervention. By running the builds on a variety of platforms, developers who do not have the facilities to test their changes everywhere before checkin will at least know shortly afterwards whether they have broken the build or not. Warning counts, lint checks, image size, compile time, and other build parameters can be tracked over time, are more visible, and are therefore easier to improve. The overall goal is to reduce tree breakage and provide a platform to run tests or code-quality checks that are too annoying or pedantic for any human to waste their time with. Developers get immediate (and potentially public) feedback about their changes, encouraging them to be more careful about testing before checkin. Features: * run builds on a variety of slave platforms * arbitrary build process: handles projects using C, Python, whatever * minimal host requirements: python and Twisted * slaves can be behind a firewall if they can still do checkout * status delivery through web page, email, IRC, other protocols * track builds in progress, provide estimated completion time * flexible configuration by subclassing generic build process classes * debug tools to force a new build, submit fake Changes, query slave status * released under the GPL * Menu: * History and Philosophy:: * System Architecture:: * Control Flow:: File: buildbot.info, Node: History and Philosophy, Next: System Architecture, Prev: Introduction, Up: Introduction 1.1 History and Philosophy ========================== The Buildbot was inspired by a similar project built for a development team writing a cross-platform embedded system. The various components of the project were supposed to compile and run on several flavors of unix (linux, solaris, BSD), but individual developers had their own preferences and tended to stick to a single platform. From time to time, incompatibilities would sneak in (some unix platforms want to use `string.h', some prefer `strings.h'), and then the tree would compile for some developers but not others. The buildbot was written to automate the human process of walking into the office, updating a tree, compiling (and discovering the breakage), finding the developer at fault, and complaining to them about the problem they had introduced. With multiple platforms it was difficult for developers to do the right thing (compile their potential change on all platforms); the buildbot offered a way to help. Another problem was when programmers would change the behavior of a library without warning its users, or change internal aspects that other code was (unfortunately) depending upon. Adding unit tests to the codebase helps here: if an application's unit tests pass despite changes in the libraries it uses, you can have more confidence that the library changes haven't broken anything. Many developers complained that the unit tests were inconvenient or took too long to run: having the buildbot run them reduces the developer's workload to a minimum. In general, having more visibility into the project is always good, and automation makes it easier for developers to do the right thing. When everyone can see the status of the project, developers are encouraged to keep the tree in good working order. Unit tests that aren't run on a regular basis tend to suffer from bitrot just like code does: exercising them on a regular basis helps to keep them functioning and useful. The current version of the Buildbot is additionally targeted at distributed free-software projects, where resources and platforms are only available when provided by interested volunteers. The buildslaves are designed to require an absolute minimum of configuration, reducing the effort a potential volunteer needs to expend to be able to contribute a new test environment to the project. The goal is for anyone who wishes that a given project would run on their favorite platform should be able to offer that project a buildslave, running on that platform, where they can verify that their portability code works, and keeps working. File: buildbot.info, Node: System Architecture, Next: Control Flow, Prev: History and Philosophy, Up: Introduction 1.2 System Architecture ======================= The Buildbot consists of a single `buildmaster' and one or more `buildslaves', connected in a star topology. The buildmaster makes all decisions about what, when, and how to build. It sends commands to be run on the build slaves, which simply execute the commands and return the results. (certain steps involve more local decision making, where the overhead of sending a lot of commands back and forth would be inappropriate, but in general the buildmaster is responsible for everything). The buildmaster is usually fed `Changes' by some sort of version control system (*note Change Sources::), which may cause builds to be run. As the builds are performed, various status messages are produced, which are then sent to any registered Status Targets (*note Status Delivery::).