Fedora Extras Build System
System Requirements:
- Python 2.3 or 2.4
- pyOpenSSL
- sqlite python bindings
- mock (http://fedoraproject.org/wiki/Projects/Mock/)
- createrepo
Getting Started
------------------------------------------
To allow users to retrieve logs and other status, you need to run an HTTP
server that allows access to the result dir (the 'server_work_dir' config
option).
You will also need to set up the infrastructure for the yum repository that the
builders connect to and retrieve the latest packages. This can be either HTTP,
NFS, SMB, etc. You then need to point 'yum' to this repo in step (5) of the
Builder Setup.
Builder Setup:
1) In the CVS checkout directory on the builder system, execute:
make DESTDIR=/ install
2) Create a user for the builder. The builder drops root privileges before
running the build, but must have root to be able to do initial setup
3) Copy the client Key, Cert, and CA Cert to the /etc/plague/builder/certs
directory
4) Things to modify in the builder's CONFIG.py:
- Modify the 'distro' and 'repo' options to match the targets you've
configured in /etc/mock/. These targets are usually in the form of
"distro-target-arch-repo". 'arch' and 'target' are passed by the build
system dynamically, but 'distro' and 'repo' are hardcoded in the config
file. Examples are "fedora-development-i386-core" and
"fedora-development-i386-extras".
5) Configure the mock target files in /etc/mock. You only need one target
file for each major arch you support. For example, you don't need separate
'ia32e' or 'amd64' config files, since these just use the normal 'x86_64'
config file
6) Start the client. ex: "/usr/bin/plague-builder 127.0.0.1 i386 i686"
On the Server:
1) Follow the instructions at the bottom of this file titled "Configuring SSL
for your Build System"
2) In the CVS checkout directory, execute:
make DESTDIR=/ install
3) Copy the server Key, Cert, and CA Cert to the /etc/plague/server/certs
directory
4) Copy the client authentication CA Cert to the /etc/plague/server/certs
directory
5) Things to modify in the server's CONFIG.py:
- Update the Key, Cert, and CA Cert, and client auth CA Cert file options
to point to the files in steps 3 and 4
- Modify the 'targets' option to add/remove the arches and targets you'll
be building
- Modify the 'builders' option to point to the build clients you'll be
using. Note the "https".
- If you want to do simple SRPM builds, set the 'use_srpm_not_cvs' option
to true
6) Start the server. ex: "/usr/bin/plague-server 127.0.0.1"
Operation:
1) You must add a user account for any user who wishes to use the build system.
This is accomplished with the 'plague-user-manager.py' tool, installed by
default in /usr/bin. You add a user like this:
/usr/bin/plague-user-manager.py /etc/plague/server/userdb add \
dcbw@redhat.com own_jobs kill_any_job modify_users server_admin
2) Clients then run plague-client to queue jobs. When first run,
plague-client creates the ~/.plague-client.cfg file
- Point the client to the server's address
- Point the client to the correct certificates
- Make sure you change the email address in ~/.plague-client to match
that of the 'user-cert' certificate
3) To build a package, you use plague-client like so:
/usr/bin/plague-client build ethtool /home/dcbw/ethtool-1.8-4.src.rpm devel
4) If the client returns "Package ethtool enqueued." then the enqueue was
successful
You can list your own jobs with:
/usr/bin/plague-client list
Builders can be listed & updated as well, see plague-client's usage message.
Architectural Overview:
------------------------------------------
The build system is composed of a single build server, and multiple builders.
Builders run an XMLRPC server to which the build server delivers build jobs.
The build server runs an XMLRPC server to allow submission of jobs, and to
relay basic status information about both builders and the build system as a
whole to users.
The Builder:
usage: /usr/bin/plague-builder -c <config_file>
Currently, builders are limited to building one job at a time, though there is
no restriction on running multiple builders on a single machine. They do not
queue pending jobs, but will reject build requests when something is already
building. The build server is expected to queue and manage jobs at this time,
and serialize requests to builders.
main()
`- Creates: XMLRPCBuilderServer
`- Creates: i386Arch, x86_64Arch, PPCArch, etc (subclasses
of BuilderMock)
The builder creates an XMLRPC server object (XMLRPCBuilderServer), and
then processes requests in an infinite loop. The build server queries each
client periodically for its status and the ID of the currently building job, if
any.
Each build job (BuilderMock and its architecture-specific subclasses like
i386Arch) proceeds through a number of states. Build jobs are periodically
given time to do work (BuilderMock.process()) by the builder controller
(XMLRPCBuilderServer._process()), which is in turn periodically given
time by the client's main loop. During their processing time, build jobs
check their state, see if any actions have completed, and advance to the next
state if needed. Communication with mock and retrieval of status from mock
are done with popen2.Popen4() so that mock does not block the XMLRPC server
from talking to the build server.
All communication with the build server is done through SSL to ensure the
identity of each party. Both the XMLRPC server and the builder's file server
are SSL-enabled, and require SSL certificates and keys to operate. See later
section in this document on how to configure SSL certificates for your build
system.
The Build Server:
usage: /usr/bin/plague-server
The build server runs two threads. The first, the XMLRPC server
(XMLRPCBuildMaster class), accepts requests to enqueue jobs for build and
stuffs them into an sqlite database which contains all job details. The second
thread, the Build Master (BuildMaster class), pulls 'waiting' jobs from the
database and builds them. A third top-level object that runs in the same
thread as the Build Master is the BuilderManager, which keeps track of
builders and their status.
main()
|- Creates: AuthedXMLRPCServer (system users talk to this object)
|- Creates: BuilderManager
|- `- Creates: Builder (one for each remote build client)
|- `- Creates: ArchJob (one for each build job
|- on each arch)
`- Creates: BuildMaster
`- Creates: PackageJob (one for each build job)
The BuilderManager object serves as a central location for all tracking and
status information about each build job on each arch. It creates a
Builder instance for each remote builder. The Builder instance keeps track
of specific jobs building on all architectures on that remote builder.
PackageJobs must queue requests with the BuilderManager for the architecture
specific build jobs they need. The BuilderManager will wait until a Builder
object is available to build the job, create the new ArchJob on the builder,
and notify the parent PackageJob of its new ArchJob.
The BuilderManager has a periodic processing routine that is called from the
BuildMaster thread. This processing routine calls the Builder.process()
routine of each Builder instance, which in turn updates its view of the remote
builder's status. Thus, the BuilderManager, through each Builder instance,
knows the status and currently building job on each remote builder.
PackageJobs track a single SRPM build through the entire build system. They are
created from the BuildMaster thread whenever the BuildMaster finds a job entry
in the sqlite database with the status of 'waiting'. PackageJobs proceed through
a number of states: "initialize", "checkout", "make_srpm", "prep", "building",
"finished", "addtorepo", "failed", "killed", and "needsign".
Flow goes like this:
initialize => checkout
checkout => make_srpm
make_srpm => prep
prep => (queue architecture-specific job requests with the BuilderManager)
<wait for first ArchJob to appear>
building
- All build jobs finished or failed? => finished
- otherwise => building
finished
- failed jobs? => failed
- otherwise => addtorepo
addtorepo
<wait for Repo object to add our packages to the repo>
=> repodone
repodone => needsign
All communication with builders is done through SSL to ensure the identity of
each party. When the builder requests the SRPM to build, SSL is used. When the
build server retrieves logs and RPMs from the builder, SSL is also used. This
ensures that builders can be more or less trusted, or at least that some random
builder is not serving you packages that might contaminate your repository. See
later section in this document on how to configure SSL certificates for your
build system.
Configuring SSL for your Build System
--------------------------------------
When you set up the build system, you essentially become a Certificate
Authority. Because the build server and the build clients communicate using
SSL, they need to exchange certificates to verify the others' identity.
You must first createa key/cert pair for the Build System Certificate
Authority, which signs both the build server's certificate, and each build
client's certificate.
The Certificates on the Server:
config_opts['server_key_and_cert'] -> server SSL certificate and private key
config_opts['ca_cert'] -> CA certificate used to sign both server and builder
certificates
config_opts['ui_ca_cert'] -> CA cert that signs package maintainer's
certificates, used to verify connections from
plague-clients are authorized
The Certificates on the Builders:
config_opts['builder_key_and_cert'] -> builder SSL certificate and private key
config_opts['ca_cert'] -> _same_ as server's 'ca_cert', the CA certificate
used to sign both server and builder certificates
Setting up the Certificates:
A tool called "certhelper.py" is included in the utils/ directory, or
possibly as /usr/bin/plague-certhelper if installed from a package. Here we
use certhelper.py for either program.
1. Create the Build System Certificate Authority key and certificate:
certhelper.py ca --outdir=/etc/plague/ca_dir --name=buildsystem
After entering the certificate's details, if there are no errors, you end up
with a self-signed certificate in /etc/plague/ca_dir/buildsystem_ca_cert.pem,
and a CA private key in /etc/plague/ca_dir/private/buildsystem_ca_key.pem. You
will need buildsystem_ca_cert.pem later.
2. Create a certificate and key for the build server:
certhelper.py normal --outdir=/etc/plague/server/certs --name=server \
--cadir=/etc/plague/ca_dir --caname=buildsystem
After entering details for the server's certificate, you end up with a combined
certificate and private key file in /etc/plague/server/certs/,
server_key_and_cert.pem. Copy the buildsystem_ca_cert.pem file from step 1 into
/etc/plague/server/certs as well.
Update the server's config file in /etc/plague/server/CONFIG.py to match the
absolute path to server_key_and_cert.pem (the 'server_key_and_cert' config
option) and also the absolute path to buildsystem_ca_cert.pem (the 'ca_cert'
config option). Both should be based in /etc/plague/server if you follow the
directions here.
IMPORTANT: make sure only the build server's user (normally root) can read
server_key_and_cert.pem, since it contains the server's private key.
3. For each builder you plan to deploy, you will need to generate a certificate
for that builder as well.
certhelper.py normal --outdir=/etc/plague/builder/certs --name=builder1 \
--cadir=/etc/plague/ca_dir --caname=buildsystem
After entering details for the builder's certificate, you end up with a single
file, /etc/plague/builder/certs/builder1_key_and_cert.pem. By default, the
plague-builder is set up to look for the file "<hostname>.pem" where <hostname>
is the host name of the builder machine. You will need to rename the
builder1_key_and_cert.pem file to the builder's hostname with a .pem extension.
If the builder is on a separate machine, you will need to copy the certificate
file to that machine along with the CA's certificate, buildsystem_ca_cert.pem,
and the server's plain certificate, server_cert.pem.
Update the builder's config file (normally in /etc/plague/builder/CONFIG.py) to
match the absolute paths to the builder's key_and_cert file, and to the
build system CA certificate, buildsystem_ca_cert.pem.
IMPORTANT: make sure only root can read builder1_key_and_cert.pem, since it
contains the buidler's private key.
------------------
Package Maintainer Certificates:
Config options from ~/.plague-client.cfg, used by /usr/bin/plague-client:
server-ca-cert -> _same_ as build server and builder's 'ca_cert'
user-ca-cert -> CA cert that signed the package maintainer's 'user-cert'
user-key -> package maintainer's private key, can be blank if private key and
certificate are in the same file
user-cert -> package maintainer's certificate, signed by 'user-ca-cert' and
sent to build server to validate the plague-client's connection
To allow package maintainers to connect and queue up packages, it is best to use
a completely separate Certificate Authority to sign user certificates. If you
use the same CA as you use for the build server and builders, any maintainer
could set up a rogue builder using his/her certificate. To prevent this, create
a new certificate authority for package maintainer certificates.
1. The procedure closely follows the steps for the build system certificate
setup. Create the CA in the same way (using a different --outdir and --name of
course). The *_ca_cert.pem file that certhelper.py spits out should be added to
the build server's CONFIG.py file for the 'ui_ca_cert' config option. It should
also be distributed to package maintainers, who enter this certificate in their
~/.plague-client.cfg file as the 'user-ca-cert' config option.
certhelper.py ca --outdir=/etc/plague/user_ca_dir --name=users
2. To create user certificates, do the same steps as for creating builder
certificates above, except MAKE SURE to enter that user's email address at the
certificate information entry's "Email Address []:" prompt. This is essential
for user validation and notification of build status.
certhelper.py normal --outdir=/tmp/user_certs --name=user1 \
--cadir=/etc/plague/user_ca_dir --caname=users
Here, certhelper.py will produce a file called user1_key_and_cert.pem in
/tmp/user_certs. Send this file, along with users_ca_cert.pem and
buildsystem_ca_cert.pem, to the package maintainer whose email address you
entered for this certificate. The package maintainer then sets up his/her
~/.plague-client.cfg to point to the correct files:
'user-ca-cert' => users_ca_cert.pem
'server-ca-cert' => buildsystem_ca_cert.pem
'user-cert' => user1_key_and_cert.pem
'user-key' => should be blank
