2003-03-01 pbuilder-doc Usage and operations documentation in progress Junichi Uekawa pbuilder stands for Personal Builder, and it is a automatic Debian Package Building system for personal environments. pbuilder aims to be an easy-to-setup system for auto-building Debian packages inside a clean-room environment, so that it is possible to verify that a package can be built on most Debian installations. Clean-room environment is achieved through use of a chroot image, so that only minimal packages will be installed inside the chroot. Debian distribution consists of free software accompanied with source. The source code within Debian "main" section must build within Debian "main", with only the explicitly specified build-dependencies installed. The aim of pbuilder is very different from other auto-building systems in that its aim is not in trying to build as many packages. It does not try to guess what a package needs, and in most cases it tries the worst choice of all if there was a choice to be made. In this way, pbuilder tries to ensure that packages tested against pbuilder will build in most Debian environments, hopefully resulting in a good overall Debian source-buildability. The goal of making Debian buildable from source is somewhat achieved, and has progressed well. It is known that Debian 3.0 is not quite buildable from source, but the next version should be better. pbuilder create will create base chroot image. Distribution code-name needs to be specified with --distribution command-line option. Usually, "sid" is the proper distribution. debootstrap is used to create the bare minimum Debian installation, and then build-essential packages are installed on top of the minimum installation using apt-get inside the chroot. For fuller documentation of command-line options, see pbuilder.8 manual page. Some configuration will be required for /etc/pbuilderrc for the mirror site The mirror site should preferably a local one, to not to overload the public mirrors with a lot of access. to use, and proxy configuration is probably required to allow access through HTTP. See pbuilderrc.5 manual page for details. pbuilder update will update the chroot image. It will extract the chroot, invoke apt-get update and apt-get dist-upgrade inside the chroot, and then recreate the base tarball. It is possible to switch the distribution which the chroot tarball is targeted at at this point. Specify --distribution sid --override-config to change the distribution to sid. However, only upgrading is really supported. For fuller documentation of command-line options, see pbuilder.8 manual page To build a package inside the chroot, invoke pbuilder build whatever.dsc. pbuilder will extract chroot image to a temporal working directory, and satisfy the build-dependency inside the chroot, and build the package. The built packages will be moved to a directory specified with --buildresult command-line option. --basetgz option can be used to specify which chroot image to use. pbuilder will extract a fresh chroot image created with pbuilder build and updated with pbuilder update, and populate the chroot with build-dependency by parsing debian/control and invoking apt-get. For fuller documentation of command-line options, see pbuilder.8 manual page pdebuild is a little wrapper script that does the most frequent of all tasks. A Debian Developer may try to do debuild, and build a package, inside a Debian source directory. pdebuild will allow similar control, and allow package to be built inside the chroot, to check that the current source tree will build happily inside the chroot. pdebuild calls dpkg-source to build the source packages, and then invoke pbuilder on the resulting source package. However, unlike debuild, the resulting deb files will be found somewhere in BUILDRESULT directory. See pdebuild.1 manual page for more details. It is possible to specify all settings by command-line option. However, for convenience it is possible to use a configuration file. /etc/pbuilderrc and ${HOME}/.pbuilderrc are read in when pbuilder is invoked. The possible options are documented in pbuilderrc.5 documentation. pbuilder requires full root privilage when it is satisfying the build-dependency but most packages do not need root privilage, or even do not build when they are root. pbuilder can create a user only used inside pbuilder and use that user id when building, and use fakeroot command when root privilage is required. BUILDUSERID needs should be set to a value for a user id that does not exist on the system, so that it is harder for packages that are being built with pbuilder to do harm to the main system. BUILDUSERNAME needs to be set to some value, and pbuilder will use that user id and use fakeroot for building. Using the fakerooting method, pbuilder will run with root privilage when it is required, when installing packages to the chroot. To be able to invoke pbuilder without being root, you need to use user-mode-linux, as explained in . pbuilder can be used for backporting software from the latest Debian distribution to older stable distribution, by using a chroot that contains image of older distribution, and building packages inside the chroot. There are several points to consider, and due to the following reasons, automatic backporting is usually not possible, and manual interaction is required: Build-Dependency in stable may not be enough to build a package in unstable distribution, so package may need more than what exists in stable Stable distribution may have bugs that have been fixed in unstable that needs to be worked around. Package in unstable distribution may have problem building even for unstable. pbuilder can be automated, because its operations are non-interactive. It is possible to run pbuilder through multiple packages noninteractively. There are several such scripts known to exist. Junichi Uekawa has been running such script since 2001, and has been filing bugs on packages that fail the test of pbuilder. There were several problems with autobuilding: Build-Dependency needs to install noninteractively, but some packages are so broken that they cannot install without interaction (like postgresql) When a library package breaks, or gcc/gcj/g++ breaks, or even bison, a large number of build failure are reported. (gcj-3.0 which had no "javac", bison which got more strict, etc.) Some people were quite hostile against build failure reports. But most of these problems are now getting solved. Only about 10% of Debian now fail to build from source (29 Dec 2002). A script that was used by Junichi Uekawa is now included in pbuilder distribution, as pbuildd.sh. It is available in /usr/share/doc/pbuilder/examples/pbuildd/ and its configuration is in /etc/pbuilder/pbuildd-config.sh. It should be easy enough to set up for people who are used to pbuilder. It has been running for quite a while, and it should be possible to set the application up on your system also. However, it is a new introduction, and please file bugs to the Debian BTS if you know of possible problems, or improved on the script considerably. To set up pbuildd, there are some points to be aware of. A file ./avoidlist needs to be available with the list of packages to avoid building. It will try building anything, even packages that are not aimed for your architecture Because you are running random build scripts, it is better to use fakeroot option of pbuilder, to avoid running build in root privilege Because not all builds are guaranteed to finish in a finite time, setting timeout is probably necessary, or build may stall with a bad build Some packages require a lot of disk space, around 2GB seems to be quite sufficient for the time being. There are some people who use pbuilder to automatically backport a subset of packages to the stable distribution. Any URL? pbuilder can be used for automated testing of packages. It has the feature of allowing hooks to be placed, and these hooks can try to install packages inside the chroot, or run them, or whatever else that can be done. Some known tests and ideas: Automatic install-remove-upgrade-remove-install-purge-upgrade-purge testsuite (distributed as an example) Automatic lintian running (distributed as an example) Automatic debian-test of the package? Most packages are compiled with gcc or g++ and using the default compiler version, which was gcc 2.95 for Debian GNU/Linux 3.0 (i386). However, Debian distributes other compilers, also, and provides binaries such as gcc-3.2 for gcc compiler version 3.2. It is therefore possible to try compiling packages against different compiler versions. pentium-builder provides an infrastructure for using different compiler for building packages than the default gcc, by becoming a wrapper script called gcc, that calls the real gcc. To use pentium-builder in pbuilder, it is possible to set up the following in the configuration: EXTRAPACKAGES="pentium-builder gcc-3.2 g++-3.2" export DEBIAN_BUILDARCH=athlon export DEBIAN_BUILDGCCVER=3.2 It will instruct pbuilder to install pentium-builder package and also the GCC 3.2 compiler packages inside the chroot, and set the environment variables required for pentium-builder to function. Here, known problems and frequently asked questions are documented. This portion was initially available in README.Debian file, but moved into here. It often happens that pbuilder cannot create latest chroot. Try upgrading pbuilder and debootstrap. It is currently only possible to create software that handles the past. Future prediction features may be added later when we can successfully handle the past. If you are setting $TMPDIR to an unusual value, of other than /tmp, you will find that some errors may occur inside the chroot, such as dpkg-source failing. There are two options, you may install a hook to create that directory, or set export TMPDIR=/tmp in pbuilderrc. Take your pick. If you have some very specialized requirements on your apt setup inside pbuilder, it is possible to specify that through --othermirror option. Try something like: --othermirror "deb http://local/mirror stable main|deb-src http://local/source/repository ./" You can use hook scripts for this. D scripts are run before satisfying build-dependency. To make distinguishing bash prompts inside pbuilder easier, it is possible to set environmental variables such as PS1 inside pbuilderrc export PS1="pbuild chroot 32165 # " There are cases when some small experimenting is required, and do not want to damage the main system, like installing experimental library packages, or compiling with experimental compilers. For such cases, pbuilder login command is available. pbuilder login is a debuggin feature for pbuilder itself, but it also allows users to have a temporal chroot. Note that chroot is cleaned after logging out of the shell, and mounting filesystems inside it is considered harmful. To facilitate using pbuilder for other uses, pbuilder execute is available. pbuilder execute will take a script specified in the command-line argument, and invoke the script inside the chroot. The script can be useful for sequence of operations such as installing ssh and adding a new user inside the chroot. There are some advanced features, above that of the basic feature of pbuilder, for some specific purposes. pbuilder-uml exists. Invoking that command instead of pbuilder it is possible to use user-mode-linux. The advantage of using user-mode-linux is that it does not require root privilege to run, and it does Copy-on-write, which is probably much faster than conventional pbuilder method. The problem is that this relies on User-mode-linux which is a relatively new project, and has not quite matured, as opposed to conventional pbuilder which rely on chroot and tar and gzip, which are known to work on most Unix systems. However, pbuilder-uml uses COW method for file access, and it is so much more faster than pbuilder when building most packages. It has been verified that pbuilder-uml works, as of version 0.59. The configuration of pbuilder-uml goes in two steps: Configuration of rootstrap Configuration of pbuilder-uml rootstrap is a program that is a wrapper to debootstrap, creating a Debian disk image inside UML. To configure rootstrap, there are several requirements. install rootstrap package add the user to uml-net group to allow access to network adduser dancer uml-net Check that compile supports tun/tap interface, and recompile the kernel if necessary Set up /etc/rootstrap/rootstrap.conf, for example, if the current host is 192.168.1.2, changing following entries to something like this seems to work. transport=tuntap interface=eth0 gateway=192.168.1.1 mirror=http://192.168.1.2:8081/debian host=192.168.1.198 uml=192.168.1.199 netmask=255.255.255.0 Some experimentation with configuration and running rootstrap ~/test.uml to actually test it would be handy. The following needs to happen: install pbuilder-uml package Set configuration file /etc/pbuilder/pbuilder-uml.conf as follows MY_ETH0=tuntap,,,192.168.1.198 UML_IP=192.168.1.199 UML_NETMASK=255.255.255.0 UML_NETWORK=192.168.1.0 UML_BROADCAST=255.255.255.255 UML_GATEWAY=192.168.1.1 PBUILDER_UML_IMAGE="/home/dancer/uml-image" and it needs to match rootstrap configuration. Run pbuilder-user-mode-linux create --distribution sid to create the image Try running pbuilder-user-mode-linux build To run pbuilder-uml parallel on a system, there are a few things to bear in mind. create and update methods must not be ran when build is in progress, or COW file will be invalid UML processes that are running in parallel needs to have different IP addresses. So, something like the following will work, for IP in 102 103 104 105; do xterm -e pbuilder-user-mode-linux build --uml-ip 192.168.0.$IP 20030107/whizzytex_1.1.1-1.dsc& done but just trying to run pbuilder-uml several times will result in failure to access the network. It is possible to use pbuilder-uml for other uses than just building Debian packages. pbuilder-user-mode-linux login will let a user use a shell inside the user-mode-linux using the pbuilder base image, and pbuilder-user-mode-linux execute will allow the user to execute a script inside the chroot. You can use the script to install ssh and add a new user, so that it is possible to access inside the UML through ssh. Note that it is not possible to use a script from /tmp due to the way pbuilder-uml mounts tmpfs at /tmp. The following is an example script that may be useful in starting a sshd inside uml. #!/bin/bash apt-get install -y ssh xbase-clients xterm echo "enter root password" passwd cp /etc/ssh/sshd_config{,-} cat /etc/ssh/sshd_config- | sed 's/X11Forwarding.*/X11Forwarding yes/' > /etc/ssh/sshd_config /etc/init.d/ssh restart ifconfig echo "Hit enter to finish" read LVM has snapshot function that features Copy-on-write images. That could be used for pbuilder just it can be used for user-mode-linux pbuilder port. It may prove to be faster, but it is not implemented yet, and so no measurement has been made, yet. --no-targz option of pbuilder will allow usage of pbuilder in a different way to conventional usage. It will try to use existing chroot, and will not try to clean up after working on it. It is an operation mode more like sbuild. It should be possible to create chroot images for dchroot with following commands: # pbuilder create --distribution potato --no-targz --basetgz /chroot/potato # pbuilder create --distribution woody --no-targz --basetgz /chroot/woody # pbuilder create --distribution sid --no-targz --basetgz /chroot/sid This document is started on 28 Dec 2002 by Junichi Uekawa, trying to document what is known about pbuilder. This documentation is available from pbuilder source tarball, and from CVS repository of pbuilder (which might not be public yet now). A copy of this documentation can be found in Netfort page for pbuilder. The homepage for pbuilder is http://www.netfort.gr.jp/~dancer/software/pbuilder.html The following is most possibly inaccurate account of how pbuilder happened to come, and other attempts to make something like pbuilder to happen. This part of document was originally in AUTHORS file, to give credit to those who existed before pbuilder. There were dbuild, which was a shell script to build Debian packages from source. Lars Wirzenius wrote that script, and it was good, short, and simple (probably). There was nothing like build-depends then (I think), and it was simple. It could have been improved, I don't have the source off-hand. debbuild was probably written by James Troup. I don't know it because I have never seen the actual code, I could only find some references to it on the net, and mailing list logs. sbuild is a perl script to build Debian package from source. It parses Build-Dependency, and performs other misc checks, and has a lot of hacks to actually get things building, including a table of what package to use when virtual packages are specified (does it do that still?). It supports use of local database for packages which do not have build-dependency. It was written by Ronan Hodek, and I think it was patched and fixed and extended by several people. It is part of wanna-build, and used extensively in Debian buildd system. I think it was maintained mostly by Ryan Murray. wanna-build (sbuild) was quite difficult to set up, and it was never a Debian package. dbuild was something that predated Build-Depends. Building package from source using Build-Depends information within a chroot sounded trivial; and pbuilder was born. It was initially a shell script with only a few lines, which called debootstrap and chroot and dpkg-buildpackage in the same run, but soon, it was decided that's too slow. Yes, and it took almost an year to get things somewhat right, and in the middle of the process, Debian 3.0 was released. Yay. Debian 3.0 wasn't completely buildable with pbuilder, but the amount of packages which are not buildable are steadily decreasing. (I hope) And someone wanted pbuilder to run as not root, and User-mode-linux has become more useful as time passed, I've started experimenting with pbuilder-uml. pbuilder-uml has not been able to run as often as it should, and bootstrapping user-mode-linux environment has been pretty hard.