The ultimate Debian/Ubuntu package building system

Mar. 3, 2011

Here is a “small” article which crosses the border between personal stuff and work but may be useful for many people.

In my current job, making Debian and Ubuntu packages takes me a lot of time.

At the beginning, I used to work on 2 virtual machine with a small pbuilderrc file, but I had the opportunity to work on a more powerful server (4 cores and 4GB of RAM), and I thought it would be useful to merge both machines and finaly have only one building machine.

My building system runs on a Debian Sid, even if I was a long time Ubuntu user, now I use Debian (including on my workstation).

Why choosing Debian Sid ?. Only to be able to install the latest versions of build tools, especially for debootstrap.

It might also work on Debian stable, and Ubuntu (but some points are different)

So let’s begin.

All the commands preceded by a “#” need to be run with the root user, commands preceded by a “$” need to be run with a normal user, without privileges.

Notice : my server is running 64 bits OS, so I can compile 32 and 64 bits packages.

First, we need to install sudo, because as time goes by, I learned the hard way that it’s dangerous to work on packages with the user root because it’s not necessary and also, after some hours of keyboard typing and with tiredness by, we can act like asses and transforming into a “Rage Guy” for the next 5 minutes (or hours).

# aptitude install sudo

Then, we need to change the settings of sudo depending of what we will do with pbuilder.

# visudo

Here is my sudo configuration file :

This file MUST be edited with the ‘visudo’ command as root.

Please consider adding local content in /etc/sudoers.d/ instead of

directly modifying this file.

See the man page for details on how to write a sudoers file.

Defaults env_reset,env_keep=“DIST ARCH CONCURRENCY_LEVEL”

Host alias specification

User alias specification

Cmnd alias specification

User privilege specification

root ALL=(ALL:ALL) ALL

Allow members of group sudo to execute any command

%sudo ALL=(ALL:ALL) ALL

#includedir /etc/sudoers.d

Members of the admin group may gain root privileges

%adm ALL=(ALL:ALL) ALL

Note the “Defaults” line, which contains instructions for keeping the values of variables DIST, ARCH and CONCURRENCY_LEVEL. Keep those variable name in head, we will use them later.

Now, we will work on the real part of compilation, we will install the tools needed for compilation.

# aptitude install debhelper build-essential dpkg-dev pbuilder devscripts debootstrap

Among those packages, the most remarkable are “pbuilder” and “debootstrap”

pbuilder allows us to build our packages in a chrooted environment which contains the linux distribution of our choice.
debootstrap is called by pbuilder to create this chrooted environment

The lintian package is also useful, but optional, it tests the Debian Policy conformity of packages.

Now all these tools are installed, we need to configure the pbuilder, with the pbuilderrc file. Since I don’t like to edit the default configuration file (/etc/pbuilderrc), I choose to create the /root/.pbuilderrc file, which here is mine.

# Codenames for Debian suites according to their alias. Update these when

needed.

UNSTABLE_CODENAME=“sid” TESTING_CODENAME=“wheezy” STABLE_CODENAME=“squeeze” STABLE_BACKPORTS_SUITE="$STABLE_CODENAME-backports" OLD_STABLE_CODENAME=“lenny” OLD_STABLE_BACKPORTS_SUITE="$OLD_STABLE_CODENAME-backports"

List of Debian suites.

DEBIAN_SUITES=($UNSTABLE_CODENAME $TESTING_CODENAME $STABLE_CODENAME $OLD_STABLE_CODENAME “unstable” “testing” “stable” “experimental”)

List of Ubuntu suites. Update these when needed.

UBUNTU_SUITES=(“natty” “maverick” “lucid” “karmic” “jaunty” “intrepid” “hardy” “gutsy”)

Mirrors to use. Update these to your preferred mirror.

DEBIAN_MIRROR=“ftp.fr.debian.org/” UBUNTU_MIRROR=“fr.archive.ubuntu.com”

Use old-releases mirrors for EOL versions

if [ “${DIST}” == “gutsy” ]; then UBUNTU_MIRROR=“old-releases.ubuntu.com” fi if [ “${DIST}” == “intrepid” ]; then UBUNTU_MIRROR=“old-releases.ubuntu.com” fi

Optionally use the changelog of a package to determine the suite to use if

none set.

if [ -z “${DIST}” ] && [ -r “debian/changelog” ]; then DIST=$(dpkg-parsechangelog | awk ‘/^Distribution: / {print $2}’) # Use the unstable suite for Debian experimental packages. if [ “${DIST}” == “experimental” ]; then DIST=“unstable” fi fi

Optionally set a default distribution if none is used. Note that you can set

your own default (i.e. ${DIST:=“unstable”}).

: ${DIST:="$(lsb_release –short –codename)"}

Optionally set the architecture to the host architecture if none set. Note

that you can set your own default (i.e. ${ARCH:=“i386”}).

: ${ARCH:="$(dpkg –print-architecture)"}

NAME="$DIST" if [ -n “${ARCH}” ]; then NAME="$NAME-$ARCH" DEBOOTSTRAPOPTS=("–arch" “$ARCH” “${DEBOOTSTRAPOPTS[@]}”) fi BASETGZ="/var/cache/pbuilder/$NAME-base.tgz" DISTRIBUTION="$DIST" BUILDRESULT="/var/cache/pbuilder/$NAME/result/" APTCACHE="/var/cache/pbuilder/$NAME/aptcache/" BUILDPLACE="/var/cache/pbuilder/build/"

if $(echo ${DEBIAN_SUITES[@]} | grep -q $DIST); then # Debian configuration MIRRORSITE=“http://$DEBIAN_MIRROR/debian/” COMPONENTS=“main contrib non-free” if $(echo “$STABLE_CODENAME stable” | grep -q $DIST); then EXTRAPACKAGES="$EXTRAPACKAGES debian-backports-keyring" OTHERMIRROR="$OTHERMIRROR | deb http://backports.debian.org/debian-backports $STABLE_BACKPORTS_SUITE $COMPONENTS" elif $(echo “$OLD_STABLE_CODENAME stable” | grep -q $DIST); then EXTRAPACKAGES="$EXTRAPACKAGES debian-backports-keyring" OTHERMIRROR="$OTHERMIRROR | deb http://backports.debian.org/debian-backports $OLD_STABLE_BACKPORTS_SUITE $COMPONENTS" elif $(echo “unstable” | grep -q $DIST); then DIST="$UNSTABLE_CODENAME" OTHERMIRROR="$OTHERMIRROR | deb http://ftp.fr.debian.org/debian/ experimental main" fi elif $(echo ${UBUNTU_SUITES[@]} | grep -q $DIST); then # Ubuntu configuration MIRRORSITE=“http://$UBUNTU_MIRROR/ubuntu/” COMPONENTS=“main restricted universe multiverse” v=0 n=0 for i in ${DEBOOTSTRAPOPTS[@]}; do if [ $v -ne 0 ]; then DEBOOTSTRAPOPTS[$n]="/usr/share/keyrings/ubuntu-archive-keyring.gpg" fi if [ $i == “–keyring” ]; then v=1; fi n=$((n+1)) done else echo “Unknown distribution: $DIST” exit 1 fi

To make this installation easier, you can download it from here

However, before jumping in the pbuilder creation of Ubuntu versions, there’s a little thing to do if you are using Debian.

Debootstrap packages from Debian and Ubuntu are different. 1 line differs in the file /usr/share/debootstrap/scripts/gutsy. This different can cause trouble, especially under Ubuntu 11.04 “Natty”.

At the line 72 of the file /usr/share/debootstrap/scripts/gutsy, add the folllowing line :

ln -nsf . “$TARGET/var/lib/dpkg/info/$ARCH”

Now, we are finally ready.

With this handsome .pbuilderrc, you will just have to type the following ligne to create a 64 bits Debian Squeeze pbuilder :

$ DIST=squeeze ARCH=amd64 sudo pbuilder create

Or a 32 bits Ubuntu Natty pbuilder :

$ DIST=natty ARCH=i386 sudo pbuilder create

But now, you are wondering “Why did he talked about the CONCURRENCY_LEVEL variable ?”.

Here is the answer.

The variable CONCURRENCY_LEVEL is used to run several processes for only one compilation, in other words, for person who use and know make, this variable acts like the “-j” option.

To compile a source package with 4 processes under a 64 bits Debian Squeeze :

$ CONCURRENCY_LEVEL=4 DIST=squeeze ARCH=amd64 sudo pbuilder build paquet.dsc

To compile a source package with 2 processes under a 32 bits Ubuntu Natty 32 bits :

$ CONCURRENCY_LEVEL=2 DIST=natty ARCH=i386 sudo pbuilder build paquet.dsc

Here we are, the build system is up and running.

But knowing that I like to have my own useful functions which help me to automate some tasks, I create a .scripts folder in the home directory of my unprivileged user, and a script called_pbuilder_utils_ which here is the content.

In the .bashrc file I added the following line : source ~/.scripts/pbuilder_utils.

From here, it might be a bit raw, but here are explanations :

update_pbuilder updates given pbuilders in both architectures, example : update_pbuilder squeeze natty
create_pbuilder creates given pbuilders in both architectures, example : create_pbuilder squeeze natty
clean_pbuilder cleans the given pbuilders in both architectures, example : clean_pbuilder squeeze natty
build_pbuilder compiles a source package for distributions given in both architectures, example : build_pbuilder paquet.dsc natty lenny squeeze

That’s nearly all.

For the pbuilderrc file given here, I took as base the one given at the following address : https://wiki.ubuntu.com/PbuilderHowto#Multiple%20pbuilders

For the remaining things, it’s hair pulling, some hacking and multiples missed tries.

I hope you enjoy reading.