How Jenkins CI builds are set up

ci.linaro.org is one AWS instance running our jenkins setup. The runs jobs on other 'machines' (AWS images) which are started up on demand, although some are run permanently in order to keep cache status and greatly speed up builds for things like openembedded builds.

Available build images

arm kernel_cloud kernel_persistent_cloud precise precise_cloud precise_hwpack_cloud etc

Ubuntu Builds

Things built for ubuntu are built on the current LTS (long term Support) release, currently precise.

Because dpkg only has one database and more than one build job can be run on an instance at a time, it is best if all builds are run in a chroot. This also makes it easy to build things for newer suites (raring, saucy). The issue here is that if two build both try to run dpkg at the same time, the dpkg database lock makes one job or the other fail.

We use sbuild to manage this process as it works nicely with schroot to manage the chroot images.

Some older jobs still run on the bare precise build image. New ones are all being run in sbuild chroots.

This page covers building. For uploading to repo.linaro.org see: https://wiki.linaro.org/Platform/Infrastructure/Repository

Setting up a build job

Examples:

The chroot to use is called raring-amd64-sbuild, whichever architecture you are actually building for. This name indicates the suite being used (raring) and the arch it will be run on (amd64).

You can build upstream source or Debian packages, and do native or cross builds.

A simple sbuild command build packages with multiarched dependencies. Sbuild will install the toolchain, all the (cross) build deps, do the build, and clear up.

Schroot sessions are used to build upstream source, or for software where multiarch cross-dependencies don't (yet) work right. In this case you get to install build-deps and a toolchain yourself.

Componentized CI builds

Instead of using sbuild directly in your CI job, you can also use the build componen jobs in jenkins. In this case all you need to do in your job is to create an CI job that creates and Debian source package.

The source package is uploaded to snapshots.linaro.org, and the CI job then requests armhf and amd64 builds of the source package:

Ubuntu-builder takes two parameters (third, distro will be added):

* https://ci.linaro.org/jenkins/job/ubuntu-builder/13/parameters/?

ubuntu-builder component will pass the successful build result to post-build-repo job, which signs and uploads the result to repo.linaro.org overlay.

Building debian packages

Cross-builds just have --host <arch> added to build packages for architecture <arch>. That's a Debian architecture name, not a GNU one, typically armhf or arm64. ( When crossing we also need to run a system-setup command on raring - see below for sordid details).

Normally you use a command like this:

sbuild --host armhf '--chroot-system-setup-commands=configchrootforarch armhf' --nolog -d raring -c raring-amd64-sbuild package.dsc

A native build would be done with:

sbuild --nolog -d raring -c raring-amd64-sbuild package.dsc

where package.dsc is the name of your debian source package, available in the current dir.

The chroot-system-setup-commands is an extra option currently (2013) only in the Linaro-overlay 0.64.0+repack1-1wook1 version of sbuild which allows running a setup command. This is needed to set up incompatible changes between arm64 and other architectures, which will disappear once arm64 supports libssp and a bug in apt is fixed.

If sbuild gets the build-deps wrong, use the 'building upstream source' method to install build-deps manually, then run dpkg-buildpackage -a <arch>. See UsingMultiArch

Building upstream source

If your software is not packaged sbuild doesn't know how to install the build-dependencies. This is how to use the schroot 'session' functionality to run a series of commands in the build chroot, by first setting it up with --begin-session  --session-name foo, running comamnds with --run-session -c foo, and tidying up with --end-session -c foo

You will need to change the commands to install build-deps and actually build the package to suite your case, but this example should give the general idea.

arch=armhf
triplet=arm-linux-gnueabihf

schroot -c raring-amd64-sbuild --begin-session --session-name build || schroot --recover-session -c build

schroot --run-session -c build -u root -- dpkg --add-architecture ${arch}
schroot --run-session -c build -u root -- apt-get update
schroot --run-session -c build -u root -- apt-get -y install crossbuild-essential-${arch}
schroot --run-session -c build -u root -- apt-get -y install libc6-dev:${arch} libncurses-dev:${arch} uuid-dev:${arch} libglib2.0-dev:${arch} libssl-dev:${arch} libaio-dev:${arch} libyajl-dev:${arch} python gettext gcc git libpython2.7-dev:${arch}

git clone git:<somewhere>/<package>; cd <package>

schroot --run-session -c build -- env CONFIG_SITE=/etc/dpkg-cross/cross-config.${arch} ./configure --build=x86_64-linux-gnu --host=${triplet}
schroot --run-session -c build -- make dist CROSS_COMPILE=${triplet}-

schroot --end-session -c build

Just to be clear the command above installing build-deps is  apt-get -y install libc6-dev:armhf libncurses-dev:armhf uuid-dev:armhf libglib2.0-dev:armhf libssl-dev:armhf libaio-dev:armhf libyajl-dev:armhf python gettext gcc git libpython2.7-dev:armhf (run as root)

and the commands doing the build are:

env CONFIG_SITE=/etc/dpkg-cross/cross-config.armhf ./configure --build=x86_64-linux-gnu --host=arm-linux-gnueabihf
make dist CROSS_COMPILE=arm-linux-gnueabihf

The rest is the boilerplate.

Setting up a matching local environment

To set up the environment used on the CI machines locally for testing your recipies do the following:

See https://wiki.linaro.org/Platform/Infrastructure/Repository#Using_the_repositories for how to add the the repo and archive key to your system.

Then do:

apt-get install sbuild raring-crosschroot

Now you can do builds inside that chroot just as described above.

You will need to sign the packages, and get your key registered with the repo admins, and use the right dput config if you want to upload the results. See https://wiki.linaro.org/Platform/Infrastructure/Repository#Uploading

Platform/DevPlatform/SettingUpCIBuilds (last modified 2013-09-26 11:30:07)