Linaro Continuous Integration Service

Continuous Integration (CI) service is going to be a cornerstone to providing added value to any development process.
It aims at providing automated build and runtime testing through LAVA. By doing so we can catch issues in software/applications as soon as patches are commited to them.

For example:
CI would allow any kernel tree [ like linux-linaro-3.0, linux-arm-soc tree for-next, linux(linus), linux-next ] to be contiuously built, booted and tested on linaro member platforms [ like panda, beagle, origen etc]. Results of the tests will then be publicly available.

The aim of the CI is to catch the build failures early in the development cycle and notify maintainers to fix the issues or revert the offending patches. The Continuous Integration Build/Test Service is an instance of jenkins that uses master/slaves provisioned from EC2.
The information here is mostly a guide on how the service was set up and how we can request for a new build to be included in the CI serivce.
The Linaro Continuous Integration Service(Linaro CI) helps you to build your changes and as well test it.

Assumptions

The service is setup assuming it will be used for building gcc and kernel trees right now and may not work well for building some other project.

Design

The configuration of each build is provided as shell commands that contains environment variables. The init script present in the global configuration helps system set up, such as to install the default-jre and bazaar tool required further for execution of each job.
The build logs will be available for a period of 60 days or a max of 100 builds, the artifacts will only be kept for a few days. The build logs will be visible on the web UI as the build progresses.

Implementation

Use Jenkins, and jenkins-ec2 plugin to provision slaves from ec2.

EC2 set up

The master node, will be the place where jenkins server will run and control the slave nodes. There are ephemeral slave nodes which are created depending on the job needs. These will be owned by role account, jenkins.
The master is basically to run the server and server as the repository for storing the archives mostly, while the slaves are the nodes where the actual job executions occurs.

Security groups

The master node is associated with the security groups jenkins-master and git-mirror. This is required for the Jenkins to be able to work over the internet and to download stuff.

The master node

The master is not intended to execute any build, no excessive memory or CPU is required.
Hence it was choosen to have a medium ebs instance. We need a reliable instance and hence an EBS instance type.
The Jenkins master node uses the elastic ip address and a friendly name and is hosted on https://ci.linaro.org and can be easily accessed using a web browser.
It seems to be working with Mozilla firefox and chrome. Jenkins/CI service is hosted behind the apache over SSL. For now we use a self signed certificate for SSL.

The slave nodes

The more CPU and memory we give them, the faster the builds go, obviously. So the slaves are little expensive when compared to the master.
An 64-bit instance of LARGE instance type is chosen to be a slave which will be provisioned when a build is requested.
The images have been choosen from http://uec-images.ubuntu.com or the images are based on Custom AMI . The maintainers of the overall system will be able to ssh into the slaves from the master.

Additional mirror/caching services

To improve build performance and reliability and alleviate load on upstream git, etc. servers, build script employ different means to buffer, cache, and mirror downloads/SCM checkouts. Among them:

(Semi)transparent HTTP caching using Squid for kernel builds

This is enabled by setting http_proxy environment variable in slave init script(now part of the lp:linaro-ci/node/setup* scripts which are inturn called in the slave init scripts).

The Squid service requires custom /etc/squid/squid.conf, but otherwise just installed from Ubuntu package. The cache efficiency is not ideal so far and need further investigation and tweaking. See lp:966998 for some additional info.

Apart from this, there is a custom squid upstart script that was developed to address bug 1003835. We need to place on ci.linaro.org machine and is available @ lp:~linaro-automation/linaro-ci/ci-infra-conf-files. The instructions how to configure this script is given in the README.

Configuring jenkins

The general configurartion

Install the upstream built deb (at least until jenkins is in the ubuntu archives).

  1. Go to the 'configure' page. Set the "# of executors" to 2 to allow only those builds which are needed to be run on master and also to prevent any builds overloading the master.
  2. Under the “Security Realm” OpenID SSO is choosen and give the https://login.launchpad.net/+openid to it so that launchpad Single Sign On information can be used to login in to the jenkins service

  3. For “Authorization” the Matrix-based security is enabled.
  4. The E-mail Notification uses a special email id ci_notify@linaro.org to which the unstable build information is sent.

EC2 setup

Install the jenkins-ec2 plugin.
The ec2 plugin requires fairly substantial configuration:

  1. Click 'new cloud'.
  2. Paste in the aws access key id and secret access key for the account that should be creating the slaves and a key pair .pem for that user.

  3. Click 'new AMI'.

  4. Follow the steps of building Custom AMI .

  5. An instance type -- this is a time/cost trade off. Using an XLARGE or XLARGE_CPU instance is sufficiently much faster that they are probably the two to choose between at this time.
  6. Remote user would be 'ubuntu'.

/mnt/ci_build as the remote root -- this is required as the build will run out of space if we used User's home directory.

  1. Slave labels are probably important, as ci.linaro.org has numerous jobs with each having different configuration.

  2. The init script contains any installation or init steps which was not made part of the Custom AMI

  3. The root command prefix to 'sudo'.

Setting up Jobs on ci.linaro.org

The following section explains how to use ci.linaro.org for building and testing the linaro GCCNative for x86 and testing kernel trees.
ci.linaro.org can be extended with some modifications to work for different projects as well.

There are 2 ways with which you can configure a job on Jenkins:

  • either request the Linaro Infrastructure team to create a new job, or
  • get access to the ci.linaro.org to create them directly.

Read more at Detailed information on setting up new job.

ci.linaro.org Jenkns config

ci.linaro.org is a host for numerous jobs. Hence it becomes very important that we back up the jenkins job config frequently. Jenkins configuration (global/jobs/users) are all stored in a lp:linaro-ci/trunk private branch.

Apart from storing the jenkins specific configuration, lp:linaro-ci/trunk also contains apache configurations for jenkins that are currently used on ci.linaro.org. The apache configuration can be found under the apache_configs directory.

The details of back up process is available at JenkinsConfigBackup.

Example set up steps for new kernel/gcc jobs

If you are someone who is well versed with Jenkins and need to create/update jobs regularly,

Get ci.linaro.org access.

Once you are added to the " Linaro CI Builder and Testing service" team you will have access to create, build and test new jobs while keeping guidelines in mind.

Here is a small step of how to build each one of them.

setting up new kernel build job

Once you have got sufficient permission to create a job, the following information should take few minutes to get a kernel build and testing started

Select one of the existing job and use any one of the jobs as a reference to setup a new job.

Here are the steps to create a new kernel job. This will cross compile the kernel and create a new debian package. At the end of this execution you will get a new hwpack created which will have your newly created deian package.

1) Login to Jenkins and click on the “New Job” icon listed on the top left of the window.

2) Give an meaningful name which describes the purpose of the Job

3) Select the “Build a free-style software project” option.

4) Copy the  <<linux-linaro-tracking-llct-branch_panda-omap2plus job>>  which explains how to build Linux Linaro Tracking for panda board using the omap2plus defconfig on a ec2 slave provisioned dynamically. All the kernel related jobs existing on the jenkins use the scripts sitting under the branch lp:~linaro-infrastructure/linaro-ci/lci-build-tools

5) Click on the “Restrict where this project can be run” option and select appropriate slave to use to build the job.

6) Under the Source Code Management select the git option [if your code is hosted on git repository] and provide the link to the git tree for the Repository URL. For example for Linux Linaro Tracking the git repo would be http://git.linaro.org/git-ro/kernel/linux-linaro-tracking.git.

7) You can specify the build to be executed at some frequency, for example to execute the job once in a day you could enter @daily under the Build Periodically option.

8) Go to the “Execute Shell” option and verify the shell commands that your project needs. If the existing commands are sufficient leave it as it.

9) If you want the build failures to be notified then you can specify the email information in the whitespace separated formation by checking the “E-mail Notification” option.

10) click save, you can try to verify  the instructions work correctly by clicking on the “Build Now” option available on the left side of the window. Sit back and relax or go for a cup of tea while your build is done.

setting up gcc native build job

More details on building and testing of linaro gcc can be found @ https://wiki.linaro.org/WorkingGroups/ToolChain/Using/GCCNative
Once you have got sufficient permission to create a job, the following information should take few minutes to get a gcc build and testing started

1) Login to Jenkins and click on the “New Job” icon listed on the top left of the window.

2) Give an meaningful name which describes the purpose of the Job

3) Select the “Build a free-style software project” option.

4) Copy the gcc-test-build_on_slave which explains how to build gcc Native for x86 on a ec2 slave provisioned dynamically

5) Click on the “Restrict where this project can be run” option

6) Under the Source Code Management select the Bazaar option  and provide lp:gcc-linaro for the Repository URL or any other gcc repository you want to build

7) Modify the “Poll SCM” option under the Build Triggers .

8) Go to the “Execute Shell” option and verify the shell commands that your project needs. If the existing commands are sufficient leave it as it.

9) If you want the build failures to be notified then you can specify the email information in the whitespace separated formation by checking the “E-mail Notification” option.

10) click save, you can try to verify the instructions work correctly by clicking on the “Build Now” option available on the left side of the window. Sit back and relax or go for a cup   of tea while your build is done.

Random Notes That Might Help CI Developers

Detailed information on setting up a new job on ci.linaro.org

Platform/Systems/LinaroCI (last modified 2018-07-23 16:20:57)