Linaro Image Tools Regression setup details

Linaro Image Tools Regression Job setup

Introduction

Linaro is evolving with every passing day, so is the tools/softwares developed. With the new development we need to make sure that the backward compatibility is not compromised. There are cases where the new changes in Linaro Image Tools(LIT) seem to be backward in compatible. For ex: https://bugs.launchpad.net/linaro-image-tools/+bug/1050349.

In order to avoid such problems in future, we setup automation to build all major board variant image files for all old linaro releases in jenkins continuously with lmc/lamc. This is to ensure that Integrate checks for all builds go green for a commit we plan to release and document that as part of the release announcement.

The work items which handles the requirements is listed in the Setup Automation for testing all old linaro releases BP.

Setup details

The lmc/lamc regression jobs are setup on ci.linaro.org and can be directly viewed from the tab Release Builds Testing.

ci.linaro.org is a service that serves more than 50 builds every day. One point that was considered is to be able to setup as much minimum number of jobs to achieve the vast list of job builds. We had several options to achieve this.

  1. Use Matrix Plugin which triggers combination of jobs in parallel.
  2. Write a simple Job for boards and others for releases and use Parametrized trigger plugin in the job which would start the release jobs which inturn calls the board specific jobs.
  3. Use a MultiJob plugin which triggers the jobs in parallel as well give a consolidated status information in upstream job depending on the downstream jobs status.

The problem with the approach 2 was once the build was triggered, jenkins messed up the build numbers and gave incorrect build numbers against the various builds running.

The jenkins might have been encountering race condition or something due to the parallel start of enoromous number of jobs. Apart from this there was not a good single view which would give us a consolidated representation of the jobs being run. So that leaves us with the options 1 and 3 which are favourable to achieve what we need. An ideal case would have been to use a multijob and matrix plugin together, but both of them are complimentary things. Also, when the various combinations of jobs using the matrix plugin was tried it always deduced that we would have longer wait periods to get builds finised. Also, there would not be a easy way to trigger just one build for example trigger just the 11.06 builds for panda although it is not something impossible but to achieve it one needs to know a little bit of how the matrix plugin work and click and the right place. So that left us with option 3. The Multi Job plugin basically helps us

  1. In case we like to stop the mess with downstream / upstream jobs chains definitions (which was one problem with simple Parametrized trigger plugin)
  2. When you want to add full hierarchy of Jenkins jobs that will be executed in sequence or in parallel
  3. Add context to your buildflow implementing parameter inheritance from the Multi Job to all its Phases and Jobs, Phases are sequential whilst jobs inside each Phase are parallel
  4. Updated the upstream job status depending on the downstream job status(which is not available in the Parametrized trigger plugin).

A pictorial representation of the implementation of the job is given in the diagram below:

LIT-regression-job-setup.svg

The source for this diagram is here: LIT-regression-job-setup.

Add missing release builds for testing

Before running the builds, Click on the builds triggers ( ex: lit-release-ubuntu-nano-builds-testing_trigger) one by one, under Downstream Projects Heading please verify that all the builds we need to test are present. The jobs are added cover the release from 11.06 uptil 12.09. If something is missing or if we have a new release, we need to append a trigger job to the appropriate trigger build job.

For ex: For a new nano release for ex: 12.10 we would need to create a new job lit-release-build-testing_12.10 and just change the platform variable. This new job can be needs to be appended to https://ci.linaro.org/jenkins/job/lit-release-ubuntu-builds-testing_trigger/configure#section63. Thats it the next time you want to trigger all the previous builds from 11.06 to 12.10, you need to start the lit-release-ubuntu-builds-testing_trigger build. Similar things need to be done for other Image(ubuntu-desktop/android/developer/alip) as well.

How to trigger the jobs and interpret the results ?

There are many ways in which one can trigger a job.

All the releases and for all the board types

Trigger Jobs for all the releases and for all the board types

View Result

One needs to go to the list of boards listed for each release to make sure there are no failures. If there is a red ball against a particular board it means the build failed. One can easily see the log of the failure by clicking the black console box listed against the failed board and then navigating to console output.

For a particular release and for all the boards types

Trigger build jobs only for a particular release ( ex 11.06) and for all the boards

View Result

known Issues

We were not able to make use of the point (4) of the multijob plugin. Currently, the upstream job does not reflect the status depending on the downstream job status. Multijob plugin provides 3 exit criteria's for marking the phase status. Failed/Successful/Complete(always continue). We cannot select criteria Failed. Selecting Successful means the phase would be executed completely if all the phase jobs are successful or at the occurance of the first job failure the parent job exits, which is not suitable for us. We want to be able to run through all the jobs in the phase.

Hence we need to ignore the status Parent Trigger jobs for example lit-release-android-builds-testing_trigger status or lit-release-android-builds-testing_11.06 trigger jobs, instead click on the release specific job(ex lit-release-android-builds-testing_11.06) --> click on the ball of the just run build (latest build number) or the build you are interested in. This will give you a good list of jobnames, buildnumbers and status which makes it easier to understand the status of jobs for the release.

How to interpret the result ? For ex: If you go to https://ci.linaro.org/jenkins/job/lit-release-android-builds-testing_11.06/6/console. The job name in the console reflects the release builds that was tested. The console output will list of all the board types that were run for the along with their status and a link to the build number where it was executed so that if you click on them you can get to the console details.

Platform/Systems/LITRegression_Setup (last modified 2014-06-24 17:47:29)