Linaro Image Tools Regression setup details
Linaro Image Tools Regression Job setup
Linaro is evolving with every passing day, so is the tools/softwares developed. With the new development we need to make sure that the backward compatibility is not compromised. There are cases where the new changes in Linaro Image Tools(LIT) seem to be backward in compatible. For ex: https://bugs.launchpad.net/linaro-image-tools/+bug/1050349.
In order to avoid such problems in future, we setup automation to build all major board variant image files for all old linaro releases in jenkins continuously with lmc/lamc. This is to ensure that Integrate checks for all builds go green for a commit we plan to release and document that as part of the release announcement.
The work items which handles the requirements is listed in the Setup Automation for testing all old linaro releases BP.
The lmc/lamc regression jobs are setup on ci.linaro.org and can be directly viewed from the tab Release Builds Testing.
ci.linaro.org is a service that serves more than 50 builds every day. One point that was considered is to be able to setup as much minimum number of jobs to achieve the vast list of job builds. We had several options to achieve this.
- Use Matrix Plugin which triggers combination of jobs in parallel.
- Write a simple Job for boards and others for releases and use Parametrized trigger plugin in the job which would start the release jobs which inturn calls the board specific jobs.
Use a MultiJob plugin which triggers the jobs in parallel as well give a consolidated status information in upstream job depending on the downstream jobs status.
The problem with the approach 2 was once the build was triggered, jenkins messed up the build numbers and gave incorrect build numbers against the various builds running.
The jenkins might have been encountering race condition or something due to the parallel start of enoromous number of jobs. Apart from this there was not a good single view which would give us a consolidated representation of the jobs being run. So that leaves us with the options 1 and 3 which are favourable to achieve what we need. An ideal case would have been to use a multijob and matrix plugin together, but both of them are complimentary things. Also, when the various combinations of jobs using the matrix plugin was tried it always deduced that we would have longer wait periods to get builds finised. Also, there would not be a easy way to trigger just one build for example trigger just the 11.06 builds for panda although it is not something impossible but to achieve it one needs to know a little bit of how the matrix plugin work and click and the right place. So that left us with option 3. The Multi Job plugin basically helps us
- In case we like to stop the mess with downstream / upstream jobs chains definitions (which was one problem with simple Parametrized trigger plugin)
- When you want to add full hierarchy of Jenkins jobs that will be executed in sequence or in parallel
- Add context to your buildflow implementing parameter inheritance from the Multi Job to all its Phases and Jobs, Phases are sequential whilst jobs inside each Phase are parallel
- Updated the upstream job status depending on the downstream job status(which is not available in the Parametrized trigger plugin).
A pictorial representation of the implementation of the job is given in the diagram below:
The source for this diagram is here: LIT-regression-job-setup.
Add missing release builds for testing
Before running the builds, Click on the builds triggers ( ex: lit-release-ubuntu-nano-builds-testing_trigger) one by one, under Downstream Projects Heading please verify that all the builds we need to test are present. The jobs are added cover the release from 11.06 uptil 12.09. If something is missing or if we have a new release, we need to append a trigger job to the appropriate trigger build job.
For ex: For a new nano release for ex: 12.10 we would need to create a new job lit-release-build-testing_12.10 and just change the platform variable. This new job can be needs to be appended to https://ci.linaro.org/jenkins/job/lit-release-ubuntu-builds-testing_trigger/configure#section63. Thats it the next time you want to trigger all the previous builds from 11.06 to 12.10, you need to start the lit-release-ubuntu-builds-testing_trigger build. Similar things need to be done for other Image(ubuntu-desktop/android/developer/alip) as well.
How to trigger the jobs and interpret the results ?
There are many ways in which one can trigger a job.
All the releases and for all the board types
Trigger Jobs for all the releases and for all the board types
android image Android Release Build Testing and click build now
nano image Ubuntu Release Build Testing and click build now
alip image Ubuntu alip Release Build Testing and click build now
ubuntu-desktop Ubuntu Desktop Release Build Testing
developer image Ubuntu Developer Release Build Testing and click build now
The results for all the releases and for all the board types of for ex for android image can be viewed at Android Image results for all releases and boards.
One needs to go to the list of boards listed for each release to make sure there are no failures. If there is a red ball against a particular board it means the build failed. One can easily see the log of the failure by clicking the black console box listed against the failed board and then navigating to console output.
For a particular release and for all the boards types
Trigger build jobs only for a particular release ( ex 11.06) and for all the boards
Go to Android Release Build Testing and then click lit-release-android-builds-testing_11.06 and click build now. This will just build the 11.06 android release for all the board types. Similar steps apply for other releases as well.
The results for 11.06 releases of android and for all the board types can be viewed by https://ci.linaro.org/jenkins/job/lit-release-android-builds-testing_11.06/<buildnum>/console. The console will list the links to all the board type builds along with the build number and status of the builds. If you need to know the details of the failure/success you need to click on the build buildnumber or status link and then click on console output.
We were not able to make use of the point (4) of the multijob plugin. Currently, the upstream job does not reflect the status depending on the downstream job status. Multijob plugin provides 3 exit criteria's for marking the phase status. Failed/Successful/Complete(always continue). We cannot select criteria Failed. Selecting Successful means the phase would be executed completely if all the phase jobs are successful or at the occurance of the first job failure the parent job exits, which is not suitable for us. We want to be able to run through all the jobs in the phase.
Hence we need to ignore the status Parent Trigger jobs for example lit-release-android-builds-testing_trigger status or lit-release-android-builds-testing_11.06 trigger jobs, instead click on the release specific job(ex lit-release-android-builds-testing_11.06) --> click on the ball of the just run build (latest build number) or the build you are interested in. This will give you a good list of jobnames, buildnumbers and status which makes it easier to understand the status of jobs for the release.
How to interpret the result ? For ex: If you go to https://ci.linaro.org/jenkins/job/lit-release-android-builds-testing_11.06/6/console. The job name in the console reflects the release builds that was tested. The console output will list of all the board types that were run for the along with their status and a link to the build number where it was executed so that if you click on them you can get to the console details.
Platform/Systems/LITRegression_Setup (last modified 2014-06-24 17:47:29)