Intro

The site ci.linaro.org enables you to build and test your code submissions. It currently supports building gcc and kernel trees and may be extended with some modifications to work for different projects as well.

This page gives a brief description of simple functions and how to get help.

Whom to contact for help

The help contacts for ci.linaro.org services is the Linaro infrastructure team [email: infrastructure@linaro.org].

Guideline on using ci.linaro.org

Creating a new job

Jobs on ci.linaro.org can be created by users belonging to Launchpad team "Linaro CI Builder and Testing service". There are 2 ways with which you can configure a job on jenkins:

Either way, please go to https://launchpad.net/linaro-ci/+filebug to track the job creation. Follow the guidelines on the bug details.

If the priority of the bug needs to be elevated, set the priority to high and make sure to contact the appropriate people associated with the bug. General help for filing bugs can be found here.

Getting Access to create new jobs

In order to be able to create jobs on ci.linaro.org one needs to be part of the launchpad team called "Linaro CI Builder and Testing service".
If someone wants to be able to create a new job then you need to be part of this team.
Anyone belonging to the linaro-infrastructure team can add you to Linaro CI Builder and Testing service.

Create new jobs on your own

Once added to this group you should be able to login to ci.linaro.org using your launchpad id and create the jobs yourself by following guidelines. .

Request a new CI job creation

If you don't have experience with Jenkins and would like us (linaro-infrastructure team) to create a job for you, then all you need to do is:

  • create a bug/raise a requirement

  • Fill in the details in the following format for build and test job and we will get back to you if we require more information to setup and we will give you information on the where and how the job is created.

    •       We want to pull in the the Linus' tree into Linaro's CI loop and build it on a regular basis.
      
            Description: quick summary why you want this builds and as part of which effort you want it       
      
            git tree URL: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
            git branch: master
      
            kernel configs:
      
            exynos4_defconfig
            u8500_defconfig
            mx51_defconfig
            omap2plus_defconfig
      
            Build frequency: daily for now.
      
            root file system image: nano
      
            testing required: build + boot to console
           
            Importance: Medium
       
            Deadline: when do you need this tree operational? 
      
        
        

PS: The importance of the bug needs to be changed to High if setting up a new job is urgent.
Also, note that the kernel CI jobs build the kernel using the make deb-pkg commands. Below is the steps that are the make commands used to build the existing kernels.

  •     make ARCH=arm CROSS_COMPILE=arm-linux-gnueabi- -j$j_count uImage
        make ARCH=arm CROSS_COMPILE=arm-linux-gnueabi- -j$j_count modules
        make ARCH=arm CROSS_COMPILE=arm-linux-gnueabi- KBUILD_DEBARCH=armel V=1 deb-pkg

If you think your build uses some other commands, then please do mention the same in the bug report as well. If your build takes prolonged time to complete, please include expected duration too, so we can adjust build monitoring and avoid prematurely killing incomplete builds due to timeout. As a rule of thumb, if build takes more than 2 hours, that should noted explicitly.

NOTE: Once the job is setup it is the responsibility of the owner of the job to monitor and fix build failures. Build problems can be notified by setting up your email id in the job, in which case the build failures are directly sent to the email id specified or if you wish to use the RSS feed to monitor the failures then refer to RSS Feeds and Build status.

A Blueprint to track the progress of job creation

There needs to be a way in order to understand the progress of the new job that is being setup on the ci.linaro.org. Hence recommendation whoever creates the new job (either linaro-infrastructure team or if you) should create a new blueprint under the Linaro-CI project for tracking purpose.

Here is the example BP whiteboard template that is required for setting up new builds for trees/defconfigs etc. The essential information are the fields Headline: Acceptance: and Work Items:

Headline: Continuous Integration jobs for panda kernel are built and submitted to LAVA.

Acceptance: Daily kernel builds from the approprate branch occur, hwpacks are produced using the latest snapshots.linaro.org hwpacks as template and hwpacks are submitted to LAVA and run using a nano image.


Work Items:
Add new job for the linaro linux 3.1 using omap2 defconfig to work on Panda board, make use of the lci-build-tools for the jobs: TBD
Validate linux linaro 3.1 kernels build successfully for Panda Board: TBD
Validate that kernels get packaged into hwpack: TBD
Validate that hwpack gets submitted to and accepted by LAVA scheduler: TBD
Validate the build for linaro linux 3.1 using omap2 defconfig to work on Panda board  : TBD
Validate the lava test execution for linaro linux 3.1 using omap2 defconfig to work on Panda board : TBD

Guidelines and Tips on job creation

Here are some tips that one needs to follow when creating a new job on ci.linaro.org

  1. We should restrain from keeping old build logs as it leads to high disk usage over a period of time. The retention policy is to keep the build log for not more than 60 days and the Max # of builds to keep should be restricted to 100. This information can be configured in the Job "Discard Old Builds" option.
  2. All the heavy duty jobs should be configured only to run on the slaves.
  3. Each time you're going to create a long-running (more than 2hrs) job, please notify Infrastructure team, as described above, to keep monitoring and stats on build time usage up to date, and to avoid getting reports from the ec2 maintenance tool of exceeding usage limits.

Monitoring Build Failures

Build failures can be tracked through using one of the following approaches depending on your role/needs:

  • Find your view which contains the job of your interest on ci.linaro.org; there, on right and the bottom of that page will have RSS links for your convenience, Click on RSS links and you will be taken to the list of builds and status
  • Find your job of interest on ci.linaro.org, click "job name", on the left and bottom of the page you will have RSS links for your convenience.
  • Go to http://validation.linaro.org/lava-server/kernel-ci-views/index, you will be directed to list of boards, click on the board for which you want to see the build results, you will be directed to a page where information with heading " Build of <defconfig> would tell you the what is the status of the build for that tree, defconfig and board.

NOTE: The information under http://validation.linaro.org/lava-server/kernel-ci-views/index is limited to only standard daily kernel builds. If you have a need to include your results in the same place, then please speak to mwhudson@linaro.org or anyone from validation team.

More details with screenshots for the RSS feature can be found on /RSSForBuildFailures.

Monitoring the Build time

ci.linaro.org makes use of the ec2 instances to run their builds, which costs us money for the time we use ec2 instances. Hence as a responsible user of the ci.linaro.org one should make sure that the builds are on an average take the same amount of time.

The build time trend can be monitored by Clicking on the job on ci.linaro.org, on the Build history Lable click "trend". You will be directed to a page which will show the time taken by each of the builds. If some builds start taking more time, then its time for us to investigate the delay in completion of builds and help save some money.

Finding Test Results

Tests are currently run in LAVA and the results go as expected into the validation dashboard database. The test results for a standard kernel tree can be seen along with the build results @ http://validation.linaro.org/lava-server/kernel-ci-views/index.

On this page click on the board for which you want to see the test results, you will be directed to a page where all the information above the heading " Build of <defconfig> would tell you the what is the status of the various test stages for that tree, defconfig and board.

NOTE: The information under http://validation.linaro.org/lava-server/kernel-ci-views/index is limited to only standard daily kernel builds. If you have a need to include your results in the same place, then please speak to mwhudson@linaro.org or anyone from validation team.

In case your tree/job is not being included in the index on validation.linaro.org, there are ways to track down your CI build and find the runtime test results for it. Here the current steps:

  1. Find the bundle stream for your tree on the Linaro validation dashboard:
  2. Find your job by using the search box
  3. click on it to get the test results
  4. the lava_results contain the meta test results that give you info about the job execution steps in the lab

Here a few examples:

Quick Links

Platform/Systems/LinaroCIQuickReference (last modified 2015-07-14 19:19:35)