This page covers various jobs/roles inside the Working Group, who does them, and who is the alternate when the primary is unavailable.

Michael has automated these as far as he can. Automating yourself out of a job is left as an exercise to the reader.

Auto build health

Check the scheduler each day to see that the auto builders are running and the backlog isn't stuck.

Who: Åsa then Zhenqiang

Check for:

  • Too deep a backlog
  • Hosts that are stuck: building too long, idle too long, etc
  • Hosts that have been reserved for too long
  • oort hosts that have been starting too long (suggests stuck EC2 instance)
  • Hosts that are always updating (suggests disk full or error)
  • Missing hosts (unlikely)

Tools and permissions needed:

  • Log in to
  • Log into auto builders (tcpanda0x.v) as user or cbuild
  • Log into the EC2 driver ( as cbuild
  • Kill EC2 instances

A machine shows in different states. They are:

  • Lurking - polling for jobs. The cloud instance is terminated
  • Idle
  • Updating - polling the scheduler for updates and the next job
  • Running - running a job


  • 'Forget' removes the host from the list. The host will be re-added next time it polls
  • pavo1 is the gateway machine and never runs jobs. It's listed for heartbeat purposes
  • gcc-build takes around five hours, as does gcc-testsuite. pgo and lto jobs can take up to 24 hours on a board.
  • You can see the screens of the cloud builders. Log in as cbuild@apus; screen -ls; screen -r <name>

  • Common problems and their solutions are listed here. Please add as they come along.


  • Michael added Åsa's public key to cbuild@apus

Proposals health

The proposals tool watches for merge requests, builds them, and then gives a web interface for recording the results. This manual step is used to catch errors in the build machines themselves.

Who: Åsa then Zhenqiang


  • Subscribe to all merge requests
  • Check the proposals twice or more a day

  • Check all proposals as they finish
  • If the build looks good, record
  • If the failures look related to the branch name, record (such as a 'free' test failing where 'free' is in the branch name)
  • If not, escalate

Check for:

  • Merge request emails but nothing on the proposals list (suggests stuck update)
  • Builds that don't come through (unlikely: suggests missed request, broken builder)
  • Architectures that don't come through
  • Diffs that get stuck in 'Finishing' (suggests missing build for the ancestor)
  • Bad build results where the failure is unrelated to the branch (suggests builder problem)

Tools and permissions needed:

  • Release jobs on the scheduler

Ticket triage

Triage new bugs in a reasonable time by running the ../BugProcess.

Who: Åsa then Zhenqiang


  • Subscribe to all new bugs
  • Check the ticket linter each day

  • Triage any new tickets
  • Monitor any In Progress tickets. Escalate any that get stale

Triage process:

  • Fetch the corresponding auto build
  • Reproduce the problem
    • Experiment with different command line options
      • -mcpu=cortex-a8 vs -mcpu=cortex-a9
      • -mfpu=vfpv3-d16 vs -mfpu=neon
      • -marm vs -mthumb
      • -O0 vs -O1 vs -O2 vs -O3
      • -g vs without
  • Minimise the problem
    • Reduce the test case
    • Reduce the command line
  • Find a work around
    • Experiment with -O levels
    • Experiment at the same -O level but turn off various passes
  • Reproduce on other builds
    • GCC 4.6/4.5 releases
    • Current trunk
    • Earlier gcc-linaro releases
  • If possible, dive in:
    • Check a range of builds and see where the fault was introduced
    • See if the fault occurs in cross
    • Get a backtrace
    • Examine the problem code
    • Examine the commit log and suggest a suspect
    • Also have a look at this example run through

Note that we're a branch of upstream. All bugs should be checked against upstream and, if present, reported and fixed there.

Tools and permissions needed:

  • ARM board in the standard configuration
  • x86_64 and i686 hosts with Natty
  • Access to the auto build results for ARM and x86
    • ARM are currently on control

    • x86 are in S3

GCC release

Run a Linaro GCC release from testing through tagging, upload, and announce.

Who: Andrew then Ramana

PENDING: Describe

GDB release

Run a Linaro GDB release from testing through tagging, upload, and announce.

Who: Thiago then Ulrich

PENDING: Describe

QEMU release

Run a Linaro QEMU release from testing through tagging, upload, and announce.

Who: Peter then Rusty

PENDING: Describe

GCC rebase

Update the Linaro GCC tree against the FSF GCC release branch.

Who: Andrew then Michael

PENDING: Describe

Binary build

Update the binary build as part of a release. Each source release should come with a binary release the same day.

Who: Zhenqiang then Michael


  • Update config/cc/ to the latest linaro release version.
  • Update config/debug/ to the latest linaro release version.
  • Update all linaro config files: samples/linaro*/crosstool.config: CT_TOOLCHAIN_PKGVERSION, linaro gcc & gdb version.

  • Update version info in contrib/linaro/doc/README.txt
  • Update dir name in contrib/linaro/patches/gcc/ amd contrib/linaro/patches/gdb to new version name.
  • Follow to build, validate and release.

  • If Windows installer is not created, need update contrib/linaro/win32installer/arm-linux-gnueabihf.mpi. Installjammer-1.3-snapshot supports lamz (solid) compression, but it is not stable. Here is the steps to update project file:
    • cp arm-linux-gnueabi.mpi tp ~/InstallJammerProjects/Install Project 1/Install Project 1.mpi
    • Launch installjammer and Install Project 1.
    • "Delete" gcc-linaro-...win32 from "Components and Files"->Program Files->

    • cp contrib/linaro/win32installer/gccvar.bat to the "bin" directory of the windows package.
    • "Add Directory to File Group". Then select the top directory of winows package.
    • click on gcc-linaro-...win32. Update the configure in the right panel.
    • Save and cp the Install Project 1.mpi to arm-linux-gnueabi.mpi


Spawn benchmark jobs for merge requested. Will be automated.

Who: Åsa then Zhenqiang


python linaro-toolchain-benchmarks/scripts/ new-logs/*run.txt > new.txt
python linaro-toolchain-benchmarks/scripts/ ancestor-logs/*run.txt > ref.txt
python linaro-toolchain-benchmarks/scripts/ ref.txt new.txt > diff.txt


Reserve the first tcpanda that comes free to reproduce a bug or benchmark result against an existing binary build.

Who: Everyone

For example, say you want to see if a fault exists on recent trunk:

The board will drop back into the queue after 48 hours. You can release it early by doing a 'killall sleep'. Please do any work under /scratch/cbuild/slave/slaves as this is automatically deleted when the next job starts.

WorkingGroups/ToolChain/Jobs (last modified 2012-12-04 22:53:07)