Moving cbuild

draft r0

cbuild is currently spread out. Here's a moving plan.

Result storage:

  • Holds all build logs and artifacts
  • Mainly new files, reads
  • > 100 GB

  • Directories with files, often text
  • Compresses well; btrfs in compress mode does a very good job
  • OK if lost. Worthwhile to set a rsync backup somewhere else
  • ACTION: add a disk to tcserver01
  • ACTION: add a live backup server

Snapshots storage:

  • Holds all original source tarballs
  • Mainly new files, reads
  • Available over HTTP
  • ACTION: add alongside result storage

builds.linaro.org:

  • Public copy of the results
  • Web server. Static content.
  • rsynced from result storage. Is a waste
  • ACTION: replace inside the lab

Scheduler:

  • Web app
  • Publicly visible
  • WSGI web.py plus Python
  • Pass through from control?
  • ACTION: set up a reverse proxy and domain. cbuild.linaro.org?

x86 builders:

  • In EC2
  • ~$300/year
  • Could replace with chroots on tcserver01
  • Need a fallback, which could be a bounce back to the cloud
  • ACTION: defer

ARM builders:

  • Already in the lab

HTTP Proxy:

  • Most things go over HTTP. A proxy greatly reduces the transfer
  • ACTION: use the Squid proxy on control

Snapshotter:

  • Polls tip, merge requests, and upstreams
  • Snapshots on change or time
  • Generates tarball, ends up in snapshots
  • Spawns jobs
  • ACTION: shift to tcserver01

Other tools:

  • Web app
  • Works directly on the filesystem. Need direct access to the results and snapshots
  • PENDING: must run on tcserver01
  • ACTION: shift to tcserver01

Summary

  • Add a >= 500 GB disk to tcserver01

  • Add/reuse a separate area for backup
  • Add a reverse proxy to expose cbuild.linaro.org
  • Shift all snapshots, test
  • Shift the snapshotter, test
  • Shift the scheduler, test
  • Replace builds.linaro.org

Risks:

  • Internet connection needs to be reliable, fast enough. Proper QoS?
  • Reverse proxy needs to be stable

Advantages:

  • Linaro provided infrastructure
  • All results in one location
  • Removes temporary result storage from control
  • Provides infrastructure for LAVA wrapped cbuild jobs
  • Stops shuffling data around the world
  • Should be more reliable than Michael's setup

MichaelHope/Sandbox/CBuildMove (last modified 2012-06-25 04:26:34)