Contents
- Overview
- Process Information
-
Specs, HowTo's & Setup Information
- Configuration Management - SCM - Ansible
- Linaro Image Tools
- EC2 Usage and Best Practices
- Continuous Integration Service
- Android-related Infrastructure
- patches.linaro.org (patchmetrics)
- releases.linaro.org , snapshots.linaro.org and License Protection Handling
- repo.linaro.org
- Jenkins server Local setup HowTo
- Setting up Jenkins development environment
- Deploying Jenkins to ci.l.o
- CBuild/LAVA integration
- Git & Gerrit servers and instructions
- Toolchain Mirror
- OpenJDK Mirror
- bugs.linaro.org
- Austin Colo Facility
- Hetzner
Overview
This page is a one stop place which hosts all the specs, manuals, howto's and other project related information handled by infrastructure team. Please note that some specs which referenced here are pre-implementation documentation they may be or may be not updated for changes during initial implementation phase, and for sure not updated during further development and maintenance phases. Thus, they are provided for completeness and historical purposes (but they also may include discussion of choices, decisions, and implementation details not available elsewhere). On the other hand, manual and howto's are expected to be updated throughout project's lifetime.
Process Information
IT Services Communication
User account management: https://login.linaro.org/
Atlassian Crowd account management (internal use; has group management): https://login.linaro.org:8443/crowd
https://collaborate.linaro.org/display/IKB/View+Group+Memberships (browse/query Crowd group memberships)
https://collaborate.linaro.org/display/IKB/View+Linaro+Login+Groups (browse/query Crowd groups)
https://collaborate.linaro.org/display/IKB/Managing+Groups+on+Linaro+Login (docs)
ITS wiki: https://collaborate.linaro.org/display/IKB/ITS+Knowledge+Base
To submit a request, go to https://servicedesk.linaro.org or send email to it-support@linaro.org.
Specs, HowTo's & Setup Information
Configuration Management - SCM - Ansible
Linaro Image Tools
Tools and Scripts to manipulate and install Linaro Images/Releases on media.
Hardware Packs
EC2 Usage and Best Practices
Few of our systems are based on/interact with EC2. We maintain list of best practices, tools, and known issues for such systems.
EC2 Best Practices and Known Issues - includes Jenkins master management practices.
Main Ubuntu package archive - may be useful to investigate Ubuntu EC2 package archive issues, like missing package in EC2 archive
linaro-ami tool to manage custom AMIs as used by various build services
Continuous Integration Service
Common to all CI services
ci.linaro.org
The Continuous Integration Jenkins Build/Test Service is an instance of Jenkins that uses slaves provisioned from EC2. The following link is mostly a guide on how the service was set up and how to create a new job on ci.linaro.org.
CI Tools
Android-related Infrastructure
Android Build System - Android CI Loop
Cloud Buildd is a generic platform to perform dynamic, parametrized builds in the Cloud (EC2), consisting of Jenkins, frontend for it, build scripts and "repo" tool mirror service. The Android Buildd is specific installation instance of Cloud Buildd, used to perform Linaro Android builds.
Original spec (historical)
Jenkins "Execute Shell With Build Status" plugin to separate infra from build failures
Android Build Downloads
All downloads are now available from http://snapshots.linaro.org/android/ . Direct downloads from Jenkins (starting with https://android-build.linaro.org/jenkins/ are internal-only and deprecated for general usage). For the cases which involve "permalinks" like lastSuccessful, links to http://android-build.linaro.org/download/ should be used for now (note http://)
"Seeded" builds approach
Linaro Android Gerrit
Gerrit is used for git code hosting and review by Linaro Android team.
Android Gerrit Continuous Integration
Joint project with Android team (led by them).
Android Gerrit Mirroring System
patches.linaro.org (patchmetrics)
patches.linaro.org is deployed in EC2 and managed by ITS.
Procedure for stakeholders to request addition of a new project
(Info for Infrastructure Team at the end of the same page)
releases.linaro.org , snapshots.linaro.org and License Protection Handling
They provide click-through licensing which is part of lp:linaro-license-protection project.
- Details on the set-up are in:
We used to have integration tests to run after each rollout, but they were not updated for latest changes (TODO: fix them): testr run testplans.test_suite
Publishing new builds
Please read Publishing page for adding new builds (includes intro on BUILD-INFO.txt).
Accessing license-protected files in automated environments
The latest and greatest tool to handle downloading of license-protected files: lp:linaro-license-protection/download. Requirements on which it is based: lp:1130810.
repo.linaro.org
This is deployed on an EC2 instance. (ec2-54-224-40-137.compute-1.amazonaws.com)
It contains reprepro repositories in /srv/repos/<subject>
Currently one repo: ubuntu/linaro-overlay, which contains two suites
- quantal (for armhf and arm64 overaly packages)
- quantal-tools (for tools/packages to be using in builds, not yet upstream (amd64, armhf)
The repo key for apt authentication is http://repo.linaro.org/ubuntu/linarorepo.key
Jenkins server Local setup HowTo
Get the source code here.
For building, follow this link. If the build fails with this exception download and use Sun Java.
To configure environment in eclipse run:
mvn -DdownloadSources=true eclipse:eclipse
and then follow this set of instructions.
For deployment you can use any of the existing web containers (instructions listed here)
If you're trying to deploy it under tomcat and get this error, switch to running Jenkins by itself (which uses Winstone.):
java -jar jenkins.war
Setting up Jenkins development environment
The codebase for Jenkins for Linaro is located here. At the moment of writing the Jenkins version we use is 1.456
After branching this code, one should do:
mvn -Plight-test clean install mvn eclipse:eclipse
if the aim is to develop code under Eclipse (recommended). Importing project to Eclipse should be straight forward.If jenkins-remoting library is marked for particular bug-fixing/development, the code should be used from here: https://code.launchpad.net/~linaro-infrastructure/jenkins-remoting/linaro-trunk
- The process for import is pretty much the same except for the first time maven build when this library source should be moved to jenkins code base due to some strange pom dependencies.
- After building jenkins-remoting or some other library used by jenkins, be sure to build jenkins code as well, since the jenkins war has to pick up the new version of the lib from the maven repository.
Deploying Jenkins to ci.l.o
- Get you jenkins.war file from war/target dir in your local jenkins build directory and copy it to ci.l.o.
Login to ci.l.o shell and execute following steps:
sudo service jenkins stop sudo rm -rf /var/run/jenkins/war/* sudo cp jenkins.war /usr/share/jenkins/ sudo service jenkins start
- Optionally you can create a backup of old jenkins.war beforehand in case something goes wrong and you want to revert quickly.
- After that check the /var/run/jenkins/war/ folder, it should contain unpacked jenkins war file.
CBuild/LAVA integration
Git & Gerrit servers and instructions
Git Problems: instructions on recovering from common problems in production
Git Server: user instructions on how to use the new git server
Code Review Server: user instructions on how to use the code review server
Gerrit deployment instructions, and gitweb+gitolite integration
Private code hosting implementations
Admin helps: erros, LDAP/Crowd integration, gerrit integration, error & how-tos...
User help pages:
Toolchain Mirror
Toolchains Mirror: instructions on how the gcc mirror has been setup and how to add new branches
OpenJDK Mirror
OpenJDK Mirror: instruction on how the OpenJDK mirrors have been setup
bugs.linaro.org
Code is hosted here: https://git.linaro.org/infrastructure/bugs.linaro.org.git
Ansible playbook is here: https://git.linaro.org/infrastructure/linaro-bugzilla-tools.git
Austin Colo Facility
Hetzner
Platform/Systems/KnowledgeBase (last modified 2016-06-20 19:59:20)