Detailed spec and design document for https://blueprints.launchpad.net/linaro-ci/+spec/switch-ci-to-use-custom-ami

Stakeholder services

  • ci.linaro.org
  • android-build.linaro.org
  • CBuild

Requirements

  1. Provide consistent process to create, update, and otherwise manage custom AMIs for use with EC2 build systems.
  2. Reuse existing slave init scripts from the stakeholder services as much as possible, preferrably requiring minimal or no changes to them.
  3. Should be streamlined implementation, easy to maintain and debug, with main purpose being upholding consistency per p.1 rather than hiding complex algorithms.

Issues/Things to decide

  1. It turns that there're 2 types of AMIs: instance store based and EBS based. EBS were introduced later, and provide streamlined way to create/update AMI: an instance can be started from existing AMI, new packages can be installed/other changes made to it, instance stopped, and new AMI created from it. For inststore-based AMIs, process is more involved, requiring manual mounting of loopback filesystem image, making changes, converting to special format, uploading to S3, etc. Per requirement 3, EBS AMIs are the way to go, and they are already used on ci.linaro.org (mixed with inst-store based ones), while android-build uses inst-store exclusively. So, first step would be testing EBS AMIs and switching to them consistently.
  2. Even with requirement 2, some changes to existing slave init scripts may be required - after all, they were written with specific usage/environment (Jenkins/CBuild) in mind. Moreover, some slave types on ci.linaro.org don't take init script from bzr, but instead have it inline in Jenkins, so those will need to be moved to bzr either.
  3. Need to separate build slave AMIs from other AMIs which may be hosted in Linaro account. Tagging? Naming convention?
  4. Some kind of naming convention would be helpful anyway, to encode to which build system AMI belongs, OS release, etc.
  5. Additional metadata?
  6. The tool will need more or less extensive config format (essentially, map AMI template name to base AMI id, slave init script, etc.), what format to use: JSON, Python ConfigParser, something else?

Meeting minutes pfalcon vs stevanr on 15.06.

Work process discussion

Technical specs

  • How many diferrent amis will we have?
    Discuss with fathi why do we have 3 different precise slave images on ci.*
    Basically we do want one ami per ubuntu version and will strive towards this goal.

  • Remove/reduce init scripts once we switch to new amis. We agree that we will have to update init script, removing most of the code from them, leaving authentication and apt upgrades.
  • Question to discuss later: how often do we want to recreate amis?

Modes of operation and their algorithms

Create

  1. Start instance from known base AMI.
  2. Log into instance.
  3. Checkout defined repository and run defined slave init script from it.
  4. Stop instance.
  5. Create AMI from instance with a given name.
  6. Terminate instance.

Update

  1. Backup current AMI.
  2. Follow "Create" algorithm.

List

  1. Query AMI list
  2. Output list filter by build slave AMI.

Delete

  1. Delete given AMI.

Info

  1. Output additional info (e.g. metadata) about given AMI.

Implementation Details

Implementation Language/Libraries

AMI management tool is intended to be simple wrapper around standard EC2 operations, and might be prototyped in shell, but it's already clear that it will need to do extensive config parsing/string manipulation, plus will need to have full-fledged testsuite, so that stage can be skipped and implementation started in Python from the beginning.

With Python, there're 2 choices: call standard EC2 tools (package ec2-api-tools), or use native Boto library. Boto allows for better integration and potentially less dependencies, but there are 2 risks: 1) Boto may lack some functionality required for the tool; 2) Boto version shipped with Ubuntu are known to be rather old, so we'll additional steps to install recent version from PyPi. still, it is recommended to go with Boto 2.0, which is already used by other tools in linaro-aws-tools.

Testing

The functionality of the tool will be largely based on EC2 operations, so we should be prepared to do integrational testing. Some level of unit testing will be obviously useful too, should be prepared to extensive mock object usage. Proposed tools are Python modules nosetests and mock.

Worklogs

pfalcon, creating AMI for a-b manually

  1. ami=ami-87c31aee
    keypair=pfalcon@home
    instance_type=m1.medium
    zone=us-east-1b
    security_groups="-g default"
    
    ec2-run-instances $ami -k $keypair -t $instance_type --availability-zone=$zone $security_groups
    • got i-dae678a3
  2. ec2-describe-instances i-dae678a3
    • got ec2-184-73-86-220.compute-1.amazonaws.com
  3. ssh ubuntu@ec2-184-73-86-220.compute-1.amazonaws.com

  4. sudo apt-get install bzr
  5. bzr clone lp:linaro-android-build-tools
  6. cd linaro-android-build-tools/node/
  7. sudo bash
  8. bash setup-build-android
    • Works as is pretty well
  9. exit
  10. cd
  11. rm -rf linaro-android-build-tools/
  12. exit
  13. ec2-stop-instances i-dae678a3
  14. ec2-create-image i-dae678a3 --name ab-natty-x86_64 --description "pfalcon's manual test custom AMI"
    • Finished asynchronously, actual AMI preparation takes few mins.

stevanr, changing ci.l.o slave AMI ID's

  • ami-08f40561(ec2cloud) => ami-d78f57be

  • ami-68ad5201(kernel_cloud) => ami-87c31aee

  • ami-68ad5201(android_cloud) => ami-87c31aee

  • ami-baba68d3(oneiric) => ami-baba68d3 (no change, already ebs)

  • ami-3c994355(precise_cloud) => ami-a29943cb

  • ami-a29943cb(precise_hwpack_cloud) => ami-a29943cb (no change, already ebs)

  • ami-a29943cb(precise_kernel_cloud) => ami-a29943cb (no change, already ebs)

Platform/Systems/Spec/CustomAMIs (last modified 2014-06-24 17:47:47)