• Launchpad Entry:

  • Created: 2010-07-02

  • Contributors: JamesWestby

Summary

We wish to provide an archive hosting solution that can be deployed internally for projects that can't be publicly released yet, and could be used for public hosting too.

Rationale

Many partners need to enable pre-release hardware, which involves modifications to packages, including, but not limited to the kernel. The details of the modifications can't be made public until the hardware is, as they will often reveal too much information.

When they are working on the modifications they need somewhere to host the packages, for sharing their work with other people, and for getting them to the person/system/service that builds the images.

Therefore they need archive hosting/management software that they can deploy internally.

User stories

  • A Linaro partner takes the open-source code provided by Linaro and deploys it internally on their Ubuntu Lucid system in under an hour.
  • The partner wishes to enable a new hardware platform, so they create a new distribution/archive in the hosting software for that platform. This takes them 3 or 4 clicks. This does not require deploying a new instance for each distribution.
  • This new distribution is visible only to a certain set of people, and only a subset of that group can upload. It's easy to see who is in each of those groups on a single page, at most 1 click from the overview page of the distribution. It is also simple to modify each of the groups, with at most 3 clicks required to add or remove someone from one of the groups.
  • Meg, who works for a partner enabling pre-release hardware, wishes to build an image containing some modified packages. She uploads the modified packages to the archive for that hardware project, and can then build images containing the modified packages. Others who are working on the project can then build on top of this work.
  • Andy is a new developer hired in to the partner. He is given access to the archive for the project that he is working on. He is able to find his way around the software and find the things that he needs to without too much trouble.
  • Lou is the lead developer on the project. He goes to a web page and sees the current state of the archive, and is able to remove some obsolete packages. This can be done from a single page, and is at most three clicks to remove a package. Bulk operation should be possible.
  • Joh is a developer on a second project, targeting hardware with a new ABI, where there are only slow emulators in existence so far. She wishes to cross-build packages for the new ABI, either locally on or a build farm.

Assumptions

  • We don't have to deal with building rpms at this point.
  • We will be basing our work on Launchpad rather than some other approach.

  • We don't require fine-grained upload permissions at this point. Granting a person the ability to upload to an entire archive is sufficient.

Requirements

  • Open source code.
  • Deployable inside partners.
  • Deployment and ongoing maintenance by partners that doesn't require a large time investment from us, as we can't scale that. Should be a couple of hours per-partner per-month.
  • Ability to create new archives for new projects in the web UI, in a few clicks.
  • Access control on the new archive, such that a limited set of people can upload/modify, and optionally a limited set of people can view.
  • Ability to easily see which people are in each of those groups.
  • Accepting of packages in to archives from uploads.
  • Publishing of the archives.
  • UI that is well suited to the task, and not hidden among other concerns.
  • Ability to remove packages via the web UI, in at most a few clicks. Optionally bulk removal of packages should be possible.
  • Ability to either accept cross-built binary packages, or cross-build them from source in the build farm.

Non requirements:

  • Cross-platform code, or even anything beyond working with a known set of dependencies on the latest Ubuntu LTS.
  • Easily running up a new architecture. This is desired, but is out of scope for this iteration.

Design

  • UI

  • Deployment
  • Creating new distributions easily.

We will be basing our work on Launchpad, but there are several ways in which it currently falls short of our goals:

  • Deployment is perhaps harder than we would like, or perhaps just less documented.
  • Creating a new distribution that actually gets published can't be done from the web UI.
  • Its UI currently contains too many things unrelated to archive management, which will make the process overly-confusing.
  • Launchpad doesn't currently easily support cross-building in the build farm, and doesn't allow binary uploads.

Deployment

The current process for getting Launchpad deployed on Ubuntu server is documented at /DeployingLaunchpad

We will provide instructions for installing vostok on a stock Ubuntu server image.

In addition, we will also provide configs that will only require light modification to get vostok up and running. The development configs that are shipped in the tree are not suitable for this, and so others must be provided. In addition, an example apache configuration is needed for proxying on to the zope appserver.

Launchpad has a lot of cronscripts that take care of periodic tasks, but many of them will not be needed by vostok, and so we should evaluate which are needed and only document how to set those ones up.

Launchpad has the concept of "admins" who can do anything, and without them some tasks are impossible. We will provide a script to create a team in the database, make it an admin, and add a specific user to it, to allow people to bootstrap their installation.

Currently Launchpad assumes that it is using the Canonical SSO provider as the openid provider for logging in. This may not be true in other circumstances, and so we want to remove this dependency.

Launchpad uses metapackages for some of its dependencies, and we will create a vostok-dependencies package in a similar vein that will only depend on the parts needed for deploying vostok.

TODO: which changes do we want to make to simplify deployment?

UI

We want to be able to have a UI targetted to archive management.

We will provide a different UI to the Launchpad functionality.

#TODO: design the UI

Cross-building

Cross-building either requires a build-farm that can cross-build, or developers doing it locally and uploading binaries. There may be some partner-preference here, and it also depends on the cross-building strategy for Linaro.

Firstly we can assume some things:

  • We will have a cross-toolchain available.
  • That toolchain will be available in a package.

Secondly look at what each solution would require:

  • Cross-building build farm
    • QEMU
      • - We fake it by having "armel" builders that are x86 machines running QEMU.
    • Cross-compilation
      • - We have LP know to cross-compile some architectures, so it will dispatch the build to an x86 machine, that will invoke the build with "-a armel" to cross-build. This requires arranging to have the cross-toolchain installed too.
  • Developer builds
    • Ability to accept binaries, arrange for them to be published, skipping the creation of build records, and with appropriate checks.
    • Policy on whether an archive can accept binaries.

Thirdly how much work that would mean based on the current state:

  • Cross-building build farm
    • QEMU
      • - ?
    • Cross-compilation
      • - Maybe requires some sbuild modifications. - Requires LP to understand cross-compilation and have a policy around when to dispatch in that fashion.
  • Developer builds
    • Change the uploadprocessor to accept binaries from arbitrary uploads. There may already be code for doing it from builders.
    • Skip the creation of build records when binaries are present.
    • Have checks of the binaries, and still require source.
    • Add policy support for this, and ensure that it remains the same for the existing archives on launchpad.net.
    • Auditing the code for assumptions around this.

Therefore we will start by focusing on allowing binaries to be uploaded, as this is much simpler solution, and requires much less infrastructure to set up.

TODO: what changes do we need to make for this?

Implementation

UI

We are building on top of Launchpad, but providing a new UI.

The way we do this is by using a different vhost which maps to a different layer. This layer uses a separate set of templates, which allow us to create the UI we want. This is the same approach as is used for shipit.ubuntu.com, so it is tried and tested.

We might re-use Launchpad view code, and some "pagelets" that form part of pages, but we will have our own templates.

Cross building

Code Changes

Code changes should include an overview of what needs to change, and in some cases even the specific details.

Test/Demo Plan

Unresolved issues


CategorySpec

internal/archive/Platform/Infrastructure/Specs/PrivateArchiveHostingInfrastructure (last modified 2013-08-23 02:14:57)