Release Review: 2011.11

  • Created: David Zinman <david.zinman AT linaro DOT org>

    Past reports

Release Highlights

  • Include any major updates, good, bad news and issues.
  • Include highlights and lowlights.
  • Things to improve.
  • Include relevant links.

Android

Highlights

  • Good publicity. ELC Europe, Connect and a number of videos on the Internet.
  • A smooth transition to Android 2.3.7 (That became a pseudoevent when ICS came out)
  • A preliminary release on Thanksgiving with ICS on all major boards. Amazing.

Issues

  • A build system that couldn't sync the code base because of the expanded size of ICS in the git mirror.
  • A cross git change that had to be reverted.

Lessons Learned

  • Gerrit doesn't handle cross git changes so it should be extended to do that.
  • We need to ensure that each target in LAVA is stable.
  • If things are starting to get chaotic then anyone can yell "stopthepresses!" in #linaro-android and we'll all stop and regroup.

Developer Platform

Highlights

  • Finished milestone before deadline
    • cleaned up the BPs before release
    • bugs much better, and cleaned up
  • This cycle went well, not as much work this cycle as others
    • Work items went pretty smoothly.
  • The process works well and still getting refined
    • managed to release in time
  • Next cycle planning started
  • Good coverage for tests even with lack of testers
  • No respin was required for Ubuntu desktop
  • Good feedback from most of the Landing Teams responding to issues with one exception

Issues

  • Linux tools for Landing team kernel are still not available
    • bug has been created
  • Lack of testers (as always)
  • Need a snowball asap
  • Too many respins this cycle from the landing teams
  • There is no integration or smoke test from LTs
  • Short month (again)
    • It is difficult to fit normal sized work packages into these short months
  • The Samsung LT was not responsive to queries from the Dev team
  • LEB releases using the landing team hwpacks:
    • Angus wanted to use the LT hwpack for the LT's own development and testing, without interfering with the LEB releases. The idea is to create the LEB hwpack, that would be maintained by both the LT and the dev platform.
    • This is needed because then only the LEB hwpack will be used for the releases, without affecting the LT development.

Lessons Learned

  • Early planning, don't add too many BPs to the cycle on a short month
  • Need validation and/or smoke test from LTs before making the kernel availables for RC
  • If Connect occurs during cycle, planning must be done during Connect to mitigate short development cycle
  • Need a backlog review with TSC and the team as required to keep backlog manageable.
  • If a POC is not available, raise the issue to his PM and the Release team in order to escalate. (re: Samsung team)

Graphics

Highlights

  • glmark2 was completed in time with a range of new tests. Also we received good feedback from users of glcompbench, who are excited about possible extensions
  • As usual the team was quite responsive, overall. No regressions, no surprises, release went smoothly and as planned.
  • Got good progress in getting graphics benchmarks running daily on LAVA
  • Planning for 11.12 was very smooth, 90% of the blueprints were already in the backlog

Issues

  • Some non-responsiveness from 1 member caused a bit of worry and distress on a personal and release status - related to possible changes coming in at the last minute
  • Short month (again) - it is difficult to manage doind much work if the month is so short
  • General issue: in limited time periods like 11.11 dealing with Rypple can be hard to achieve at least for some. We should think carefully the use model and incentive provided for its usage in order to strike a good balance. Otherwise it can become a hindrance.

Lessons Learned

  • Early signs of trouble should be dealt with immediately - shadow a team member's work, or take over, if at all possible. At least the quality assurance should be there, if someone is not around due to illness or other issue, then we should provide coverage for any possible bugs, whilst freezing any new released features for later. This can cause some changes, but if reaction is fast then we have a good chance to avoid late surprises.
    • In 11.12 we will try some developers pairing for glcompbench release activities
  • In a short month perhaps it is best to work on bugs and overall quality assurance, or to enhance testing. We should limit the new functionality, if possible, and be especially careful with features which can prove hard to resolve at a short time

Kernel

Highlights

Issues

Lessons Learned

Multimedia

Highlights

  • Delivered all the code promised and smoothly in terms of observing the deadlines - even exceeded expectations in some cases (UMM work for 11.12 was almost complete around release time for 11.11 which prompted taking more tasks for 11.12, also community involved folks released ahead of time)
  • First time integration with LAVA, for LJT
  • Upstream patches were through smoothly
  • 11.12 planning was smoother, though still backlog management relied on 1 bottleneck getting the blueprints created

Issues

  • Need to make LAVA benchmarking part of the primary focus for each optimised codec, not an afterthought. Not just for benchmarking (this is a natural start though) but also for functional and regression testing
  • Had some issues with snowballs for CMA testing which delayed our LAVA experiment for testing CMA through debugfs/tracing. This should be ok now
  • There was a problem with the version chosen for speex, this could have been easily resolved with earlier and tighter communication about the integration of speex
  • We encountered an issue with a BP which needed to be superseded by a second BP from outside MMedia projects (part of another team's projects). LP does not give as option of "Superseded by" blueprints which are not part of the same project - so such cross-project superseding of BPs is an issue (solution chosen: changed the title of the superseded BP to include the string "(SUPERSEDED)" and added a link to the superseding BP)

Lessons Learned

  • LAVA should become a focus, maybe we need some dedicated resource to improve our usage of LAVA
  • Backlog management should be more intensive and frequent, and involve more than 1 person per project

OCTO

Highlights

Issues

Lessons Learned

Power Management

Highlights

Issues

Lessons Learned

Toolchain

Highlights

Issues

Lessons Learned

Validation

Highlights

  • Upgrades and reconfiguration of networking and server gave us a nice performance boost.
  • LAVA UI improvements. Steps on the right way.
  • More documentation added.
  • Good discussions and training at the Linaro Connect Q4.11.
  • Good progress on the flexible LAVA deployment.

Issues

Lessons Learned

  • Blueprints should be worked at Linaro Connect. PM can start to work on them at the hacking sessions.
  • Cross team blueprints requires a kick off and regular sync meeting to avoid duplicated work.
  • Tutorials/training at Connect helps team members to ramp up. We want more.

Infrastructure

Highlights

  • l-i-t released with not too many issues
    • documentation was good
  • Short milestone with first week in Orlando but the short time was handled well.
    • good with the changes that were made with Infra
  • Integrated 2.3.7, ICS (Icecream Sandwich) into android-build
  • Seeded builds have been put into production for android
  • ci.linaro.org more reliable at pulling from git.linaro.org

Issues

  • Package building on l-i-t did not work due to some dependencies
    • fixed by salgado or fabo
  • Index file for released.linaro.org not generated automatically
    • cycle information was not added
    • once added, the dependency was not available
      • ACTION: Danilo to look into splitting up indexer and .yaml file out of l-i-t, and file a bug if appropriate
  • status.linaro.org needs a new config file for new release, and then we need to update apache config for it
    • ACTION: Danilo to look with IS into how we can simplify this

Lessons Learned

  • knowledge to build the packages should be available in the infrastructure team
  • ensure deployment on mombin is up to date
  • a BP with a checklist will be used to track infra release tasks for each cycle
    • ACTION: dzin to create
  • Use IS for IS stuff - do not try to duplicate work that IS can do.

Release

Highlights

  • ST-Ericsson Landing Team was very helpful during release. Thanks for the support!

Issues

  • LT communicates directly with Platform TLs about release related schedule or issues. Release Team isn't in the loop as it supposed to be and documented in the components release process.
  • Too much respins.
  • Reminder announcements were late.

Lessons Learned

  • Communication channel between Release Team and Landing Teams should be improved.

ST-Ericsson Landing Team

  • No real release this month:
    • First week we had Connect - wholey meetings with ST-Ericcson PMs
    • Second week we were at ST-Ericsson & Movial

    • Third week LT Lead (lag: who usually releases) was on vacation
      • Contingency was in place (mpoirer), but kernel release was low priority - high priority was Ice Cream Sandwich (ICS) enablement
  • This lead to:
    • Lack of testing (smoke or otherwise) before and after the release
    • Multiple (avoidable) re-spins
  • Not much to learn from this, it was an unusual comedy of errors - hasn't happened before, unlikley to happen again

ARM Landing Team

  • Updated kernel and released updated A9 hwpack
  • Tixy worked with Android to get 2.3 working
  • Tixy swapped to Android 4.0, it's not compiling yet
  • Issue with devirtualised PPA is outstanding, but in progress
  • Ricardo is still handling the DS-5 PPA for us

Samsung Landing Team

  • Release went smoothly for the most part.
  • For the monthly releases, it is recommended not to modify the lt- branded hardware packs. If the platform team needs to modify the release hwpack they can create an leb- hwpack and modify that one instead.

Freescale Landing Team

  • Release went smoothly. Good support to do a re-spin.
  • Good team work on debugging issues such as the DS-5 driver issue. Although collaboration can be improved there was good spirit from both platform and landing team to resolve the issues.
  • Downloading speed from snapshots/releases is very slow. As a result downloading and testing took longer than expected.

Key Points for wider discussion

Cycles/1111/Release/Review (last modified 2011-11-30 14:58:25)