OverallStatusReporting

Summary

We wish to provide better high-level status information about the progress of the Linaro project towards its goals for the cycle.

Rationale

While much of the information is available about the status of individual specs, tying this all together to get summary progress, and matching this against the goals set out be the TSC is an involved process.

Stakeholders

  • Jamie Bennett
  • Anmar Oueja
  • David Rusling
  • George Grey
  • TSC Members
  • StephenDoel

Contact

  • Jamie Bennett

User Stories

  • A TSC member wishes to see progress against one of the TechnicalRequirements for the cycle. They go to a summary page and can see a percentage completion and what the recent activity has been. If they wish they can delve deeper in to the progress of individual items, and find out who is responsible for delivery.

  • A Linaro engineer can see what work items are assigned to them, and when they are targeted, and can plan their work accordingly.
  • A team lead can see the work items assigned to each member of the team, and so judge whether one particular person is overloaded. They can also see the progress being made across the team.

Requirements

The following information is desired:

  • Work item percent completion (TODO: (DONE + POSTPONED)/(TOTAL)?) to show current state.
  • Burn-down chart of all Linaro workitems to show progress.
  • Total workitems in each state.
  • For each team the % completion.
  • For each team, each blueprint they are working on, along with their status, workitem summary and assignee.
  • Bugs (TODO: what is this supposed to show?)
    • Any relavent bugs in Linaro projects (filed against a set of projects that Linaro is interested or has a specific tag Linaro is interested in).
      • Yes, but show what? Just the number of open bugs over time?
    • Bugs are out of scope for the first round of implementation, which will focus on technical requirements and workitems.

It is desirable that a time period can be selected (TODO: what would changing the time period change?)

  • Only data from that period would be show, i.e. 1 months of burndown chart/work items, 1 week of bugs, 1 cycle of engineering effort.
    • You mean things like only showing workitems that are targeted to a milestone within that time period?
  • Time period selection is out of scope for the first round of implementation.

The information needs to be linked back to TechnicalRequirements.

  • Be able to show the status of a TechnicalRequirement as a summary of the individual blueprint statuses.

  • Be able to show whether the priority has changed since UDS.
    • This is out of of scope for the first round of implementation.
  • Be able to show which requirements were introduced after UDS.
    • This is out of of scope for the first round of implementation.

Original notes

Design

The landing page is a summary page for the whole of Linaro.

On this page there are visualisations of the current state and trend of progress.

This includes:

  • A "thermometer" showing raw progress towards completion of all work items, with sections for each status within.
  • A burndown chart showing progress towards the completion of all work items.
  • A stacked bar chart, with one bar for each team so that the relative progress of each team is visible. The bar is made up of the work items in each status.
    • We may want to do something like using negative values for the uncompleted work items, so that the relative progress is more obvious to the eye, when accounting for the different number of work items and relative completion percentages.
  • A summary table of progress towards each TR, showing its status, and the totals of the work items for each contributing blueprint.
    • This table should be split based on the team assigned to the TR.
    • Each TR should show below it the status of the contributing blueprints. This will show status, work item counts and assignee.
    • Each team should have a link to their team-specific page.

This satisfies some of the first user story, as a TSC member can see the progress of TRs on this page, and can see which blueprints are contributing to that TR, the progress on the blueprint, and who is assigned.

The team-specific pages are similar to the landing page, but with the data limited to show things relevant to that team.

  • The thermometer will only count work items on blueprints related to a TR assigned to the team, or blueprints assigned to members of the team.
  • The burndown chart will only include work items on those same blueprints.
  • The bar chart with the per-team information now includes per-member information.
  • The summary table is the same, but only including TRs assigned to that team. It will include a section at the bottom for blueprints not associated with a TR.
  • There will also be the work item tables on the current team pages on the work item tracker, that shows the status of work items for each person, across all blueprints.

This satisfies the third user story, as the team lead can see what is assigned to each team member.

For each person there will then be a person specific page, this will be similar again, but will have:

  • A thermometer showing the percentage of work items assigned to the person completed.
  • The burndown chart that includes work items assigned to them.
  • There could be a bar chart with per-blueprint information, but we will leave that out for now.
  • The summary table will include TRs that the person is contributing work items for. The counts will only include their work items. There will still be per-blueprint information.
  • The work items table will also be present.

Mock-up by JamieBennett showing some of these concepts.

Linaro Status Mock-up.jpg

Milestone Handling

Milestone start dates

Linaro blueprints are tracked in many projects, each of which can have its own milestones.

Each workitem can be targetted for a specific milestone, and we know that it should be completed prior to that milestone.

This means that for any milestone we know the list of workitems that are expected to be completed for that milestone, and so can record the counts of those workitems in each status at each point leading up to that milestone.

From this we can draw a graph of those counts over time, with the end of the graph being the delivery date of the milestone.

For this to be a full burndown chart we need to know the "start date" of each milestone, as well as its end date. This is the date at which engineers will start to concentrate on that milestone. If we choose an earlier date then the trend line will be too shallow to be of use.

Currently the code picks the delivery date of the previous milestone as the start date of the next (where next is defined by the chronological order).

However, when there are multiple projects are involved this can't work the same way, as some milestones don't make sense for some projects.

We can solve this by having milestones start on the day that the previous milestone in the same project completes.

This will give similar behaviour to before, and allows project leaders to control the start dates of particular milestones by adding another milestone at the start date that they want.

There are two cases that are not handled that well here:

  • Teams who span projects.
    • They won't have burndown charts that support a workflow of concentrating on the next milestone chronologically, as there may be multiple milestones "burning down" for them at any one time. They can alleviate this by registering their milestones in all projects that they work on, whether they make sense or not. This is an ugly but functional solution.
  • Projects with multiple milestone tracks
    • There may be projects where there are two threads in progress, and the proposed solution would mean that these threads were not tracked separately. Consider two milestones that are one day apart, but worked on by separate subteams. The proposed solution would mean that one subteam had a useful burndown, but the second team would have a burndown that ran for one day, from the earlier milestone to the later. I predict that this is rare occurence, and one that could perhaps be handled with series, but dealing with it is out of scope at this time.

Equivalent milestones

Another issue with milestone handling is milestones that are separate, but delivered on the same day, and considered as one. For instance, we currently have a "natty-alpha-1" milestone in Ubuntu, and an "11.05-alpha-1" milestone in linaro for the 11.05 series. These are logically one milestone, as they are on the same day, but the code will consider them differently. There are two solutions to this issue:

  • Give the milestones the same name. If the Linaro milestone was also named "natty-alpha-1" then the code would coalesce them. This isn't really an acceptable solution for some milestones though. For instance the beta milestones, which Ubuntu names "ubuntu-11.04-beta" and Linaro "11-05-beta".
  • Provide a way to collapse milestones that are due on the same day. When showing the page for a milestone we could include all workitems targeted to milestones that fall on the same day.
    • The naïve solution to this that generates a page for each milestone that falls on the same day would work.
    • This could be improved slightly by not generating Linaro milestone pages for non-linaro teams, but we would ideally want to do that with no configuration.

We will use the second option.

There may be groups that do not like this approach as they consider the equivalent milestones to be separate, and don't want to track them together, despite them having to deliver all the work on the same day. If that is the case then we will have to look at a modification to this solution that accommodates that desire, or convince those teams to use the proposed approach.

Only generating pages that teams expect

There are two cases where we will generate too many pages:

  • Ubuntu teams that don't have anything to do with Linaro will still find a page generated for their team for each linaro milestone.
  • Equivalent milestones will still generate a page for each constituent milestone, each showing the same information.

For the first we would need to decide whether a team had any interest in a milestone whatsoever (The fact that a linaro team has no workitems targeted for a Linaro milestone is still interesting).

The second is trickier as we have to decide for each team which milestone name is more appropriate for them. For Ubuntu teams it is the Ubuntu name, and for Linaro teams it is the Linaro one. It would be nice to do this without explicit configuration.

We currently have no solutions to these two issues, but they are not pressing problems so we will live with the issues for now.

Linked bugs

Blueprints can be linked to bugs, and we want to allow people to use that to indicate workitems.

However, it is bugs that are linked to blueprints, and there can be many bug tasks on each bug, so it's not clear which of them should be workitems for Linaro to do.

To decide which bug tasks we should create workitems for:

  • Get a list of all projects that we are scraping blueprints from
  • For each bug linked to a workitem:
    • If there are no bug tasks against any of the projects in the first step then make one workitem, which is TODO as long as any of the tasks are open.
    • If there is are bug tasks against the projects in the first step then make a workitem for each of those bug tasks, with the status of the workitem reflecting the status of the bug task.

Implementation

Data requirements

In order to be able to implement the above we need the following information:

  • The list of linaro teams and their members.
  • The list of technical requirements for the cycle, the teams that they are assigned to, and their status and priority.
  • The list of blueprints that are contributing to each technical requirement and their status, priority, and assignees.
  • The list of blueprints assigned to a linaro engineer and their status and priority.
  • The list of blueprints that a linaro engineer has a work item for, their assignee and their status and priority.
  • The work items for each of those blueprints, along with their status and which blueprint they are for.
  • The history of status changes to those work items.

Data gathering

  • We can get the list of linaro teams and their members from an explicit list in the tracker config.
  • The list of technical requirements can be found by looking for "tr-" blueprints in the linaro project.
    • We should use the series goal before the start of the next series, but we don't need to deal with that yet.
    • The assigned team can be parsed from the blueprint name, but will need a mapping to the launchpad team.
      • TODO: do we want to use the assignee field instead?
    • Their status can be easily parsed.
  • To get the list of blueprints contributing to each TR we need to examine the dependencies of each TR.
    • Status and assignee are easy to get.
  • With the current model of the tracker we need a list of projects to consider blueprints on, and can then easily find blueprints and work items assigned to linaro engineers.
  • The workitems for blueprints are already parsed.
  • The history is already tracked.

Data presentation

Overall page:

  • The thermometer can be drawn by considering the current status of all the workitems assigned to a linaro engineer, or part of a blueprint assigned to a linaro engineer, or part of a blueprint contributing to a TR, taking care to avoid duplicates.
    • The thermometer is a block for each status, with the length of the block proportional to the number of work items in that status. Blocks for incomplete statuses are coloured dark, complete statuses light, so that the complete/incomplete ratio can be easily seen, but e.g. the postponed count is still evident.
      • A key should be provided to interpret the colours.
  • The burndown chart should be created as now.
  • The stacked bar chart should be created similarly to the thermometer, but split by team.
    • - See the note above about presenting this such that the relative completion percentages are obvious even in the face of different workloads.
  • The summary table should be created by iterating over the list of teams, and for each showing the status of the TRs that are assigned to them, sorted by status.
    • Each TR should show the work item completion counts, by summing over the work items for the contributing blueprints.
    • Under each TR the contributing blueprints should be shown, sorted by priority, showing their status, assignee and work item counts.
    • There should be a section at the bottom for all blueprints assigned to members of the team that are not contributing to a TR.

The other pages work similarly.

Milestone Handling

Milestones should become associated with a particular project, but remain accessible as workitem targets from other projects.

When searching for the start date of a milestone, the code will look for the previous milestone in the same project.

When searching for the workitems targeted to a milestone the code will look for workitems targeted to any milestone that has the same date as the milestone in question.

Code Changes

Data gathering

  • We need to add fetching of TR blueprints to the collect script. This can be done by matching on the name. We should try and find a way to make this not too linaro-specific.
  • We need to examine multiple projects for blueprints. The configuration should list the projects to examine.
  • We need to associate technical requirements with the blueprints that implement them. For this we need to look at the dependencies of the TRs, which we will need to use the API for this.
  • We need a way to associate TRs with the team that will implement them. If we decide not to go with the assignee field then we need to add a mapping to the configuration.

Data presentation

  • TODO

Other approaches

BIRT

One possibility is to use the Business Intelligence and Reporting Tools (BIRT) project to generate these reports.

This was discounted, as it would take a while to get the reporting service running, and we would have to create the reports and probably make code changes to allow us to gather the needed data. It was estimated that this would take longer than modifying the existing infrastructure. We may want to revisit BIRT at a later date, as it would provide more flexibility in report creation.

Design

BIRT provides a report designer and a report viewer. The first is used as part of Eclipse, and the second is run inside Tomcat on a server.

BIRT works by querying databases and displaying custom reports based on the output.

There are three areas of work that would be needed:

Database creation

We need to create a database with information on the current status of blueprints, workitems etc.

This database needs to contain:

  • blueprints and their metadata (status, assignee, etc.)
  • work item information (status, blueprint, assignee, etc.)
  • historical work item data (counts of workitems in particular statuses each day)
  • Team membership information.
  • Technical requirements
    • which blueprints make up each TR
    • Either:
      • what the status was at UDS, or
      • whether it was added post-UDS
  • This database is available and is used by the launchpad-work-items-tracker script.

Report creation

Once we have the database we need to create the reports that will present the data.

TODO: design of the reports.

Report display

Once we have the reports we need a BIRT viewer instance to make them available to people.

TODO: investigate whether this is as simple as deploying an instance, e.g. how do we ACL?

internal/archive/Platform/Infrastructure/Specs/OverallStatusReporting (last modified 2013-08-23 02:00:59)