Summary

We want to allow people to make use of the data stored in launch-control. Current approach based on writing python modules that interact with the database is hard to use and requires knowledge of python, django, and the reporting framework at lest. We need to try a more direct approach to allow users to actually access the data. The selected approach is to allow people to experiment with direct SQL queries in a sand-boxed environment (that is only read access is allowed).

Release Note

Implementing this spec will change data privacy settings in launch-control. Users will no longer be able to have truly private data, only data that other users cannot modify (but can see).

Rationale

N/A

User stories

N/A

Assumptions

Users will need to know the SQL schema of launch-control to proceed. Any incompatible changes to the schema (that was, so far, a private implementation detail) will require updates to user-written SQL queries.

Design

Data View Definition Files

Users will be able to define new data sets with the following file format.

<data-view name="unique-data-view-name">
  <summary>
  One line summary that shows up in listings
  </summary>
  <sql>SELECT * FROM results ORDER BY {argument}</sql>
  <arguments>
     <argument type="string" name="argument" help="description string" default="default value (may be omitted)"/>
     <!-- more arguments follow here -->
  </arguments>
  <documentation>
  Free-form documentation on the purpose and implementation of this data views.
  </documentation>
</data-view>

Some comments:

Arguments would not be able to place arbitrary SQL. They would be placeholders for prepared statement arguments. In other words there would be no free-form text expansion possible.

Supported argument types:

Name

type="" value

Comment

String

string

Any Unicode text allowed

Number

number

Integer or floating point

Boolean

bool

Date/Time

timestamp

Any date+time value

XML-RPC API extensions

The following new functions will be available:

query_data_view

query_data_view(name, arguments)

Arguments:
  name: xml-rpc string - name of the data view a defined in <data-view> name attribute
  arguments: xml-rpc struct - collection of named arguments, each argument as a separate field of the structure.

Returns:
  xml-rpc struct with the following fields:
    rows: xml-rpc array of xml-rpc arrays with data for each row/column
    columns: xml-rpc array of xml-rpc structs with the following fields:
       name: xml-rpc string - the name of the column (derived from the SQL query)
       type: xml-rpc string - type hint the renderer (TBD)

data_views

data_views()

Arguments:
  none

Returns:
  xml-rpc array of xml-rpc structs with the following fields:
    name: xml-rpc string - name of the data view
    summary: xml-rpc string - one line summary of the data view

data_view_info

data_view_info(name)

Arguments:
  name: xlm-rpc string - data view name

Returns:
  xml-rpc struct with the following fields:
    name: xml-rpc string - name of the data view
    summary: xml-rpc string - one line summary of the data view
    documentation: xml-rpc string
    sql: xml-rpc string
    arguments: xml-rpc array of xml-rpc structs with the following fields:
      name: xml-rpc string
      type: xml-rpc string
      help: xml-rpc string
      default: xml-rpc string

AJAX extensions

A new AJAX call would be available, it would be identical to query_data_view() XML-RPC API.

This URL would be available as:

 /data-views/(?P<data_view_name>[a-zA-Z0-9_-]+)/" 

Implementation

The existing IDataSource framework will be used to provide new data sub-type called DataView. At startup the system will load a set of configuration files from a well-known directory. All those files will be loaded, validated and added to the set of registered data views. A new table for storing data viewss will be created to make this possible. This table will be updated from disk but otherwise should be persistent. This will allow us various methods of defining/changing/removing new data viewss that is not directly bound to an on-disk file. A new AJAX and XML-RPC method will be added that allows one to invoke any data view by name. A collection (dictionary) of parameters may be provided to this call to substitute any parameters defined in the SQL query. The result of that query will be returned the caller. For experimenting lc-tool will be extended to support this new XML-RPC API, existing generic tabular dataset renderer present in lc-tool will be used to display the results.

UI Changes

The XML-RPC API page will now show the new command, other UI changes are not planned.

Code Changes

Code changes should include an overview of what needs to change, and in some cases even the specific details.

Migration

Existing GCC report can be migrated to this technology in the future.

Test/Demo Plan

This feature will be delivered in the 0.4 series for launch-control https://launchpad.net/launch-control/0.4 Alpha and beta releases will be subsequently pushed to validation.linaro.org for review.

Unresolved issues

We should have some namespace system for identifying the data view so that multiple people working together don't cause clashes. This needs to be investigated in the future.

BoF agenda and discussion

None


CategorySpec CategoryTemplate

Platform/Validation/Specs/Data views for Launch Control (last modified 2011-04-08 12:21:11)