Objective

To design a multimedia validation suite which can run specific set of multimedia test cases and provide the consolidated report of metrics data of all different test cases in some text file and summary of pass/fail at the end of report.

Summary

This specification is for capturing metrics data generated by instrumented player with set of different input. based on the metrics data, it will give summary. This instrumented player will be based on the gstreamer. If some error comes during decoding of stream or decoded output is not right then that test case will be treated as fail.

Rationale

At present no validation suite exist which can validate the h/w or s/w codec based on specific set of inputs. So this validation suite will show the capabilites of codec so that vendor can decide which codec is best for his platform.

User Stories

  • User has integrated lot of freely available s/w codec inside linaro, now he wants to know which codec works best for his platform.
  • User wants consolidated report of all codecs available in his platform which shows fps, drop rate and properties of each file such as bit rate, resolution etc.
  • User wants consolidated report of pass/fail test cases for each codec available on platform

Assumptions

  • Development platform based on ARM
  • Development platform is able to boot with Linaro kernel and rootfs
  • Gst-omx and gst-tools must be installed inside linaro rootfs
  • Xserver must be up and running over linaro rootfs
  • Video playback is working with gst-launch over Linaro rootfs
  • Instrumented player is used to capture the metrics data during video playback
  • If some validation suite exist already, that can also be used as base.
  • This validation suite will capture metric data as specified below, metric data can be extended on the need basis furhter

Test Input

  • All different codecs available on that platform( for e.g. h/w codec, s/w codec etc.)
  • All different bit streams(for e.g. 240P, 480P, 720P, 1080P etc. )
  • Different bitrates of file
  • Fps of file(30,60,90 etc.)

Metrics data

Instrumented Player/Recorder Data

  • average FPS
  • Drop rate
  • Current resolution of file
  • Bit rate of file
  • Decoder speed

Design

  • Validation suite will run the specifid number of test cases and metrics data generated for each test case will be captured in text file to generate report
  • If some error comes during test case execution or decoded stream is not proper that test cases is considered as fail
  • In the end of generated report, summary of pass/fail test cases will be prepared for each codec.
  • If some validation tool exist, that can also be used as base for runing set of test cases and generating report
  • If some validation tool exist, that can also be extended/ modified further acc. to our requirements

Test Summary

  • Generate the consolidated report in text file for all different inputs
  • Generate the consolidated report for each codec, so that codecs can be compared on that platform
  • In end, total number of pass/fail test case for each codec will be prepared

Test/Demo Plan

  • Video stream will be played using gst-launch in background
  • Instrumented player will capture the player/recorder metric data
  • Based on this metric data for different set of input, consolidated report will be generated by validation suite.

WorkingGroups/Middleware/Multimedia/Specs/1105/ValidationSuite (last modified 2010-12-17 05:54:48)