Contents

  1. Prerequisites
  2. Download
  3. Quickstart
  4. External Binaries
  5. Test Case Specifications
    1. cpufreq
      1. test the cpufreq framework is available for frequency
      2. test the cpufreq framework is available for governor
      3. test the governor change is effective
      4. test the change of the frequency is effective in 'userspace' mode
      5. test 'ondemand' and 'conservative' trigger correctly the configuration directory
      6. test the change of the frequencies affect the performances of a test program
      7. test the load of the cpu affects the frequency with 'ondemand'
      8. test the load of the cpu does not affect the frequency with 'userspace'
      9. test the load of the cpu does not affect the frequency with 'powersave'
    2. cputopology
      1. test the topology is implemented in the kernel
      2. test the topology files are present
      3. test if the sched_domain flags are present
      4. test capacity calculation based on the algorithm in arch/arm/kernel/topology.c
    3. cpuhotplug
      1. check cpu online status before running cpuhotplug subtests
      2. test the sysfs files are present
      3. test the cpu state is reflected in sysfs
      4. test the affinity does not work on an offline cpu
      5. test task is migrated with task affinity
      6. test cpus are reflected on /proc/cpuinfo correctly
      7. test interrupts show the right configuration in /proc/interrupts
      8. test userspace notification
      9. stress test
      10. check cpu online status after running cpuhotplug subtests
    4. Cpuidle
      1. Check sysfs files presence
      2. Shake the cpuidle driver
      3. Shake the cpuidle driver and unplug cpu
      4. Do some cpuidle statistics
      5. Try to detect L1 flush cache missing
    5. Suspend
      1. perform a suspend test via dbus message
      2. perform a suspend test via pm-suspend
      3. perform a suspend test via sysfs interface
      4. perform tests involving removing ac power
      5. perform a battery consumption test
    6. Thermal
      1. list thermal zones and cooling devices
      2. check the thermal zone attributes
      3. check the cooling device attributes
      4. check the temperature varies with the load
      5. check cooling device lowers the temperature down
      6. check the cpufreq cooling device can clip down the frequency
      7. check that different trip points are crossed when load is maximum
  6. Information on thermal testing
    1. Turn cpu utilization to 100%
    2. Automate Thermal measurement
  7. Questions/Comments
  8. Bugs


Prerequisites

The test cases are designed to be run in an automated test environment such as LAVA. They are designed for a specific host configured to have a minimal number of services running on it and no pending cron job in order to not alter the expected results.

Download

To download the PM-QA test suite, run 'git clone git://git.linaro.org/tools/pm-qa.git'.

Quickstart

In order to launch the test suite, invoke at the topmost directory 'make check' as root. If you wish to have the tests to continue even if one fails use the '-k' option of the Makefile, 'make -k check'

External Binaries

The following external binaries (located in the utils directory) are executed in the background while running the PM-QA test cases:

  • cpucycle : Gets a cpu name as parameter and sets the affinity on it. In one second, it increments a counter and displays the result. It checks for the performance changes related to the frequency changes.
  • cpuburn : Gets a cpu name as parameter and sets the affinity on it. It runs an infinite loop consuming the cpu resources without invoking any syscalls.
  • nanosleep : Waits for the frequency change latency for a certain number of nanoseconds.
  • cpuidle_killer : It forks for each online cpu and set the affinity of the newly created process to a cpu. Each cpu will run an instance of the testing program. Then during a time interval, each process does by chunk of time, half of the time a busy-loop and nothing during the second half. If there is something wrong with the cpuidle driver, this program will very likely trigger it (block the system, a kernel panic or some task hanged). Make sure the kernel is compiled with 'detect hung tasks'.

Test Case Specifications

cpufreq


test the cpufreq framework is available for frequency

  • name : cpufreq_01
  • script : check the following files are present in the sysfs path:
    • /sys/devices/system/cpu/cpu[0-9].*
    • -> cpufreq/scaling_available_frequencies

    • -> cpufreq/scaling_cur_freq

    • -> cpufreq/scaling_setspeed

test the cpufreq framework is available for governor

  • name : cpufreq_02
  • script : check the following files are present in the sysfs path:
    • /sys/devices/system/cpu/cpu[0-9].*
    • -> cpufreq/scaling_available_governors

    • -> cpufreq/scaling_governor

test the governor change is effective

  • name : cpufreq_03
  • script :
    • check for each cpu
      • For each available governors
        • set the governor
        • read the current governor.
        • check the governor has changed to the new one.

test the change of the frequency is effective in 'userspace' mode

  • name : cpufreq_04
  • script :
    • set the governor to 'userspace'
      • for each frequency and cpu
        • write the frequency
        • wait at least cpuinfo_transition_latency
        • read the frequency
        • check the frequency is the one you expect
        • reset the frequency to the initial state
        • reset the governor to the initial state

test 'ondemand' and 'conservative' trigger correctly the configuration directory

  • name : cpufreq_05
  • script :
    • set the governor to 'ondemand' on each cpu
      • check the /sys/device/system/cpu/cpufreq/ondemand directory is present and unique
    • set the governor to 'conservative on each cpu
      • check the /sys/device/system/cpu/cpufreq/conservative directory is present and unique
    • if more than one cpu:
      • set on the first cpu 'ondemand'
      • set on the second cpu 'conservative'
      • check the /sys/device/system/cpu/cpufreq directory contains the 'ondemand' and 'conservative' directories

test the change of the frequencies affect the performances of a test program

  • name : cpufreq_06
  • script :
    • for each cpu
      • set the governor to userspace policy
      • for each available frequency
        • change the frequency
        • wait at least cpuinfo_transition_latency
        • run 'cpucycle'
        • check the result is better when the frequency is higher with a deviation
  • note about the deviation computation
    • the script computes the ratio between the frequency and the number of iterations of the counter
    • the script computes the average ratio for all the frequency
    • the script computes the deviation of a ratio versus the average ratio (abs(ratio - avg_ratio) / avg_ratio) * 100
    • the following indications give a clue about the correctness of the frequency scaling, when the deviation is:
      • less than 0.5 % : VERY GOOD
      • between 0.5 % and 1 % : GOOD
      • more than 1 % and 2 % : BAD
      • more than 2 % and 5 % : VERY BAD
      • more than 5 % and 6 % : SUSPECT
      • more than 6 % : BOGUS These values are impacted by any application other than the pm-qa test program consuming cpu

test the load of the cpu affects the frequency with 'ondemand'

  • name : cpufreq_07
  • script :
    • for each cpu
      • set the governor to ondemand policy
      • run 'cpuburn' in background
      • wait at least cpuinfo_transition_latency * nr_scaling_available_frequencies
      • read the frequency
      • kill 'cpuburn'
      • check the frequency is equal to the highest available frequency
      • wait at least cpuinfo_transition_latency * nr_scaling_available_frequencies
      • read the frequency
      • check the frequency is the lowest available

test the load of the cpu does not affect the frequency with 'userspace'

  • name : cpufreq_08
  • script :
    • for each cpu
      • set the governor to 'userspace'
      • set the frequency between min and max frequencies
      • wait at least cpuinfo_transition_latency * nr_scaling_available_frequencies
      • run 'cpuburn' in background
      • read the frequency
      • kill 'cpuburn'
      • check the frequency is equal to the one we set

test the load of the cpu does not affect the frequency with 'powersave'

  • name : cpufreq_09
  • script :
    • for each cpu
      • set the governor to 'powersave'
      • wait at least cpuinfo_transition_latency * nr_scaling_available_frequencies
      • run 'cpuburn' in background
      • read the frequency
      • kill 'cpuburn'
      • check the frequency is the lowest available frequency


  1. test the load of the cpu affect the frequency with 'conservative'
    • for each cpu
      • - set the governor to 'conservative' for each freq step - set the up_threshold to the freq step - wait at least cpuinfo_transition_latency * nr_scaling_available_frequencies - run (*) in background - read the frequency - check the frequency is equal to higher we have with the freq_step
      kill (*)

cputopology


test the topology is implemented in the kernel

  • name : cputopology_01
  • script :
    • for each cpu
      • check the value of the physical_package_id file is different from "-1"

test the topology files are present

  • name : cputopology_02
  • script :
    • for each cpu
      • check the presence of:
        • core_id
        • core_siblings
        • core_siblings_list
        • physical_package_id
        • thread_siblings
        • thread_siblings_list

test if the sched_domain flags are present

  • name: cputopology_03
  • script:
    • for each cpu:
      • check the presence of shed_domain flags

test capacity calculation based on the algorithm in arch/arm/kernel/topology.c

  • name : cputopology_04
  • script :
    • check that the information in /sys/firmware/devicetree/base/cpus corresponds to that reported by the kernel log.

cpuhotplug


check cpu online status before running cpuhotplug subtests

  • name : cpuhotplug_00
  • script :
    • check if all cpus are online before running remaining cpuhotplug subtests

test the sysfs files are present

  • name : cpuhotplug_01
  • script :
    • for each cpu
      • check /sys/devices/system/cpu/cpuX/online is there

test the cpu state is reflected in sysfs

  • name : cpuhotplug_02
  • script :
    • for each cpu
      • set cpuX offline
      • check cpuX status is offline
      • set cpuX back online
      • check cpuX status is online

test the affinity does not work on an offline cpu

  • name : cpuhotplug_03
  • script :
    • for each cpu
      • set cpuX offline
      • check 'taskset -c X /bin/true' fails
      • set cpuX back online

test task is migrated with task affinity

  • name : cpuhotplug_04
  • script :
    • for each cpu
      • run a task on dedicated cpuX
      • get cpumask for this task
      • offline cpuX
      • get cpumask for this task
      • check cpumask differs from the previous one

test cpus are reflected on /proc/cpuinfo correctly

  • name : cpuhotplug_05
  • script :
    • for each cpu
      • set cpuX offline
      • check /proc/cpuinfo 'processor' keyword count is 'NRCPUS - 1'
      • check the 'processor' keyword contains a cpu id different from the cpu we offlined
      • set cpuX back online
      • check one of the 'processor' keyword contains cpuX

test interrupts show the right configuration in /proc/interrupts

  • name : cpuhotplug_06
  • script :
    • for each cpu
      • set cpuX offline
      • check cpuX is not present in /proc/interrupts
      • set cpuX back online
      • check cpuX is present in /proc/interrupts

test userspace notification

  • name : cpuhotplug_07
  • script :
    • for each cpu
      • set cpuX offline
      • check from userspace notification has been done with the command 'udevadm monitor --kernel --subsystem-match=cpu' and the event is related to the cpuX and 'offline'
      • set the cpu online
      • check from userspace notification has been done with the command 'udevadm monitor --kernel --subsystem-match=cpu' and the event is related to the cpuX and 'online'

stress test

  • name : cpuhotplug_08
  • script :
    • for i in 1 .. 50
      • randomly choose a cpu
        • set cpuX offline
        • set cpuX online

check cpu online status after running cpuhotplug subtests

  • name : cpuhotplug_99
  • script :
    • check if all cpus are back online after running cpuhotplug subtests

Cpuidle


Check sysfs files presence

  • name : cpuidle_01
  • script :
    • for each cpu
      • check 'desc' 'latency' 'name' 'power' 'time' 'usage' files are there for each states
      • check 'current_driver' 'current_governor_ro' files are there

Shake the cpuidle driver

  • name : cpuidle_02
  • script
    • run cpuidle_killer utility

Shake the cpuidle driver and unplug cpu

  • name : cpuidle_03
  • script
    • for each cpu
      • set offline cpu if different from cpu0
      • run cpuidle_killer
    • for each cpu
      • set online cpu if different from cpu0

Do some cpuidle statistics

  • name : cpuidle_04
  • script
    • disabled

Try to detect L1 flush cache missing

  • name : cpuidle_05
  • script
    • run cpuidle_l1 10 times

Suspend


perform a suspend test via dbus message

  • name : suspend_01
  • script :
    • suspend the machine by dbus message
    • the machine will be resumed by rtc alarm

perform a suspend test via pm-suspend

  • name : suspend_02
  • script :
    • suspend the machine by pm-suspend
    • the machine will be resumed by rtc alarm

perform a suspend test via sysfs interface

  • name : suspend_03
  • script :
    • "echo mem > /sys/power/state" if it is supported

    • the machine will be resumed by rtc alarm

perform tests involving removing ac power

  • name : suspend_04
  • script :
    • suspend the machine with AC disconnected
    • suspend the machine with AC connected
    • remove the AC cord while the machine is suspended
    • insert the AC cord while the machine is suspended
    • every time the machine should be resumed by rtc

perform a battery consumption test

  • name : suspend_05
  • script :
    • suspend the machine with AC disconnected
    • the suspend time $args_power_sleep is 10 minutes by default
    • when machine is resumed, the consumption of battery is calculated

Thermal


list thermal zones and cooling devices

  • name : thermal_00
  • script:
    • list the thermal zones and cooling devices

check the thermal zone attributes

  • name : thermal_01
  • script :
    • check the mode, temp, type and uevent attributes are there for each thermal zone
    • check the enable and disable mode work for each thermal zone
    • check for each thermal zone, for each trip point the temp and type attributes are there
    • check every cooling device is binded

check the cooling device attributes

  • name : thermal_02
  • script :
    • check the cur_state, max_state, type and uevent attributes are there for each cooling device
    • check every cooling state works for each cooling device

check the temperature varies with the load

  • name : thermal_03
  • script :
    • start cpu and gpu heat binary
    • check the temperature varies with the load

check cooling device lowers the temperature down

  • name : thermal_04
  • script :
    • start heat cpu binary, set cooling device to state 0 and then to another valid state supported, check the temperature is cooled down by valid cooling state

check the cpufreq cooling device can clip down the frequency

  • name : thermal_05
  • script :
    • if there is a cpufreq cooling device, set the cpufreq cooling device to every valid state and check the frequency is changed

check that different trip points are crossed when load is maximum

  • name : thermal_06
  • script :
    • start cpu and gpu heat binary, check every trip point is crossed

Information on thermal testing


Turn cpu utilization to 100%

Git repository: git://git.linaro.org/people/amitdanielk/thermal-test.git master

File to check: heat_cpu.c

Current version is version .5. Just runs a bunch of C integer and floating calculations. Provides no optimized assembly.

Workloads run in IDLE scheduler context to allow other SoC heating programs to fully run also (GPUI and VPU thermal test programs, etc.)

how to build on target: make

how to build on host for target: make CROSS_COMPILE=<insert your cross compiler prefix>

To test this program is working properly on your device:

1. sudo apt-get install sysstat

2. In one terminal run "sar -u 2 -P ALL"

3. In another terminal run ./heat_cpu

You should see all CPUs at near 100% load.

Automate Thermal measurement

Git repository: git://git.linaro.org/people/amitdanielk/thermal-test.git master

File to check: run_thermal.sh

The above script runs in both android and native filesystem. In android, this invokes the 3D based application glmark2 application repeatedly and also runs the cpu intensive binary cpu_heat with slight low priority. In native filesystem cpu_heat is executed in busy looping.

Requirement:

Glmark2 application, busybox, Above heat_cpu binary

Execute as:

sh run_thermal.sh 100

Output as:

thermal.png

Questions/Comments

Submit questions and comments to the linaro-dev mailing list with "PM-QA" as part of the subject line.

Bugs

Please file bugs on Linaro Bugzilla. Select "Power Management" and then "PM-QA". Be detailed as possible to describe the environment you are running on, steps taken, and provide logs.

WorkingGroups/PowerManagement/Resources/TestSuite/PmQaSpecification (last modified 2016-02-02 23:41:34)