QEMU/KVM Migration Testing

This page is a work in progress where I'm sketching out what we need to do to test the VM migration support we're implementing under CARD-363. That card's validation criteria say:

  • Validation of code; successful migration of VM between two instances of kvm/QEMU on same platform. If two boards are available to the developer migration between boards will be tested. VM configuration will be a simple config with either versatile-express or (if available) mach-virt, with an SMP 2 core guest and one SD card for storage.

There are two axes to this testing:

Simple vm save-and-load vs live-migration

Simple save/load test:

  • boot the guest; in the console window it creates, use alt-2 to switch to the qemu monitor prompt
  • NB that you do need your filesystem to be QCOW2, not raw image [to convert to qcow2 use "qemu-img convert -O qcow2 my-fs.img my-fs.qcow2" ]
  • type 'savevm name' to save [where 'name' is whatever you want to call the save]; the vm will save, and then resume execution
  • you can then type 'quit' to kill the VM (or just ctrl-c it)
  • for loading, rerun the guest with "-loadvm name" as an extra command line argument; it should resume the boot from wherever you saved it.
  • alternatively you can boot a guest as usual but then in the monitor use 'loadvm name'
  • a robust test process should aim to do save-and-restore while the guest is doing a variety of things (middle of kernel boot, idle at a user prompt, under heavy load, etc)

TODO: investigate, test and document live-migration setup

TCG vs KVM-no-vgic/vtimer vs KVM-with-vgic/vtimer

All these configs should work in both save/load and migration cases.

If the baseline command is:

./arm-softmmu/qemu-system-arm -machine type=vexpress-a15   -sd ~/test-images/a15-3.8/arm-wheezy.qcow2  -serial stdio -append 'root=/dev/mmcblk0 rw console=ttyAMA0 rootwait earlyprintk'   -kernel ~/test-images/a15-3.8/zImage   -dtb ~/test-images/a15-3.8/vexpress-v2p-ca15-rtsm.dtb -smp 2


  • for TCG use the command as above;
  • for KVM-with-vgic/vtimer add "-enable-kvm";
  • for KVM-without-vgic-vtimer add "-enable-kvm -machine kernel_irqchip=off" and you may also need to remove the "arm,armv7-timer" node from the dtb

Current status:

  • TCG and KVM-without-vgic-vtimer should work
  • KVM-with-vgic-vtimer requires the VIRT-42/43/44/45 kernel work and VIRT-48 QEMU work (not yet complete)

TODO: nail down what our actual test kernel etc is so this is a fully reproducible test process

LAVA integration

The deliverables on the card state:

  • Corresponding tests developed and added to LAVA to ensure regression testing is in place.

TODO: need to check this and see what is required.

The LEG/Engineering/Virtualization/KVM-CI wiki page gives the current status and lists the config files that would need to be edited to add extra tests like migration.

Since the tests are driven by a shell script it should be straightforward to make it start two simultaneous KVM sessions to migrate between.

We probably want to use QEMU's command line options to connect the QEMU monitor to a TCP socket or file descriptor so that we can monitor the status of the VM and send it the savevm command in a programmatic way.

TODO: check what upstream does for KVM testing and whether there are simple test cases that can be easily integrated with LAVA.

Updates for 2014 -- PMM

Mostly the above is still reasonable; minor updates:

We should probably use the 'virt' machine rather than 'vexpress-a15'.

VIRT-42/43/44/45 kernel work and VIRT-48 QEMU work is now done, so KVM+vgic+vtimer migration should work OK.

PeterMaydell/MigrationTesting (last modified 2014-05-07 16:08:34)