Instructions for platform device passthrough on a Calxeda Midway

In those guidelines we use some prompt conventions that indicate where the commands should be issued:

H> your own machine prompt
L> the lab machine (lab.validation.linaro.org)
N> the Midway node which is assigned to you (for instance midway01-04-XX)
G> The guest OS prompt
C> Node console session 

For example midway01-04-XX, 01 indicates the server, 04 is the node and XX is the interface. Each node has 3 interfaces, each assigned an IP. The 00, 01 correspond to the xgmac interfaces. The 02 corresponds to the console through which commands can be sent to the node (using the ipmi commands).

In the following text midwaySS-NN-II is used.

For sake of clarity I sometimes put my OWN node coordinates which are midway01-04-00 IP=10.5.3.7 (host xgmac) midway01-04-01 IP=10.5.3.8 (guest xgmac) midway01-04-02 IP=10.5.3.9 (console)

Booting the saucy installer on Midway

Please refer to https://wiki.ubuntu.com/ARM/Server/Install. Download the kernel image and the initrd on your machine.

H> 
wget http://ports.ubuntu.com/ubuntu-ports/dists/saucy/main/installer-armhf/current/images/generic-lpae/netboot/initrd.gz
wget http://ports.ubuntu.com/ubuntu-ports/dists/saucy/main/installer-armhf/current/images/generic-lpae/netboot/vmlinuz

Note that the vmlinuz and zImage file formats are interchangeable (u-boot's bootz boots both of them).

Log on to the lab server:

H> ssh <login>@lab.validation.linaro.org

create a directory on /var/lib/tftpboot/<user>, further referred to as $TFTPDIR

transfer the saucy kernel and initrd to the lab server.

H>
scp initrd.gz <login>@labfree:$TFTPDIR/initrd.gz
scp vmlinuz <login>@labfree:$TFTPDIR/vmlinuz

Connect to the node console

L> ipmitool -I lanplus -U admin -P admin -H midwaySS-NN-02 sol activate

in another lab login session, power the node off and on and be ready to interrupt the boot in the console (type "s")

L>
ipmitool -I lanplus -U admin -P admin -H midwaySS-NN-02 power off
ipmitool -I lanplus -U admin -P admin -H midwaySS-NN-02 power on

In the console window (where the boot was interrupted), launch the boot of the installer

C>
setenv serverip 10.0.0.10
setenv ipaddr 10.5.3.10
tftpboot ${kernel_addr_r} 10.0.0.1:<user>/vmlinuz
tftpboot ${ramdisk_addr_r} 10.0.0.1:<user>/initrd.gz
bootz ${kernel_addr_r} ${ramdisk_addr_r}:${filesize} ${fdt_addr}

where <user> corresponds to /var/lib/tftpboot/<user>.

complete the installation process which is quite long. chose the base Ubuntu server plus ssh service. For network configuration let dhcp automatic configuration execute. Normally the host will be assigned interface 0 IP. Never overwrite the IP of the console (interface 2). It would prevent the node from beeing controlled. For formatting option, use entire disk (guided).

Add some important packages for local development:

N> apt-get install gcc make ssh g++ libfdt-dev libpixman-1-dev git bison flex pkg-config libglib2.0-dev zlib1g-dev autoconf libtool

Build your host kernel

Now let's replace this kernel with our own, featuring all the requested pieces for KVM device passthrough. VFIO platform driver and irqfd/arm is available from 4.1-rc1 onwards.

H> git clone git://git.linaro.org/people/eric.auger/linux.git; checkout the master.

CROSS_COMPILE=arm-linux-gnueabihf- ARCH=arm make multi_v7_defconfig
scripts/config -e CONFIG_ARM_LPAE
scripts/config -e CONFIG_PROC_DEVICETREE
scripts/config -e CONFIG_MDIO_BUS_MUX_GPIO
scripts/config -e CONFIG_MDIO_BUS_MUX_MMIOREG
scripts/config -e CONFIG_LBDAF
scripts/config -e CONFIG_ARCH_VIRT
scripts/config -e CONFIG_BLK_DEV_LOOP
scripts/config --set-val CONFIG_BLK_DEV_LOOP_MIN_COUNT 8
scripts/config -e CONFIG_VIRTUALIZATION
scripts/config -e CONFIG_KVM
scripts/config -e CONFIG_KVM_ARM_HOST
scripts/config -e CONFIG_KVM_ARM_TIMER
scripts/config -e CONFIG_KVM_ARM_VGIC
scripts/config -e CONFIG_MMU_NOTIFIER
scripts/config --set-val CONFIG_KVM_ARM_MAX_VCPUS 4
scripts/config -e CONFIG_IOMMU_SUPPORT
scripts/config -e CONFIG_ARM_SMMU
scripts/config --set-val CONFIG_ARM_DMA_IOMMU_ALIGNMENT 8
scripts/config -e CONFIG_VHOST_NET
scripts/config -e CONFIG_NETDEVICES
scripts/config -e CONFIG_CAIF_VIRTIO
scripts/config -e CONFIG_VFIO
scripts/config -e CONFIG_VFIO_PLATFORM
scripts/config -e CONFIG_VFIO_PCI
scripts/config -e CONFIG_IKCONFIG_PROC
scripts/config -e CONFIG_EXT2_FS_POSIX_ACL
scripts/config -e CONFIG_EXT2_FS_SECURITY
scripts/config -e CONFIG_EXT2_FS_XIP
scripts/config -e CONFIG_EXT2_FS
scripts/config -e CONFIG_EXT2_FS_XATTR
scripts/config -e CONFIG_EXT3_DEFAULTS_TO_ORDERED
scripts/config -e CONFIG_EXT3_FS_XATTR
scripts/config -e CONFIG_EXT3_FS_POSIX_ACL
scripts/config -e CONFIG_EXT3_FS_SECURITY
scripts/config -e CONFIG_EXT3_FS
scripts/config -e CONFIG_JBD_DEBUG
scripts/config -e CONFIG_HUGETLBFS
scripts/config -e CONFIG_TRANSPARENT_HUGEPAGE
scripts/config -e CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE
scripts/config -e CONFIG_SYS_SUPPORTS_HUGETLBFS
scripts/config -e CONFIG_DYNAMIC_DEBUG
scripts/config -e CONFIG_BRIDGE
scripts/config -e CONFIG_TUN

CROSS_COMPILE=arm-linux-gnueabihf- ARCH=arm make -j 4 zImage

transfer the zImage to the tftp space:

H> scp arch/arm/boot/zImage <login>@lab.validation.linaro.org:$TFTPDIR/zImage

reboot the node using the same ipmi procedure as for installer. On the console window, once the boot is interrupted, run the following command, replacing 10.5.3.7 by the IP address of your interface 0 and the host name midway01-04-0 by your interface 0 name

C>
setenv serverip 10.0.0.10
setenv ipaddr 10.5.3.7
tftp $ramdisk_addr_r 10.0.0.10:augere/smmu-swap-gic.dtb
tftp $kernel_addr_r 10.0.0.10:augere/zImage
setenv bootargs "loglevel=7 console=ttyAMA0  earlyprintk nosplash break root=/dev/sda2 rw ip=10.5.3.7:10.0.0.1:10.0.0.1:255.0.0.0:midway01-04-00:eth0:off:10.0.0.1"
bootz ${kernel_addr_r} - ${ramdisk_addr_r}

the .dts is available on demand. Few modifications were needed in original dts to add smmu, swap the eth nodes and change GIC reg size.

Building QEMU

On the node, clone & build QEMU featuring VFIO platform device:

N>
git clone git://git.linaro.org/people/eric.auger/qemu.git; checkout master;
$ ./configure --target-list=arm-softmmu --enable-kvm --enable-trace-backends=stderr
$ make -j 4

Build the Guest

Compile a guest kernel:

H>
CROSS_COMPILE=arm-linux-gnueabihf- ARCH=arm make multi_v7_defconfig
scripts/config -e CONFIG_ARM_LPAE
scripts/config -e CONFIG_PROC_DEVICETREE
scripts/config -e CONFIG_MDIO_BUS_MUX_GPIO
scripts/config -e CONFIG_MDIO_BUS_MUX_MMIOREG
scripts/config -e CONFIG_EXT2_FS
scripts/config -e CONFIG_EXT2_FS_XATTR
scripts/config -e CONFIG_EXT2_FS_POSIX_ACL
scripts/config -e CONFIG_EXT2_FS_SECURITY
scripts/config -e CONFIG_EXT2_FS_XIP
scripts/config -e CONFIG_EXT3_FS
scripts/config -e CONFIG_EXT3_DEFAULTS_TO_ORDERED
scripts/config -e CONFIG_EXT3_FS_XATTR
scripts/config -e CONFIG_EXT3_FS_POSIX_ACL
scripts/config -e CONFIG_EXT3_FS_SECURITY
scripts/config -e CONFIG_VIRTIO_BLK
scripts/config -e CONFIG_SCSI_VIRTIO
scripts/config -e CONFIG_VIRTIO_NET
scripts/config -e CONFIG_VIRTIO_CONSOLE
scripts/config -e CONFIG_VIRTIO
scripts/config -e CONFIG_VIRTIO_BALLOON
scripts/config -e CONFIG_VIRTIO_MMIO

build a rootfs:

H>
sudo qemu-debootstrap --arch=armhf saucy ./saucy
sudo chroot ./saucy
edit /etc/init/ttyAMA0.conf, copy of tty1.conf
change the last line
create a user

dd if=/dev/zero of=./ubuntu.img bs=1GiB count=1
mkfs.ext3 ./ubuntu.img
mkdir mnt
sudo mount -o loop ubuntu.img mnt
sudo cp -a saucy/* mnt/
sudo umount mnt

then transfer the resulting zImage to the node as zImage.kvm_guest and the rootfs as ubuntu.img.

Run the VM

Eventually we are now ready to run the guest and assign eth1 to it. eth0 is used by host only.

on host, unbind the native xgmac driver from fff51000.ethernet device and bind the device to the VFIO platform meta-driver:

N>
echo vfio-platform | sudo tee -a /sys/bus/platform/devices/fff51000.ethernet/driver_override
echo fff51000.ethernet | sudo tee -a /sys/bus/platform/devices/fff51000.ethernet/driver/unbind
echo fff51000.ethernet | sudo tee -a /sys/bus/platform/drivers_probe

Launch the guest; replace 10.5.3.8 by your interface #1 IP:

N> 
sudo arm-softmmu/qemu-system-arm  \
        -M virt -smp 1 -m 512\
        -serial stdio \
        -display none \
        -drive if=none,cache=writeback,file=../ubuntu.img,id=guestrootfs \
        -device virtio-blk-device,drive=guestrootfs \
        -device vfio-calxeda-xgmac,host="fff51000.ethernet"\
        -append 'loglevel=8 root=/dev/vda rw console=ttyAMA0 ip=10.5.3.8:10.0.0.1:10.0.0.1:255.0.0.0:midway01-04-01:eth0:off:10.0.0.1' \
        --enable-kvm \
        -kernel ../zImage.kvm_guest

This is the "-device vfio-calxeda-xgmac,..." line that triggers the dynamic instantiation of the Calxeda xgmac VFIO platform device.

Once logged you should be able to ping the gateway.

LEG/Engineering/Virtualization/Platform_Device_Passthrough_on_Midway (last modified 2015-06-24 11:38:18)