Introduction
Ian Campbell gave a great talk on using Xen on ARM during LCA14 in Macao.
The text-based 'slides' below serve as a sort of cook-book on how to get started using Xen.
Refer to the Xen wiki for more information: http://wiki.xen.org/wiki/Xen_ARM_with_Virtualization_Extensions
Slides
Components of a Xen system
==========================
- Xen Hypervisor
- Domain 0 Kernel (Linux)
- Device Tree
- Domain 0 Userspace
- Xen Tools
Getting Xen Source
==================
$ git clone git://xenbits.xen.org/xen.git
Branches:
master: Current development (tested)
staging: Current development (pre-test)
stable-4.4: Current stable release (imminent)
staging-4.4: Current stable release (pre-test)
Recommended for development: master unless more specific need
e.g. known prerequisite patches in staging.
Xen Hypervisor
==============
Can cross compile or build natively.
xen.git$ make xen XEN_TARGET_ARCH=arm32 CROSS_COMPILE=arm-linux-gnueabi- [-jN -s etc]
xen.git$ make xen XEN_TARGET_ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- [-jN -s etc]
Not much to configure, no Kconfig, menuconfig etc.
Common options:
debug=y
-------
xen.git$ make xen debug=y [XEN_TARG...etc]
Enabled during development cycles, disabled during rc for release.
Early printk
------------
xen.git$ make xen XEN_TARGET_ARCH=.. debug=y CONFIG_EARLY_PRINTK=<machine>
Where <machine> is:
- vexpress
- midway
- sun7i
- fastmodel
- ... more...
See: docs/misc/arm/early-printk.txt. Requires debug=y which is the
default during development phases.
Easy to add support for any pl011 or 8250 platform, see
xen/arch/arm/Rules.mk. Other UARTs are more work but still trivial.
Can write options to .config. (I think, I never do....)
xen.git$ make dist-xen
=> Install to xen.git/dist/install for rsync, tar+scp etc.
xen.git$ sudo make install-xen
=> Install on local system
xen.git$ sudo make install-xen DESTDIR=/tmp/my-dir
=> Install to a staging area
Domain 0 Kernel (Linux)
=======================
Normal Linux kernel from e.g. torvalds/linux.git later than v3.8
required, recommend latest e.g. v3.13.
Must support native operation on the host platform. Use normal .config
and enable Xen options:
CONFIG_XEN=y
CONFIG_XEN_BLKDEV_FRONTEND=y
CONFIG_XEN_BLKDEV_BACKEND=y
CONFIG_XEN_NETDEV_FRONTEND=y
CONFIG_XEN_NETDEV_BACKEND=y
CONFIG_HVC_XEN=y
CONFIG_HVC_XEN_FRONTEND=y
CONFIG_XEN_DEV_EVTCHN=y
CONFIG_XEN_BACKEND=y
CONFIG_XENFS=y
CONFIG_XEN_COMPAT_XENFS=y
CONFIG_XEN_SYS_HYPERVISOR=y
CONFIG_XEN_XENBUS_FRONTEND=y
CONFIG_XEN_GNTDEV=y
CONFIG_XEN_GRANT_DEV_ALLOC=y
CONFIG_SWIOTLB_XEN=y
CONFIG_XEN_PRIVCMD=y
make Image/zImage in the normal way. Producing arch/arm/boot/zImage or
arch/arm64/boot/Image
Device Tree
===========
Unmodified host device tree. e.g. from linux/arch/arm/boot/dts
Domain 0 Userspace
==================
Your favourite Linux distribution.
Consider using LVM and leaving most of VG free for guest disks, or can
use image back guest filesystem via /dev/loop.
Configure bridging using distro tools, suggest name "xenbr0" for ease
of use.
Xen Tools
=========
Build natively or cross (given a suitable cross build environment) or
native.
xen.git: ./configure [--prefix=... etc]
xen.git: make tools [-jN -s etc]
xen.git: make dist-tools
xen.git: make install-tools
xen.git: make install-tools DESTDIR=/tmp/my-dir
Configure /etc/init.d/xencommons to run on boot.
Make sure you have a getty on hvc0.
Workarounds:
- cp /usr/share/misc/config.* .
=> http://lists.xen.org/archives/html/xen-devel/2014-03/msg00003.html
http://bugs.xenproject.org/xen/mid/%3C1393648440-21407-1-git-send-email-ian.campbell@citrix.com%3E/raw
- https://bugs.launchpad.net/linaro-aarch64/+bug/1169164
Booting
=======
Xen hypervisor binary is compatible with the Linux boot protocol for
arm or arm64 (as appropriate):
- xen.git/docs/misc/arm/booting.txt
Pass domain 0 kernel as a "boot module":
- xen.git/docs/misc/arm/device-tree/booting.txt
4 potential configurations:
- Fastmodel w/ boot-wrapper w/ semihosting (arm32)
- Fastmodel w/ boot-wrapper w/o semihosting (arm64)
- u-boot
- UEFI
Command line options:
- Xen:
- On models use "no-bootscrub".
- dtuart= references device tree UART node.
- http://xenbits.xen.org/docs/unstable/misc/xen-command-line.html
- Linux:
- console=hvc0
- Otherwise as normal (root= etc)
Fastmodel w/ boot-wrapper w/ semihosting (arm32)
------------------------------------------------
- http://wiki.xen.org/wiki/Xen_ARM_with_Virtualization_Extensions/FastModels#arm32
- git://xenbits.xen.org/people/ianc/boot-wrapper.git
$ RTSM_VE_Cortex-A15x2 -C cluster.cpu0.semihosting-cmd_line="
--kernel xen.git/xen/xen \
--module linux.git/arch/arm/boot/zImage <DOMAIN 0 COMMAND LINE>
--dtb rtsm_ve-cortex_a15x2.dtb -- <XEN COMMAND LINE>"
<MODEL OPTIONS> boot-wrapper.git/linux-system-semi.axf
Workaround: "mmc_select_voltage only if host->ocr_avail_mmc"
=> http://lkml.org/lkml/2013/1/17/269
Fastmodel w/ boot-wrapper w/o semihosting (arm64)
-------------------------------------------------
No semihosting boot-wrapper for arm64, so more complicated.
- http://wiki.xen.org/wiki/Xen_ARM_with_Virtualization_Extensions/FastModels#arm64
- git://xenbits.xen.org/people/ianc/boot-wrapper-aarch64.git
Need to modify DTS to add boot module:
chosen {
#address-cells = <1>;
#size-cells = <1>;
xen,xen-bootargs = "<XEN COMMAND LINE>";
module@1 {
compatible = "xen,linux-zimage", "xen,multiboot-module";
reg = <0x80080000 0x800000>;
bootargs = "<DOMAIN 0 COMMAND LINE>";
};
};
<XEN COMMAND LINE> dtuart=serial0 no-bootscrub
<DOMAIN 0 COMMAND LINE> root=/dev/mmcblk0 console=hvc0 rw debug init=/root/init.sh
/chosen/module@1/reg should match bootwrapper model.lds.
Build boot-wrapper:
boot-wrapper-aarch64.git$ ln -s ../xen.git/xen/xen Xen
boot-wrapper-aarch64.git$ ln -s ../arm-dts.git/fast_models/rtsm_ve-aemv8a.dtb fdt.dtb
boot-wrapper-aarch64.git$ ln -s ../linux.git/arch/arm64/boot/Image Image
boot-wrapper-aarch64.git$ make CROSS_COMPILE=aarch64-linux-gnu- xen-system.axf
Run it:
$ RTSM_VE_AEMv8A <MODEL OPTIONS> boot-wrapper-aarch64.git/xen-system.axf
u-boot
------
Can use u-boot command line "fdt" to setup boot-modules.
- http://wiki.xen.org/wiki/Xen_ARM_with_Virtualization_Extensions#Boot_Modules
fdt addr ${fdt_addr}
fdt resize
fdt set /chosen \#address-cells <1>
fdt set /chosen \#size-cells <1>
fdt mknod /chosen module@0
fdt set /chosen/module@0 compatible "xen,linux-zimage" "xen,multiboot-module"
fdt set /chosen/module@0 reg <${kernel_addr_r} 0x${filesize} >
fdt set /chosen/module@0 bootargs "<DOMAIN 0 COMMAND LINE>"
- http://wiki.xen.org/wiki/Xen_ARM_with_Virtualization_Extensions/Allwinner#Boot_script
UEFI
----
WIP (Fu Wei, Linaro). Eventually boot via grub.efi...
Starting a guest
================
Linux or NetBSD.
Create rootfs. Either LVM image or raw file. Use debootstrap or other
distro mechanisms. Linaro rootfs?
Kernel: use any kernel with Xen support enabled, same kernel as dom0
is fine/convenient. Or a distro kernel if available.
Create a config file. See xl.cfg(5).
$ cat xen.cfg
kernel = '/boot/vmlinuz'
ramdisk = '/boot/initrd.img' # optional
vcpus = '2'
memory = '512'
extra = "console=hvc0 earlyprintk debug root=/dev/xvda ro"
disk = ["phy:/dev/loop0,xvda,rw"] # or phy:/dev/mapper/VG-LV etc.
name = 'my-xen-domain'
vif = [ 'mac=8e:a7:8e:3c:f4:f6' ]
Start it:
$ xl create xen.cfg
Console:
$ xl console my-xen-domain
List:
$ xl list
Name ID Mem VCPUs State Time(s)
Domain-0 0 128 2 r----- 8.0
my-xen-domain 1 128 1 r----- 0.5
Debug:
$ xl -vvv [command...]
More info xl(1).
Logs /var/log/xen/xl-$name.log
Supporting a new platform
=========================
Basics should Just Work (tm). Platform interface in
xen/include/asm-arm/platform.h and existing implementations in
xen/arch/arm/platforms/*.c.
Hooks for SMP init (but prefer generic PSCI), platform quirks, device
blacklists etc.
Contributing to Xen
===================
Workflow very similar to Linux kernel. Signed-off-by patches sent to
xen-devel@lists.xen.org, ideally CC maintainers (see MAINTAINERS file
in xen.git)
http://wiki.xen.org/wiki/Submitting_Xen_Patches
Release Cycle
=============
Traditionally a ~9 month release cadence. Trying to pull this in to ~6
months. Xen 4.4 is due for release next week.
Xen 4.5 is therefore ~Aug/Sep. Expect code freeze ~6 weeks before.
More Information
================
http://wiki.xen.org/wiki/Xen_ARM_with_Virtualization_Extensions
http://xenbits.xen.org/docs
Includes links to platform specific advice.
IRC: #xenarm on freenode.
http://lists.xenproject.org/mailman/listinfo/xen-devel
Appendix
========
/root/init.sh:
#!/bin/sh
set -x
mount -o remount,rw /
mount -t proc none /proc
mount -t sysfs none /sys
mount -t tmpfs none /run
mkdir /run/lock
#/etc/init.d/udev start
mount -t devtmpfs dev /dev
/sbin/udevd --daemon
udevadm trigger --action=add
mkdir /dev/pts
mount -t devpts none /dev/pts
hostname -F /etc/hostname
mknod -m 640 /dev/xconsole p
chown root:adm /dev/xconsole
/sbin/klogd -c 1 -x
/usr/sbin/syslogd
/etc/init.d/xencommons start
echo 9 > /proc/sysrq-trigger
cd /root
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
exec /bin/bashLEG/Engineering/Virtualization/Xen_ARM_Guide (last modified 2014-03-17 16:58:40)