Ian Campbell gave a great talk on using Xen on ARM during LCA14 in Macao.

The text-based 'slides' below serve as a sort of cook-book on how to get started using Xen.

Refer to the Xen wiki for more information:


Components of a Xen system

- Xen Hypervisor
- Domain 0 Kernel (Linux)
- Device Tree
- Domain 0 Userspace
- Xen Tools

Getting Xen Source

$ git clone git://

  master:       Current development (tested)
  staging:      Current development (pre-test)
  stable-4.4:   Current stable release (imminent)
  staging-4.4:  Current stable release (pre-test)

Recommended for development: master unless more specific need
e.g. known prerequisite patches in staging.

Xen Hypervisor

Can cross compile or build natively.

xen.git$ make xen XEN_TARGET_ARCH=arm32 CROSS_COMPILE=arm-linux-gnueabi- [-jN -s etc]
xen.git$ make xen XEN_TARGET_ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- [-jN -s etc]

Not much to configure, no Kconfig, menuconfig etc.

Common options:


xen.git$ make xen debug=y [XEN_TARG...etc]

Enabled during development cycles, disabled during rc for release.

Early printk

xen.git$ make xen XEN_TARGET_ARCH=.. debug=y CONFIG_EARLY_PRINTK=<machine>

Where <machine> is:
  - vexpress
  - midway
  - sun7i
  - fastmodel
  - ... more...

See: docs/misc/arm/early-printk.txt. Requires debug=y which is the
default during development phases.

Easy to add support for any pl011 or 8250 platform, see
xen/arch/arm/ Other UARTs are more work but still trivial.

Can write options to .config. (I think, I never do....)

xen.git$ make dist-xen
  => Install to xen.git/dist/install for rsync, tar+scp etc.

xen.git$ sudo make install-xen
  => Install on local system

xen.git$ sudo make install-xen DESTDIR=/tmp/my-dir
  => Install to a staging area

Domain 0 Kernel (Linux)

Normal Linux kernel from e.g. torvalds/linux.git later than v3.8
required, recommend latest e.g. v3.13.

Must support native operation on the host platform. Use normal .config
and enable Xen options:

make Image/zImage in the normal way. Producing arch/arm/boot/zImage or

Device Tree

Unmodified host device tree. e.g. from linux/arch/arm/boot/dts

Domain 0 Userspace

Your favourite Linux distribution.

Consider using LVM and leaving most of VG free for guest disks, or can
use image back guest filesystem via /dev/loop.

Configure bridging using distro tools, suggest name "xenbr0" for ease
of use.

Xen Tools

Build natively or cross (given a suitable cross build environment) or

xen.git: ./configure [--prefix=... etc]
xen.git: make tools [-jN -s etc]

xen.git: make dist-tools
xen.git: make install-tools
xen.git: make install-tools DESTDIR=/tmp/my-dir

Configure /etc/init.d/xencommons to run on boot.

Make sure you have a getty on hvc0.

- cp /usr/share/misc/config.* .


Xen hypervisor binary is compatible with the Linux boot protocol for
arm or arm64 (as appropriate):

- xen.git/docs/misc/arm/booting.txt

Pass domain 0 kernel as a "boot module":

- xen.git/docs/misc/arm/device-tree/booting.txt

4 potential configurations:

- Fastmodel w/ boot-wrapper w/ semihosting (arm32)
- Fastmodel w/ boot-wrapper w/o semihosting (arm64)
- u-boot

Command line options:

- Xen:
  - On models use "no-bootscrub".
  - dtuart= references device tree UART node.

- Linux:
  - console=hvc0
  - Otherwise as normal (root= etc)

Fastmodel w/ boot-wrapper w/ semihosting (arm32)

- git://

$ RTSM_VE_Cortex-A15x2 -C cluster.cpu0.semihosting-cmd_line="
    --kernel xen.git/xen/xen \
    --module linux.git/arch/arm/boot/zImage <DOMAIN 0 COMMAND LINE>
    --dtb rtsm_ve-cortex_a15x2.dtb -- <XEN COMMAND LINE>"
  <MODEL OPTIONS> boot-wrapper.git/linux-system-semi.axf

Workaround: "mmc_select_voltage only if host->ocr_avail_mmc"

Fastmodel w/ boot-wrapper w/o semihosting (arm64)

No semihosting boot-wrapper for arm64, so more complicated.

- git://

Need to modify DTS to add boot module:

    chosen {
        #address-cells = <1>;
        #size-cells = <1>;

        xen,xen-bootargs = "<XEN COMMAND LINE>";
        module@1 {
                 compatible = "xen,linux-zimage", "xen,multiboot-module";
                 reg = <0x80080000 0x800000>;
                 bootargs = "<DOMAIN 0 COMMAND LINE>";

<XEN COMMAND LINE>      dtuart=serial0 no-bootscrub
<DOMAIN 0 COMMAND LINE> root=/dev/mmcblk0 console=hvc0 rw debug init=/root/

/chosen/module@1/reg should match bootwrapper

Build boot-wrapper:
    boot-wrapper-aarch64.git$ ln -s ../xen.git/xen/xen Xen
    boot-wrapper-aarch64.git$ ln -s ../arm-dts.git/fast_models/rtsm_ve-aemv8a.dtb fdt.dtb
    boot-wrapper-aarch64.git$ ln -s ../linux.git/arch/arm64/boot/Image Image
    boot-wrapper-aarch64.git$ make CROSS_COMPILE=aarch64-linux-gnu- xen-system.axf

Run it:
$ RTSM_VE_AEMv8A <MODEL OPTIONS> boot-wrapper-aarch64.git/xen-system.axf


Can use u-boot command line "fdt" to setup boot-modules.


   fdt addr ${fdt_addr}
   fdt resize
   fdt set /chosen \#address-cells <1>
   fdt set /chosen \#size-cells <1>
   fdt mknod /chosen module@0
   fdt set /chosen/module@0 compatible "xen,linux-zimage" "xen,multiboot-module"
   fdt set /chosen/module@0 reg <${kernel_addr_r} 0x${filesize} >
   fdt set /chosen/module@0 bootargs "<DOMAIN 0 COMMAND LINE>"



WIP (Fu Wei, Linaro). Eventually boot via grub.efi...

Starting a guest

Linux or NetBSD.

Create rootfs. Either LVM image or raw file. Use debootstrap or other
distro mechanisms. Linaro rootfs?

Kernel: use any kernel with Xen support enabled, same kernel as dom0
is fine/convenient. Or a distro kernel if available.

Create a config file. See xl.cfg(5).
    $ cat xen.cfg
    kernel      = '/boot/vmlinuz'
    ramdisk     = '/boot/initrd.img' # optional

    vcpus       = '2'
    memory      = '512'

    extra       = "console=hvc0 earlyprintk debug root=/dev/xvda ro"
    disk        = ["phy:/dev/loop0,xvda,rw"] # or phy:/dev/mapper/VG-LV etc.

    name        = 'my-xen-domain'

    vif         = [ 'mac=8e:a7:8e:3c:f4:f6' ]

Start it:
    $ xl create xen.cfg

    $ xl console my-xen-domain

    $ xl list
    Name                                        ID   Mem VCPUs      State   Time(s)
    Domain-0                                     0   128     2     r-----       8.0
    my-xen-domain                                1   128     1     r-----       0.5

    $ xl -vvv [command...]

More info xl(1).

Logs /var/log/xen/xl-$name.log

Supporting a new platform

Basics should Just Work (tm). Platform interface in
xen/include/asm-arm/platform.h and existing implementations in

Hooks for SMP init (but prefer generic PSCI), platform quirks, device
blacklists etc.

Contributing to Xen

Workflow very similar to Linux kernel. Signed-off-by patches sent to, ideally CC maintainers (see MAINTAINERS file
in xen.git)

Release Cycle

Traditionally a ~9 month release cadence. Trying to pull this in to ~6
months. Xen 4.4 is due for release next week.

Xen 4.5 is therefore ~Aug/Sep. Expect code freeze ~6 weeks before.

More Information

Includes links to platform specific advice.

IRC: #xenarm on freenode.


set -x
mount -o remount,rw /
mount -t proc none /proc
mount -t sysfs none /sys
mount -t tmpfs none /run
mkdir /run/lock
#/etc/init.d/udev start
mount -t devtmpfs dev /dev
/sbin/udevd --daemon
udevadm trigger --action=add
mkdir /dev/pts
mount -t devpts none /dev/pts
hostname -F /etc/hostname

mknod -m 640 /dev/xconsole p
chown root:adm /dev/xconsole

/sbin/klogd -c 1 -x

/etc/init.d/xencommons start

echo 9 > /proc/sysrq-trigger 
cd /root
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
exec /bin/bash

LEG/Engineering/Virtualization/Xen_ARM_Guide (last modified 2014-03-17 16:58:40)