Introduction

Ian Campbell gave a great talk on using Xen on ARM during LCA14 in Macao.

The text-based 'slides' below serve as a sort of cook-book on how to get started using Xen.

Refer to the Xen wiki for more information: http://wiki.xen.org/wiki/Xen_ARM_with_Virtualization_Extensions

Slides

Components of a Xen system
==========================

- Xen Hypervisor
- Domain 0 Kernel (Linux)
- Device Tree
- Domain 0 Userspace
- Xen Tools

Getting Xen Source
==================

$ git clone git://xenbits.xen.org/xen.git

Branches:
  master:       Current development (tested)
  staging:      Current development (pre-test)
  
  stable-4.4:   Current stable release (imminent)
  staging-4.4:  Current stable release (pre-test)

Recommended for development: master unless more specific need
e.g. known prerequisite patches in staging.

Xen Hypervisor
==============

Can cross compile or build natively.

xen.git$ make xen XEN_TARGET_ARCH=arm32 CROSS_COMPILE=arm-linux-gnueabi- [-jN -s etc]
xen.git$ make xen XEN_TARGET_ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- [-jN -s etc]

Not much to configure, no Kconfig, menuconfig etc.

Common options:

debug=y
-------

xen.git$ make xen debug=y [XEN_TARG...etc]

Enabled during development cycles, disabled during rc for release.

Early printk
------------

xen.git$ make xen XEN_TARGET_ARCH=.. debug=y CONFIG_EARLY_PRINTK=<machine>

Where <machine> is:
  - vexpress
  - midway
  - sun7i
  - fastmodel
  - ... more...

See: docs/misc/arm/early-printk.txt. Requires debug=y which is the
default during development phases.

Easy to add support for any pl011 or 8250 platform, see
xen/arch/arm/Rules.mk. Other UARTs are more work but still trivial.

Can write options to .config. (I think, I never do....)

xen.git$ make dist-xen
  => Install to xen.git/dist/install for rsync, tar+scp etc.

xen.git$ sudo make install-xen
  => Install on local system

xen.git$ sudo make install-xen DESTDIR=/tmp/my-dir
  => Install to a staging area

Domain 0 Kernel (Linux)
=======================

Normal Linux kernel from e.g. torvalds/linux.git later than v3.8
required, recommend latest e.g. v3.13.

Must support native operation on the host platform. Use normal .config
and enable Xen options:
    CONFIG_XEN=y
    CONFIG_XEN_BLKDEV_FRONTEND=y
    CONFIG_XEN_BLKDEV_BACKEND=y
    CONFIG_XEN_NETDEV_FRONTEND=y
    CONFIG_XEN_NETDEV_BACKEND=y
    CONFIG_HVC_XEN=y
    CONFIG_HVC_XEN_FRONTEND=y
    CONFIG_XEN_DEV_EVTCHN=y
    CONFIG_XEN_BACKEND=y
    CONFIG_XENFS=y
    CONFIG_XEN_COMPAT_XENFS=y
    CONFIG_XEN_SYS_HYPERVISOR=y
    CONFIG_XEN_XENBUS_FRONTEND=y
    CONFIG_XEN_GNTDEV=y
    CONFIG_XEN_GRANT_DEV_ALLOC=y
    CONFIG_SWIOTLB_XEN=y
    CONFIG_XEN_PRIVCMD=y

make Image/zImage in the normal way. Producing arch/arm/boot/zImage or
arch/arm64/boot/Image

Device Tree
===========

Unmodified host device tree. e.g. from linux/arch/arm/boot/dts

Domain 0 Userspace
==================

Your favourite Linux distribution.

Consider using LVM and leaving most of VG free for guest disks, or can
use image back guest filesystem via /dev/loop.

Configure bridging using distro tools, suggest name "xenbr0" for ease
of use.

Xen Tools
=========

Build natively or cross (given a suitable cross build environment) or
native.

xen.git: ./configure [--prefix=... etc]
xen.git: make tools [-jN -s etc]

xen.git: make dist-tools
xen.git: make install-tools
xen.git: make install-tools DESTDIR=/tmp/my-dir

Configure /etc/init.d/xencommons to run on boot.

Make sure you have a getty on hvc0.

Workarounds:
- cp /usr/share/misc/config.* .
  => http://lists.xen.org/archives/html/xen-devel/2014-03/msg00003.html
     http://bugs.xenproject.org/xen/mid/%3C1393648440-21407-1-git-send-email-ian.campbell@citrix.com%3E/raw
- https://bugs.launchpad.net/linaro-aarch64/+bug/1169164

Booting
=======

Xen hypervisor binary is compatible with the Linux boot protocol for
arm or arm64 (as appropriate):

- xen.git/docs/misc/arm/booting.txt

Pass domain 0 kernel as a "boot module":

- xen.git/docs/misc/arm/device-tree/booting.txt

4 potential configurations:

- Fastmodel w/ boot-wrapper w/ semihosting (arm32)
- Fastmodel w/ boot-wrapper w/o semihosting (arm64)
- u-boot
- UEFI

Command line options:

- Xen:
  - On models use "no-bootscrub".
  - dtuart= references device tree UART node.
  - http://xenbits.xen.org/docs/unstable/misc/xen-command-line.html

- Linux:
  - console=hvc0
  - Otherwise as normal (root= etc)

Fastmodel w/ boot-wrapper w/ semihosting (arm32)
------------------------------------------------

- http://wiki.xen.org/wiki/Xen_ARM_with_Virtualization_Extensions/FastModels#arm32
- git://xenbits.xen.org/people/ianc/boot-wrapper.git

$ RTSM_VE_Cortex-A15x2 -C cluster.cpu0.semihosting-cmd_line="
    --kernel xen.git/xen/xen \
    --module linux.git/arch/arm/boot/zImage <DOMAIN 0 COMMAND LINE>
    --dtb rtsm_ve-cortex_a15x2.dtb -- <XEN COMMAND LINE>"
  <MODEL OPTIONS> boot-wrapper.git/linux-system-semi.axf

Workaround: "mmc_select_voltage only if host->ocr_avail_mmc"
 => http://lkml.org/lkml/2013/1/17/269

Fastmodel w/ boot-wrapper w/o semihosting (arm64)
-------------------------------------------------

No semihosting boot-wrapper for arm64, so more complicated.

- http://wiki.xen.org/wiki/Xen_ARM_with_Virtualization_Extensions/FastModels#arm64
- git://xenbits.xen.org/people/ianc/boot-wrapper-aarch64.git

Need to modify DTS to add boot module:

    chosen {
        #address-cells = <1>;
        #size-cells = <1>;

        xen,xen-bootargs = "<XEN COMMAND LINE>";
        module@1 {
                 compatible = "xen,linux-zimage", "xen,multiboot-module";
                 reg = <0x80080000 0x800000>;
                 bootargs = "<DOMAIN 0 COMMAND LINE>";
        };
    };

<XEN COMMAND LINE>      dtuart=serial0 no-bootscrub
<DOMAIN 0 COMMAND LINE> root=/dev/mmcblk0 console=hvc0 rw debug init=/root/init.sh

/chosen/module@1/reg should match bootwrapper model.lds.

Build boot-wrapper:
    boot-wrapper-aarch64.git$ ln -s ../xen.git/xen/xen Xen
    boot-wrapper-aarch64.git$ ln -s ../arm-dts.git/fast_models/rtsm_ve-aemv8a.dtb fdt.dtb
    boot-wrapper-aarch64.git$ ln -s ../linux.git/arch/arm64/boot/Image Image
    boot-wrapper-aarch64.git$ make CROSS_COMPILE=aarch64-linux-gnu- xen-system.axf

Run it:
$ RTSM_VE_AEMv8A <MODEL OPTIONS> boot-wrapper-aarch64.git/xen-system.axf

u-boot
------

Can use u-boot command line "fdt" to setup boot-modules.

- http://wiki.xen.org/wiki/Xen_ARM_with_Virtualization_Extensions#Boot_Modules

   fdt addr ${fdt_addr}
   fdt resize
   
   fdt set /chosen \#address-cells <1>
   fdt set /chosen \#size-cells <1>
   
   fdt mknod /chosen module@0
   fdt set /chosen/module@0 compatible "xen,linux-zimage" "xen,multiboot-module"
   fdt set /chosen/module@0 reg <${kernel_addr_r} 0x${filesize} >
   fdt set /chosen/module@0 bootargs "<DOMAIN 0 COMMAND LINE>"

- http://wiki.xen.org/wiki/Xen_ARM_with_Virtualization_Extensions/Allwinner#Boot_script

UEFI
----

WIP (Fu Wei, Linaro). Eventually boot via grub.efi...

Starting a guest
================

Linux or NetBSD.

Create rootfs. Either LVM image or raw file. Use debootstrap or other
distro mechanisms. Linaro rootfs?

Kernel: use any kernel with Xen support enabled, same kernel as dom0
is fine/convenient. Or a distro kernel if available.

Create a config file. See xl.cfg(5).
    $ cat xen.cfg
    kernel      = '/boot/vmlinuz'
    ramdisk     = '/boot/initrd.img' # optional

    vcpus       = '2'
    memory      = '512'

    extra       = "console=hvc0 earlyprintk debug root=/dev/xvda ro"
    disk        = ["phy:/dev/loop0,xvda,rw"] # or phy:/dev/mapper/VG-LV etc.

    name        = 'my-xen-domain'

    vif         = [ 'mac=8e:a7:8e:3c:f4:f6' ]

Start it:
    $ xl create xen.cfg

Console:
    $ xl console my-xen-domain

List:
    $ xl list
    Name                                        ID   Mem VCPUs      State   Time(s)
    Domain-0                                     0   128     2     r-----       8.0
    my-xen-domain                                1   128     1     r-----       0.5

Debug:
    $ xl -vvv [command...]

More info xl(1).

Logs /var/log/xen/xl-$name.log

Supporting a new platform
=========================

Basics should Just Work (tm). Platform interface in
xen/include/asm-arm/platform.h and existing implementations in
xen/arch/arm/platforms/*.c.

Hooks for SMP init (but prefer generic PSCI), platform quirks, device
blacklists etc.

Contributing to Xen
===================

Workflow very similar to Linux kernel. Signed-off-by patches sent to
xen-devel@lists.xen.org, ideally CC maintainers (see MAINTAINERS file
in xen.git)

http://wiki.xen.org/wiki/Submitting_Xen_Patches

Release Cycle
=============

Traditionally a ~9 month release cadence. Trying to pull this in to ~6
months. Xen 4.4 is due for release next week.

Xen 4.5 is therefore ~Aug/Sep. Expect code freeze ~6 weeks before.

More Information
================

http://wiki.xen.org/wiki/Xen_ARM_with_Virtualization_Extensions

http://xenbits.xen.org/docs

Includes links to platform specific advice.

IRC: #xenarm on freenode.

http://lists.xenproject.org/mailman/listinfo/xen-devel

Appendix
========

/root/init.sh:
#!/bin/sh
set -x
mount -o remount,rw /
mount -t proc none /proc
mount -t sysfs none /sys
mount -t tmpfs none /run
mkdir /run/lock
#/etc/init.d/udev start
mount -t devtmpfs dev /dev
/sbin/udevd --daemon
udevadm trigger --action=add
mkdir /dev/pts
mount -t devpts none /dev/pts
hostname -F /etc/hostname

mknod -m 640 /dev/xconsole p
chown root:adm /dev/xconsole

/sbin/klogd -c 1 -x
/usr/sbin/syslogd 

/etc/init.d/xencommons start

echo 9 > /proc/sysrq-trigger 
cd /root
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
exec /bin/bash

LEG/Engineering/Virtualization/Xen_ARM_Guide (last modified 2014-03-17 16:58:40)