Introduction
Ian Campbell gave a great talk on using Xen on ARM during LCA14 in Macao.
The text-based 'slides' below serve as a sort of cook-book on how to get started using Xen.
Refer to the Xen wiki for more information: http://wiki.xen.org/wiki/Xen_ARM_with_Virtualization_Extensions
Slides
Components of a Xen system ========================== - Xen Hypervisor - Domain 0 Kernel (Linux) - Device Tree - Domain 0 Userspace - Xen Tools Getting Xen Source ================== $ git clone git://xenbits.xen.org/xen.git Branches: master: Current development (tested) staging: Current development (pre-test) stable-4.4: Current stable release (imminent) staging-4.4: Current stable release (pre-test) Recommended for development: master unless more specific need e.g. known prerequisite patches in staging. Xen Hypervisor ============== Can cross compile or build natively. xen.git$ make xen XEN_TARGET_ARCH=arm32 CROSS_COMPILE=arm-linux-gnueabi- [-jN -s etc] xen.git$ make xen XEN_TARGET_ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- [-jN -s etc] Not much to configure, no Kconfig, menuconfig etc. Common options: debug=y ------- xen.git$ make xen debug=y [XEN_TARG...etc] Enabled during development cycles, disabled during rc for release. Early printk ------------ xen.git$ make xen XEN_TARGET_ARCH=.. debug=y CONFIG_EARLY_PRINTK=<machine> Where <machine> is: - vexpress - midway - sun7i - fastmodel - ... more... See: docs/misc/arm/early-printk.txt. Requires debug=y which is the default during development phases. Easy to add support for any pl011 or 8250 platform, see xen/arch/arm/Rules.mk. Other UARTs are more work but still trivial. Can write options to .config. (I think, I never do....) xen.git$ make dist-xen => Install to xen.git/dist/install for rsync, tar+scp etc. xen.git$ sudo make install-xen => Install on local system xen.git$ sudo make install-xen DESTDIR=/tmp/my-dir => Install to a staging area Domain 0 Kernel (Linux) ======================= Normal Linux kernel from e.g. torvalds/linux.git later than v3.8 required, recommend latest e.g. v3.13. Must support native operation on the host platform. Use normal .config and enable Xen options: CONFIG_XEN=y CONFIG_XEN_BLKDEV_FRONTEND=y CONFIG_XEN_BLKDEV_BACKEND=y CONFIG_XEN_NETDEV_FRONTEND=y CONFIG_XEN_NETDEV_BACKEND=y CONFIG_HVC_XEN=y CONFIG_HVC_XEN_FRONTEND=y CONFIG_XEN_DEV_EVTCHN=y CONFIG_XEN_BACKEND=y CONFIG_XENFS=y CONFIG_XEN_COMPAT_XENFS=y CONFIG_XEN_SYS_HYPERVISOR=y CONFIG_XEN_XENBUS_FRONTEND=y CONFIG_XEN_GNTDEV=y CONFIG_XEN_GRANT_DEV_ALLOC=y CONFIG_SWIOTLB_XEN=y CONFIG_XEN_PRIVCMD=y make Image/zImage in the normal way. Producing arch/arm/boot/zImage or arch/arm64/boot/Image Device Tree =========== Unmodified host device tree. e.g. from linux/arch/arm/boot/dts Domain 0 Userspace ================== Your favourite Linux distribution. Consider using LVM and leaving most of VG free for guest disks, or can use image back guest filesystem via /dev/loop. Configure bridging using distro tools, suggest name "xenbr0" for ease of use. Xen Tools ========= Build natively or cross (given a suitable cross build environment) or native. xen.git: ./configure [--prefix=... etc] xen.git: make tools [-jN -s etc] xen.git: make dist-tools xen.git: make install-tools xen.git: make install-tools DESTDIR=/tmp/my-dir Configure /etc/init.d/xencommons to run on boot. Make sure you have a getty on hvc0. Workarounds: - cp /usr/share/misc/config.* . => http://lists.xen.org/archives/html/xen-devel/2014-03/msg00003.html http://bugs.xenproject.org/xen/mid/%3C1393648440-21407-1-git-send-email-ian.campbell@citrix.com%3E/raw - https://bugs.launchpad.net/linaro-aarch64/+bug/1169164 Booting ======= Xen hypervisor binary is compatible with the Linux boot protocol for arm or arm64 (as appropriate): - xen.git/docs/misc/arm/booting.txt Pass domain 0 kernel as a "boot module": - xen.git/docs/misc/arm/device-tree/booting.txt 4 potential configurations: - Fastmodel w/ boot-wrapper w/ semihosting (arm32) - Fastmodel w/ boot-wrapper w/o semihosting (arm64) - u-boot - UEFI Command line options: - Xen: - On models use "no-bootscrub". - dtuart= references device tree UART node. - http://xenbits.xen.org/docs/unstable/misc/xen-command-line.html - Linux: - console=hvc0 - Otherwise as normal (root= etc) Fastmodel w/ boot-wrapper w/ semihosting (arm32) ------------------------------------------------ - http://wiki.xen.org/wiki/Xen_ARM_with_Virtualization_Extensions/FastModels#arm32 - git://xenbits.xen.org/people/ianc/boot-wrapper.git $ RTSM_VE_Cortex-A15x2 -C cluster.cpu0.semihosting-cmd_line=" --kernel xen.git/xen/xen \ --module linux.git/arch/arm/boot/zImage <DOMAIN 0 COMMAND LINE> --dtb rtsm_ve-cortex_a15x2.dtb -- <XEN COMMAND LINE>" <MODEL OPTIONS> boot-wrapper.git/linux-system-semi.axf Workaround: "mmc_select_voltage only if host->ocr_avail_mmc" => http://lkml.org/lkml/2013/1/17/269 Fastmodel w/ boot-wrapper w/o semihosting (arm64) ------------------------------------------------- No semihosting boot-wrapper for arm64, so more complicated. - http://wiki.xen.org/wiki/Xen_ARM_with_Virtualization_Extensions/FastModels#arm64 - git://xenbits.xen.org/people/ianc/boot-wrapper-aarch64.git Need to modify DTS to add boot module: chosen { #address-cells = <1>; #size-cells = <1>; xen,xen-bootargs = "<XEN COMMAND LINE>"; module@1 { compatible = "xen,linux-zimage", "xen,multiboot-module"; reg = <0x80080000 0x800000>; bootargs = "<DOMAIN 0 COMMAND LINE>"; }; }; <XEN COMMAND LINE> dtuart=serial0 no-bootscrub <DOMAIN 0 COMMAND LINE> root=/dev/mmcblk0 console=hvc0 rw debug init=/root/init.sh /chosen/module@1/reg should match bootwrapper model.lds. Build boot-wrapper: boot-wrapper-aarch64.git$ ln -s ../xen.git/xen/xen Xen boot-wrapper-aarch64.git$ ln -s ../arm-dts.git/fast_models/rtsm_ve-aemv8a.dtb fdt.dtb boot-wrapper-aarch64.git$ ln -s ../linux.git/arch/arm64/boot/Image Image boot-wrapper-aarch64.git$ make CROSS_COMPILE=aarch64-linux-gnu- xen-system.axf Run it: $ RTSM_VE_AEMv8A <MODEL OPTIONS> boot-wrapper-aarch64.git/xen-system.axf u-boot ------ Can use u-boot command line "fdt" to setup boot-modules. - http://wiki.xen.org/wiki/Xen_ARM_with_Virtualization_Extensions#Boot_Modules fdt addr ${fdt_addr} fdt resize fdt set /chosen \#address-cells <1> fdt set /chosen \#size-cells <1> fdt mknod /chosen module@0 fdt set /chosen/module@0 compatible "xen,linux-zimage" "xen,multiboot-module" fdt set /chosen/module@0 reg <${kernel_addr_r} 0x${filesize} > fdt set /chosen/module@0 bootargs "<DOMAIN 0 COMMAND LINE>" - http://wiki.xen.org/wiki/Xen_ARM_with_Virtualization_Extensions/Allwinner#Boot_script UEFI ---- WIP (Fu Wei, Linaro). Eventually boot via grub.efi... Starting a guest ================ Linux or NetBSD. Create rootfs. Either LVM image or raw file. Use debootstrap or other distro mechanisms. Linaro rootfs? Kernel: use any kernel with Xen support enabled, same kernel as dom0 is fine/convenient. Or a distro kernel if available. Create a config file. See xl.cfg(5). $ cat xen.cfg kernel = '/boot/vmlinuz' ramdisk = '/boot/initrd.img' # optional vcpus = '2' memory = '512' extra = "console=hvc0 earlyprintk debug root=/dev/xvda ro" disk = ["phy:/dev/loop0,xvda,rw"] # or phy:/dev/mapper/VG-LV etc. name = 'my-xen-domain' vif = [ 'mac=8e:a7:8e:3c:f4:f6' ] Start it: $ xl create xen.cfg Console: $ xl console my-xen-domain List: $ xl list Name ID Mem VCPUs State Time(s) Domain-0 0 128 2 r----- 8.0 my-xen-domain 1 128 1 r----- 0.5 Debug: $ xl -vvv [command...] More info xl(1). Logs /var/log/xen/xl-$name.log Supporting a new platform ========================= Basics should Just Work (tm). Platform interface in xen/include/asm-arm/platform.h and existing implementations in xen/arch/arm/platforms/*.c. Hooks for SMP init (but prefer generic PSCI), platform quirks, device blacklists etc. Contributing to Xen =================== Workflow very similar to Linux kernel. Signed-off-by patches sent to xen-devel@lists.xen.org, ideally CC maintainers (see MAINTAINERS file in xen.git) http://wiki.xen.org/wiki/Submitting_Xen_Patches Release Cycle ============= Traditionally a ~9 month release cadence. Trying to pull this in to ~6 months. Xen 4.4 is due for release next week. Xen 4.5 is therefore ~Aug/Sep. Expect code freeze ~6 weeks before. More Information ================ http://wiki.xen.org/wiki/Xen_ARM_with_Virtualization_Extensions http://xenbits.xen.org/docs Includes links to platform specific advice. IRC: #xenarm on freenode. http://lists.xenproject.org/mailman/listinfo/xen-devel Appendix ======== /root/init.sh: #!/bin/sh set -x mount -o remount,rw / mount -t proc none /proc mount -t sysfs none /sys mount -t tmpfs none /run mkdir /run/lock #/etc/init.d/udev start mount -t devtmpfs dev /dev /sbin/udevd --daemon udevadm trigger --action=add mkdir /dev/pts mount -t devpts none /dev/pts hostname -F /etc/hostname mknod -m 640 /dev/xconsole p chown root:adm /dev/xconsole /sbin/klogd -c 1 -x /usr/sbin/syslogd /etc/init.d/xencommons start echo 9 > /proc/sysrq-trigger cd /root export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin exec /bin/bash
LEG/Engineering/Virtualization/Xen_ARM_Guide (last modified 2014-03-17 16:58:40)