How to boot Xen with GRUB on AArch64 FVP_base model

The main purpose of this Wiki page is describing How to boot Xen and Dom0 kernel with GRUB2 on FVP_Base_AEMv8A(ARM FastModel emulator).

For this purpose, we need several components:

SD image (for FVP base module, arm64)
Firmware binary (bl1.bin, bl2.bin, bl31.bin)
UEFI binary (uefi_fvp-base.bin)
*GRUB2 binary (multiboot support)*
DTB (fvp-base-gicv2-psci.dtb)
*Xen binary*
Dom0 Linux kernel (xen support)
Dom0 Linux Initrd (option, will add this part in the future)

MMC Image

To boot the FVP_Base_AEMv8A into Dom0 console, we need a basic SD Image file included boot partition and rootfs.

For now, the best image for this test is The latest Linaro LEG Image.

Some Bootloader Binaries

We can get some required bootloader binary(Firmware and UEFI) here :





As a alternative, you also can get all the binaries above in the boot partition of MMC Image : Decompressing the image file, use "kpartx" command to map the image to loop device, then mount them on some dir you want:

$ gunzip -c vexpress64-leg-sd.img.gz > vexpress64-leg-sd-xen.img
$  sudo kpartx -av vexpress64-leg-sd-xen.img
[sudo] password for tekkamanninja: 
add map loop0p1 (252:0): 0 106432 linear /dev/loop0 63
add map loop0p2 (252:1): 0 4087808 linear /dev/loop0 106496
$ mkdir boot rootfs
$ sudo mount /dev/mapper/loop0p1 boot/


We already have it(fvp-base-gicv2-psci.dtb) in the boot partition of MMC Image. But we don't need to abstract this out of the Image, we will use that when we boot the Xen by grub.cfg. :-)

See grub.cfg below.


Preparing source code

Get upstream GRUB source code

git clone git://git.savannah.gnu.org/grub.git
git checkout master

Applying the patch

The GRUB multiboot support patch for aarch64(V2)

Please rename the file(attachment.bin) to Add-multiboot-module-command-support-in-linux-module.patch

git apply  Add-multiboot-module-command-support-in-linux-module.patch

Build the binary

Please download the toolchain here.

The content of the default.cfg :

search -l boot -s root
set prefix=($root)/grub/

Build steps :

GRUB_BUILD_CONFIG_V8=--target=aarch64-linux-gnu --with-platform=efi --prefix=${GRUB_INSTALL_DIR_V8}

make distclean
./configure ${GRUB_BUILD_CONFIG_V8}
make -j12
make install

./bin/grub-mkimage -v -c ${GRUB_DEFAULT_CFG} -o grub_v8.efi -O arm64-efi -p "efi" boot chain configfile configfile efinet ext2 fat gettext help hfsplus loadenv lsefi normal normal ntfs ntfscomp part_gpt part_msdos part_msdos read search search_fs_file search_fs_uuid search_label terminal terminfo tftp linux

The GRUB binary is ${GRUB_INSTALL_DIR_V8}/grub_v8.efi


Preparing source code

Get upstream Xen source code

git clone git://xenbits.xen.org/xen.git
git checkout master

Build the binary


git clean -fdx xen
make clean

Dom0 Linux kernel

Get upstream Linux source code

git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
git checkout v3.13

Build the binary

xen_defaultconfig : config.gz

gunzip -c config.gz > .config

export ARCH=arm64

make -j12 Image

Modify original LEG Image

Now we have all the components we need, so we can modify the image and boot the model.

The of grub.cfg :

set default="0"
set timeout=1
menuentry 'FVP base AEMV8 Xen test' {
    set debug=linux
    set root=(hd4,msdos1)
    multiboot /xen no-bootscrub console=dtuart conswitch=x dtuart=serial0 dom0_mem=512M dom0_max_vcpus=2
    module /Image console=hvc0 root=/dev/vda2 ro
    devicetree /fvp-base-gicv2-psci.dtb

The modification steps for boot partition:

#create EFI/BOOT dir in boot partition for grub_v8.efi binary
mkdir -p boot/EFI/BOOT
#copy grub_v8.efi binary
cp grub_v8.efi boot/EFI/BOOT/

#create grub dir in boot partition for grub menu
mkdir boot/grub
#copy grub.cfg config file
cp grub.cfg boot/grub/

#copy xen binary to boot partition
cp xen boot/
#copy Dom0 kernel image file to boot partition
cp Image boot/

FVP_Base_AEMv8A shell script for running model



sudo -u www-data  ARMLMD_LICENSE_FILE="8224@localhost" $rtsm_model                                             \
        -C bp.flashloader0.fname=$rtsm_uefi                             \
        -C pctl.startup=                                         \
        -C cache_state_modelled=0                                       \
        -C bp.secure_memory=0                                           \
        -C cluster0.NUM_CORES=4                                         \
        -C cluster1.NUM_CORES=4                                         \
        -C bp.pl011_uart0.untimed_fifos=1                               \
        -C bp.virtioblockdevice.image_path=$rtsm_mmc                    \
        -C bp.secureflashloader.fname=$rtsm_uefi_sec                    \
        -C bp.hostbridge.interfaceName=$nic                             \
        -C bp.smsc_91c111.enabled=true                                  \
        -C bp.smsc_91c111.mac_address=$mac                              \
        -C bp.flashloader1.fname=uefi-vars-v8.fd       \
        -C bp.flashloader1.fnameWrite=uefi-vars-v8.fd

FVP_Base_AEMv8A boot log

The default boot selection will start in  10 seconds
[1] Linaro disk image on virtio
        - VenHw(C5B9C74A-6D72-4719-99AB-C59F199091EB)/Image
        - Arguments: console=ttyAMA0 earlyprintk=pl011,0x1c090000 debug user_debug=31 loglevel=9 root=/dev/vda2
        - FDT: VenHw(C5B9C74A-6D72-4719-99AB-C59F199091EB)/fvp-base-gicv2-psci.dtb
        - LoaderType: Linux kernel with Local FDT
Global FDT Config
        - VenHw(C5B9C74A-6D72-4719-99AB-C59F199091EB)/fdt.dtb
[a] Boot Manager
[b] Shell
[c] Reboot
[d] Shutdown
Start: b

UEFI Interactive Shell v2.0. UEFI v2.40 (ARM Fixed Virtual Platform EFI Jan 24 2014 17:53:36, 0x00000000). Revision 1.02
Mapping table
      FS0: Alias(s):F8:
      FS1: Alias(s):HD12b:;BLK5:
     BLK3: Alias(s):
     BLK0: Alias(s):
     BLK1: Alias(s):
     BLK2: Alias(s):
     BLK4: Alias(s):
     BLK6: Alias(s):

Press ESC in 4 seconds to skip startup.nsh or any other key to continue.
Shell> grub_v8.efi
Welcome to GRUB!
                          GNU GRUB  version 2.02~beta2

 �*FVP base AEMV8 Xen test                                                    � 
 �                                                                            �
 �                                                                            �
 �                                                                            �
 �                                                                            �
 �                                                                            �
 �                                                                            �
 �                                                                            �
 �                                                                            �
 �                                                                            �
 �                                                                            �
 �                                                                            �
 �                                                                            � 

      Use the 
 and  keys to select which entry is highlighted.          
      Press enter to boot the selected OS, `e' to edit the commands       
      before booting or `c' for a command-line.  

loader/arm64/multiboot.c:539: Multiboot Image file size: 656816
loader/arm64/multiboot.c:545: Multiboot Image numpages: 161
loader/arm64/multiboot.c:560: Multiboot Image @ 0xd8000000
loader/arm64/multiboot.c:575: Multiboot Image cmdline @ 0xf6d28220 no-bootscrub
console=dtuart conswitch=x dtuart=serial0 dom0_mem=512M dom0_max_vcpus=2, size:
loader/arm64/multiboot.c:424: Module0 file size: 5473624
loader/arm64/multiboot.c:429: Module0 numpages: 1337
loader/arm64/multiboot.c:445: Module0 @ 0xd78dc000
loader/arm64/multiboot.c:461: Module0 cmdline @ 0xf6c67f40 BOOT_IMAGE=/Image
console=hvc0 root=/dev/vda2 ro, size: 50
loader/arm64/linux.c:100: allocating 10101 bytes for fdt
loader/arm64/multiboot.c:135: Multiboot Image cmdline : no-bootscrub
console=dtuart conswitch=x dtuart=serial0 dom0_mem=512M dom0_max_vcpus=2 @
0xf6d28220 size:91
loader/arm64/multiboot.c:151: Module0 @ 0xd78dc000 size:0x538558
loader/arm64/multiboot.c:165: Module0 compatible = xen,linux-zimage
xen,multiboot-module size = 38
loader/arm64/multiboot.c:176: Module0 reg = <0x0000d78dc000 0x000000538558>
loader/arm64/multiboot.c:181: Module0 cmdline : BOOT_IMAGE=/Image console=hvc0
root=/dev/vda2 ro @ 0xf6c67f40 size:50
loader/arm64/multiboot.c:227: Installed/updated FDT configuration table @
loader/arm64/multiboot.c:266: Jumping to Multiboot Image @ 0xd8000000
- UART enabled -
- CPU 00000000 booting -
- Current EL 00000008 -
- Xen starting at EL2 -
- Zero BSS -
- Setting up control registers -
- Turning on paging -
- Ready -
Checking for initrd in /chosen
RAM: 0000000080000000 - 00000000ffffffff
RAM: 0000000880000000 - 00000008ffffffff

MODULE[1]: 00000000f6e67000 - 00000000f6e69775 
MODULE[2]: 00000000d78dc000 - 00000000d7e14558 BOOT_IMAGE=/Image console=hvc0 root=/dev/vda2 ro
 RESVD[0]: 0000000080000000 - 0000000080010000

Command line: no-bootscrub console=dtuart conswitch=x dtuart=serial0 dom0_mem=512M dom0_max_vcpus=2
Placing Xen at 0x00000000ffe00000-0x0000000100000000
WARNING: only using 1 out of 2 memory banks
Looking for UART console serial0
 Xen 4.5-unstable
(XEN) Xen version 4.5-unstable (tekkamanninja@) (aarch64-linux-gnu-gcc (crosstool-NG linaro-1.13.1-4.8-2013.08 - Linaro GCC 2013.08) 4.8.2 20130805 (prerelease)) debug=y Tue Mar 11 20:28:50 CST 2014
(XEN) Latest ChangeSet: Thu Mar 6 11:32:48 2014 +0000 git:4888ac58-dirty
(XEN) Processor: 410fd0f0: "ARM Limited", variant: 0x0, part 0xd0f, rev 0x0
(XEN) 64-bit Execution:
(XEN)   Processor Features: 0000000001002222 0000000000000000
(XEN)     Exception Levels: EL3:64+32 EL2:64+32 EL1:64+32 EL0:64+32
(XEN)     Extensions: FloatingPoint AdvancedSIMD
(XEN)   Debug Features: 00000000f0f0f106 0000000000000000
(XEN)   Auxiliary Features: 0000000000000000 0000000000000000
(XEN)   Memory Model Features: 0000000000001122 0000000000000000
(XEN)   ISA Features:  0000000000000000 0000000000000000
(XEN) 32-bit Execution:
(XEN)   Processor Features: 00000131:10011011
(XEN)     Instruction Sets: AArch32 Thumb Thumb-2 Jazelle
(XEN)     Extensions: GenericTimer Security
(XEN)   Debug Features: 03010066
(XEN)   Auxiliary Features: 00000000
(XEN)   Memory Model Features: 10101105 20000000 01260000 02102111
(XEN)  ISA Features: 02101110 13112111 21232042 01112131 00011142 00000001
(XEN) Using PSCI for SMP bringup
(XEN) Generic Timer IRQ: phys=30 hyp=26 virt=27
(XEN) Using generic timer at 100000 KHz
(XEN) GIC initialization:
(XEN)         gic_dist_addr=000000002f000000
(XEN)         gic_cpu_addr=000000002c000000
(XEN)         gic_hyp_addr=000000002c010000
(XEN)         gic_vcpu_addr=000000002c02f000
(XEN)         gic_maintenance_irq=25
(XEN) GIC: 256 lines, 8 cpus, secure (IID 00000000).
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Allocated console ring of 64 KiB.
(XEN) Bringing up CPU1
- CPU 00000001 booting -
- Current EL 00000008 -
- Xen starting at EL2 -
- Setting up control registers -
- Turning on paging -
- Ready -
(XEN) CPU 1 booted.
(XEN) Bringing up CPU2
- CPU 00000002 booting -
- Current EL 00000008 -
- Xen starting at EL2 -
- Setting up control registers -
- Turning on paging -
- Ready -
(XEN) CPU 2 booted.
(XEN) Bringing up CPU3
- CPU 00000003 booting -
- Current EL 00000008 -
- Xen starting at EL2 -
- Setting up control registers -
- Turning on paging -
- Ready -
(XEN) CPU 3 booted.
(XEN) Bringing up CPU4
(XEN) Failed to bring up CPU4
(XEN) Failed to bring up CPU 4 (error -2)
(XEN) Bringing up CPU5
(XEN) Failed to bring up CPU5
(XEN) Failed to bring up CPU 5 (error -2)
(XEN) Bringing up CPU6
(XEN) Failed to bring up CPU6
(XEN) Failed to bring up CPU 6 (error -2)
(XEN) Bringing up CPU7
(XEN) Failed to bring up CPU7
(XEN) Failed to bring up CPU 7 (error -2)
(XEN) Brought up 4 CPUs
(XEN) Populate P2M 0xa0000000->0xc0000000 (1:1 mapping for dom0)
(XEN) Loading kernel from boot module 2
(XEN) Loading zImage from 00000000d78dc000 to 00000000a0080000-00000000a05b8558
(XEN) Loading dom0 DTB to 0x00000000a8000000-0x00000000a8002006
(XEN) Std. Loglevel: All
(XEN) Guest Loglevel: All
(XEN) *** Serial input -> DOM0 (type 'CTRL-x' three times to switch input to Xen)
(XEN) Freed 252kB init memory.
Initializing cgroup subsys cpu
Linux version 3.13.0 (tekkamanninja@Super-MAGI) (gcc version 4.8.2 20130805 (prerelease) (crosstool-NG linaro-1.13.1-4.8-2013.08 - Linaro GCC 2013.08) ) #4 SMP PREEMPT Fri Jan 24 20:57:09 CST 2014
CPU: AArch64 Processor [410fd0f0] revision 0
psci: probing function IDs from device-tree
PERCPU: Embedded 10 pages/cpu @ffffffc01ffdd000 s12288 r8192 d20480 u40960
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 129280
Kernel command line: BOOT_IMAGE=/Image console=hvc0 root=/dev/vda2 ro
PID hash table entries: 2048 (order: 2, 16384 bytes)
Dentry cache hash table entries: 65536 (order: 7, 524288 bytes)
Inode-cache hash table entries: 32768 (order: 6, 262144 bytes)
software IO TLB [mem 0xbb000000-0xbf000000] (64MB) mapped at [ffffffc01b000000-ffffffc01effffff]
Memory: 437532K/524288K available (3647K kernel code, 229K rwdata, 1280K rodata, 184K init, 166K bss, 86756K reserved)
Virtual kernel memory layout:
    vmalloc : 0xffffff8000000000 - 0xffffffbbffff0000   (245759 MB)
    vmemmap : 0xffffffbc02300000 - 0xffffffbc02a00000   (     7 MB)
    modules : 0xffffffbffc000000 - 0xffffffc000000000   (    64 MB)
    memory  : 0xffffffc000000000 - 0xffffffc020000000   (   512 MB)
      .init : 0xffffffc000551000 - 0xffffffc00057f000   (   184 kB)
      .text : 0xffffffc000080000 - 0xffffffc000550d34   (  4932 kB)
      .data : 0xffffffc00057f000 - 0xffffffc0005b8540   (   230 kB)
SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
Preemptible hierarchical RCU implementation.
        RCU restricting CPUs from NR_CPUS=8 to nr_cpu_ids=2.
NR_IRQS:64 nr_irqs:64 0
Architected cp15 and mmio timer(s) running at 100.00MHz (virt/phys).
sched_clock: 56 bits at 100MHz, resolution 10ns, wraps every 2748779069440ns
Console: colour dummy device 80x25
Calibrating delay loop (skipped), value calculated using timer frequency.. 200.00 BogoMIPS (lpj=1000000)
pid_max: default: 32768 minimum: 301
Mount-cache hash table entries: 256
hw perfevents: enabled with arm/armv8-pmuv3 PMU driver, 9 counters available
CPU1: Booted secondary processor
Brought up 2 CPUs
SMP: Total of 2 processors activated.
devtmpfs: initialized
Xen 4.5 support found, events_irq=31 gnttab_frame_pfn=b0000
xen:grant_table: Grant tables using version 1 layout
Grant table initialized
atomic64 test passed
regulator-dummy: no parameters
NET: Registered protocol family 16
Xen: initializing cpu0
Xen: initializing cpu1
of_amba_device_create(): amba_device_add() failed (-19) for /smb/motherboard/iofpga@3,00000000/sysctl@020000
vdso: 2 pages (1 code, 1 data) at base ffffffc000585000
hw-breakpoint: found 16 breakpoint and 16 watchpoint registers.
xen:swiotlb_xen: Warning: only able to allocate 4 MB for software IO TLB
software IO TLB [mem 0xba400000-0xba800000] (4MB) mapped at [ffffffc01a400000-ffffffc01a7fffff]
Serial: AMBA PL011 UART driver
1c0a0000.uart: ttyAMA1 at MMIO 0x1c0a0000 (irq = 38, base_baud = 0) is a PL011 rev2
1c0b0000.uart: ttyAMA2 at MMIO 0x1c0b0000 (irq = 39, base_baud = 0) is a PL011 rev2
1c0c0000.uart: ttyAMA3 at MMIO 0x1c0c0000 (irq = 40, base_baud = 0) is a PL011 rev2
bio: create slab <bio-0> at 0
xen:balloon: Initialising balloon driver
xen_balloon: Initialising balloon driver
3V3: 3300 mV 
SCSI subsystem initialized
Switched to clocksource arch_sys_counter
NET: Registered protocol family 2
TCP established hash table entries: 4096 (order: 3, 32768 bytes)
TCP bind hash table entries: 4096 (order: 4, 65536 bytes)
TCP: Hash tables configured (established 4096 bind 4096)
TCP: reno registered
UDP hash table entries: 256 (order: 1, 8192 bytes)
UDP-Lite hash table entries: 256 (order: 1, 8192 bytes)
NET: Registered protocol family 1
RPC: Registered named UNIX socket transport module.
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RPC: Registered tcp NFSv4.1 backchannel transport module.
fuse init (API version 7.22)
msgmni has been set to 854
io scheduler noop registered
io scheduler cfq registered (default)
xen:xen_evtchn: Event-channel device installed
console [hvc0] enabled
Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
blk-mq: CPU -> queue map
  CPU 0 -> Queue 0
  CPU 1 -> Queue 0
 vda: vda1 vda2
smc91x 1a000000.ethernet (unregistered net_device): smc91x: IOADDR ffffff8000054000 doesn't match configuration (300).
smc91x.c: v1.1, sep 22 2004 by Nicolas Pitre <nico@fluxnic.net>
smc91x 1a000000.ethernet eth0: SMC91C11xFD (rev 1) at ffffff8000054000 IRQ 47

smc91x 1a000000.ethernet eth0: Ethernet addr: 20:12:01:04:10:00
xen_netfront: Initialising Xen virtual ethernet driver
mousedev: PS/2 mouse device common for all mice
mmci-pl18x 1c050000.mmci: dummy supplies not allowed
mmci-pl18x 1c050000.mmci: mmc0: PL180 manf 41 rev0 at 0x1c050000 irq 41,42 (pio)
TCP: cubic registered
NET: Registered protocol family 17
EXT3-fs (vda2): error: couldn't mount because of unsupported optional features (240)
EXT2-fs (vda2): error: couldn't mount because of unsupported optional features (244)
EXT4-fs (vda2): INFO: recovery required on readonly filesystem
EXT4-fs (vda2): write access will be enabled during recovery
EXT4-fs (vda2): recovery complete
EXT4-fs (vda2): mounted filesystem with ordered data mode. Opts: (null)
VFS: Mounted root (ext4 filesystem) readonly on device 254:2.
Freeing unused kernel memory: 184K (ffffffc000551000 - ffffffc00057f000)
INIT: version 2.88 booting


LEG/Engineering/Grub2/Xen_booting_on_FVP_Base_AEMv8A (last modified 2014-04-19 17:32:41)