This wiki is address to enable lxc and estimate it's latest status on arm64 platform, and also try to find some workaround for it on arm64 architecture.

Build Target Kernel Image

Preparation

Now, assuming you're in your target board already, and using Ubuntu. We encourage you to use the latest version of Ubuntu, my preference is always the developing version. Above all, please update the apt-get by follow.

sudo apt-get update

If you are building your kernel target on D02 board[1], you have better to remove the original gcc in distribution firstly, otherwise it will cause a variety building errors.

sudo apt-get remove --purge gcc
sudo apt-get -y autoremove

Before building kernel target image natively, you need install kernel build-deps firstly as follows.

sudo apt-get install -y make gcc bc libncurses5-dev

And then you need to get the main line kernel[2] 3.19.0 source code as follows for building.

mkdir ~/workdir
cd ~/workdir
git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
git checkout bfa76d49

To resolve issues on apparmor and overlayfs to pass all lxc tests better, firstly need to apply a patch[3] modified by Ubuntu based on pure kernel 3.19.0. For more detail information about the issues, please refer to latest section "Issues Description" in this page.

cd ~/workdir
git clone https://github.com/hisilicon/patches.git
cd ~/workdir/linux
git am < ~/workdir/patches/0001-resolved-apparmor-overlayfs-issues-for-lxc-tests.patch

Then you also need do 'make menuconfig' as follows to config your kernel configurations firstly so that your target Image can support basic lxc's dependent components, e.g.: cgroup, namespace, chroot, NAT, VNET, Overlayfs, Apparmor and etc.

cd ~/workdir/linux
make mrproper
make defconfig
make menuconfig

Kernel config

General setup  --->
    ...
    -*- Control Group support  --->
        [ ]   Example debug cgroup subsystem                                    
        [*]   Freezer cgroup subsystem                                          
        [*]   Device controller for cgroups                                     
        [*]   Cpuset support                                                    
        [*]     Include legacy /proc/<pid>/cpuset file                          
        [*]   Simple CPU accounting cgroup subsystem                            
        [*]   Memory Resource Controller for Control Groups                     
        [*]     Memory Resource Controller Swap Extension                       
        [*]       Memory Resource Controller Swap Extension enabled by default  
        [*]     Memory Resource Controller Kernel Memory accounting             
        [*]   HugeTLB Resource Controller for Control Groups                    
        [*]   Enable perf_event per-cpu per-container group (cgroup) monitoring
        [*]   Group CPU scheduler  --->
            -*-   Group scheduling for SCHED_OTHER                 
            [*]     CPU bandwidth provisioning for FAIR_GROUP_SCHED
            [*]   Group scheduling for SCHED_RR/FIFO
        [*]   Block IO controller                   
        [ ]     Enable Block IO controller debugging
    -*- Namespaces support
        [*]   UTS namespace
        [*]   IPC namespace
        [*]   User namespace
        [*]   PID Namespaces
        [*]   Network namespace
    ...
[*] Networking support  --->
    Networking options  --->
        ...
        <*> 802.1d Ethernet Bridging                        
        [*]   IGMP/MLD snooping                             
        [*]   VLAN filtering                                
        <*> 802.1Q/802.1ad VLAN Support                     
        [*]   GVRP (GARP VLAN Registration Protocol) support
        [*]   MVRP (Multiple VLAN Registration Protocol) support
        ...
    ...
Device Drivers  --->
    ...
    [*] Network device support  --->
        ...
        <*>     MAC-VLAN support                             
        <*>       MAC-VLAN based tap driver                  
        < >     Virtual eXtensible Local Area Network (VXLAN)
        < >     Network console logging support              
        <*>     Universal TUN/TAP device driver support      
        <*>     Virtual ethernet pair device                 
        <*>     Virtio network driver
    Input device support  --->
    Character devices  --->
        ...
        -*- Unix98 PTY support
        [*]   Support multiple instances of devpts
        ...
    ...

Default D01\D02[1] kernel don't support iptables\NAT\VNET\Ethernet Bridge, so we'd enable them too by activating following kernel options:

[*] Networking support  --->
    Networking options  --->
        ...
        [*] TCP/IP networking                                      
        [ ]   IP: multicasting                                     
        [ ]   IP: advanced router                                  
        [*]   IP: kernel level autoconfiguration                   
        [*]     IP: DHCP support                                   
        [*]     IP: BOOTP support                                  
        [ ]     IP: RARP support                                   
        < >   IP: tunneling                                        
        < >   IP: GRE demultiplexer                                
        [ ]   IP: TCP syncookie support                            
        < >   Virtual (secure) IP: tunneling                       
        < >   IP: Foo (IP protocols) over UDP                      
        < >   Generic Network Virtualization Encapsulation (Geneve)
        < >   IP: AH transformation                                
        < >   IP: ESP transformation                               
        < >   IP: IPComp transformation                            
        <*>   IP: IPsec transport mode                             
        <*>   IP: IPsec tunnel mode                                
        <*>   IP: IPsec BEET mode                                  
        < >   Large Receive Offload (ipv4/tcp)                     
        <*>   INET: socket monitoring interface
        ...
        [*] Network packet filtering framework (Netfilter)  --->
            [ ]   Network packet filtering debugging
            [*]   Advanced netfilter configuration            
                <*>     Bridged IP/ARP packets filtering
            Core Netfilter Configuration  --->
                ...
                <*> Netfilter connection tracking support
                ...
                [*] Supply CT list in procfs (OBSOLETE)
                ...
                -*- Netfilter Xtables support (required for ip_tables)
                      *** Xtables combined modules ***
                ...
                -*-   "SNAT and DNAT" targets support
                ...
                      *** Xtables matches ***
                <*>   "addrtype" address type match support
                ...
                <*>   "comment" match support
                ...
                <*>   "conntrack" connection tracking match support
                ...
                <*>   "hl" hoplimit/TTL match support
                ...
                <*>   "limit" match support
                ...
                <*>   "multiport" Multiple port match support
                ...
                <*>   "recent" match support
                ...
            < >   IP set support  ----             
            < >   IP virtual server support  ----  
                  IP: Netfilter Configuration  --->
                      <*> IPv4 connection tracking support (required for NAT)        
                      [*]   proc/sysctl compatibility with old connection tracking
                      < > ARP packet logging                                      
                      < > IPv4 packet logging                                     
                      < > IPv4 packet rejection                                   
                      -*- IPv4 NAT                                                
                      <*>   IPv4 masquerade support                               
                      <*> IP tables support (required for filtering/masq/NAT)     
                      < >   "ah" match support                                    
                      < >   "ecn" match support                                   
                      < >   "ttl" match support                                   
                      < >   Packet filtering                                      
                      < >   SYNPROXY target support                               
                      <*>   iptables NAT support
                      ...
            <*>   Ethernet Bridge tables (ebtables) support  --->
        ...

And also need to enable overlayfs and apparmor\lsm options as follows.

File systems  --->
    <*> Overlay filesystem support
    [*] Overlayfs filesystem (V1) format support
...
Security options  --->
    [*] Enable access key retention support                          
    [ ]   Enable register of persistent per-UID keyrings             
    [ ]   Large payload keys                                         
    < >   TRUSTED KEYS                                               
    < >   ENCRYPTED KEYS                                             
    [ ]   Enable the /proc/keys file by which keys may be viewed     
    [ ] Restrict unprivileged access to the kernel syslog            
    [*] Enable different security models                             
    -*- Enable the securityfs filesystem                             
    -*- Socket and Networking Security Hooks                         
    [ ]   XFRM (IPSec) Networking Security Hooks                     
    -*- Security hooks for pathname based access control             
    (32768) Low address space for LSM to protect from user allocation
    [*] NSA SELinux Support                                          
    [ ]   NSA SELinux boot parameter                                 
    [ ]   NSA SELinux runtime disable                                
    [*]   NSA SELinux Development Support                            
    [*]   NSA SELinux AVC Statistics                                 
    (1)   NSA SELinux checkreqprot default value                     
    [ ]   NSA SELinux maximum supported policy format version        
    [ ] Simplified Mandatory Access Control Kernel Support           
    [ ] TOMOYO Linux Support                                         
    [*] AppArmor support                                             
    (1)   AppArmor boot parameter default value                      
    [ ]   enable debug statistics                                    
    [*]   Set init to unconfined on boot                             
    [*]   enable introspection of sha1 hashes for loaded profiles    
    [*]     Enable policy hash introspection by default              
    [ ] Yama support                                                 
    [*] Integrity subsystem                                          
    [ ]   Digital signature verification using multiple keyrings     
    [*]   Enables integrity auditing support                         
    [*]   Integrity Measurement Architecture(IMA)                    
            Default template (ima-ng (default))  --->                
            Default integrity hash algorithm (SHA1 (default))  --->  
    [ ]     Appraise integrity measurements                          
    [ ]   EVM support                                                
        Default security module (AppArmor)  --->

For above configurations, the IPV6 options are not necessary, you can select or unselect them according to your requirements.

Kernel building

Now, you can try to build and install your kernel image as follows, and there must be more than 500M free space in your boot partition at least.

make Image -j16
make modules -j16
make modules_install -j16
make install

If it's successful, then you can get all your target binary files (Image\initrd\dtb) in /boot/ and your new libraries files in /lib/modules/.

Boot Your New Image

When rebooting your board, you can press any key to interrupt uboot, then input following commands to boot the binaries built by yourself.

scsi init
ext4ls scsi 0 /
setenv bootargs 'rdinit=/init console=ttyS0,115200n8 root=/dev/sda2 rw rootwait' //the 'root' should be set correctly depending on your real status.
ext4load scsi 0 ${kernel_addr_r} uImage-3.19.0+
ext4load scsi 0 ${ramdisk_addr_r} uInitrd-3.19.0+
ext4load scsi 0 ${fdt_addr_r} apm-mustang-3.19.0+.dtb
bootm ${kernel_addr_r} ${ramdisk_addr_r} ${fdt_addr_r}

How to use LXC

From here, all following commands are supposedly run in your target board with your own kernel Image\dtb files.

  • Firstly, add following strings into /etc/apt/sources.list for Ubuntu vivid system.

deb http://ports.ubuntu.com/ubuntu-ports/ <your system codename, e.g. utopic> main restricted universe multiverse
deb-src http://ports.ubuntu.com/ubuntu-ports/ <your system codename, e.g. utopic> main restricted universe multiverse
deb http://ports.ubuntu.com/ubuntu-ports/ <your system codename, e.g. utopic>-updates main restricted universe multiverse
deb-src http://ports.ubuntu.com/ubuntu-ports/ <your system codename, e.g. utopic>-updates main restricted universe multiverse
deb http://ports.ubuntu.com/ubuntu-ports/ <your system codename, e.g. utopic>-security main restricted universe multiverse
deb-src http://ports.ubuntu.com/ubuntu-ports/ <your system codename, e.g. utopic>-security main restricted universe multiverse
  • Secondly, do "sudo apt-get update" to update apt sources.

Install lxc

Then install lxc and it's dependences.

 sudo apt-get install -y lxc

use lxc

* check whether you have all needed configurations and applications in place, you must make sure each checked items are green firstly.

sudo lxc-checkconfig

if you intend to use, make sure your have CONFIG_IKCONFIG_PROC enable when build your kernel. * check whether you have lxcbr0 bridge up, all traffic of guest OS will go through this bridge if you choose veth as network type. * create your first container by running

sudo lxc-create -t ubuntu -n mycontainer 

The debootstrap takes a while to download all packages from Ubuntu's archive, so it all depends on your network connectivity. the reason I choose ubuntu as the template is because Ubuntu supports LXC very well. * Start your first container just created

sudo lxc-start -n mycontainer 

* Attach your container to use it

sudo lxc-attach -n mycontainer

or

sudo lxc-console -n mycontainer

If everything goes smooth, you'll have login promote, default username:password is ubuntu:ubuntu.

Install, Build and Debug lxc-tests

Install The lxc-tests

You can install the lxc-tests suit as follows:

sudo apt-get install -y lxc-tests

You can get all test cases by

ls /usr/bin/lxc-test-*

Get Source Code and Debug

And you can download the lxc-tests source code and build them as follows:

sudo apt-get install -y dpkg-dev
mkdir lxc
cd lxc
apt-get source lxc-tests
sudo apt-get install build-dep lxc-tests
cd lxc-x.x.x
./autogen.sh
./configure --enable-tests
./make
./sudo make install

You will get the new test binaries from /usr/local/bin/lxc-test-*, and you can debug the test case with source code in src/tests, but before you start to run any binaries built by yourself, you must firstly make sure the /usr/lib/liblxc.so.1.1.2 has been replaced by /usr/local/lib/liblxc.so.1.1.2 correctly, otherwise the binaries will report undefined symbol error.

Validate All lxc-tests Cases on APM

You can run following commands in shell to validate all lxc-tests cases on ARM platform.

sudo su
cd /tmp
rm -rf *.FAILED
for i in /usr/bin/lxc-test-*; do $i || touch $(basename $i).FAILED; done

If there is not any new *.FAILED file, this means all lxc-tests cases passed successfully, you can get the test result by following command.

ls *.FAILED

if you still get some failures on your board, the ls will indicate which tests are still failing.

Some Conclusions

According to massive testing on different architecture machine\kernel configuration\distributions, we can get some conclusions as follows:

1. All lxc-tests cases can pass successfully on Ubuntu trusty in any Intel architecture machine.
2. All lxc-tests cases can pass successfully on Ubuntu trusty\vivid in APM Arm platform.
3. Following lxc-tests cases will fail on D02 ARM board[1] with D02 kernel tree[4] and APM Arm platform with pure kernel tree 3.9.0-rc4, 3.9.0 and 4.0.0[2].
    lxc-test-apparmor
    lxc-test-apparmor-mount
    lxc-test-autostart
    lxc-test-cloneconfig
    lxc-test-console
    lxc-test-ubuntu
    lxc-test-unpriv
    lxc-test-usernic
4. After applying the patch[3] based on pure kernel tree 3.9.0[2], both APM and D02 boards can pass all lxc tests on Ubuntu vivid.
5. There are still some failed cases in utopic on D02 board,they are:
    lxc-test-autostart
    lxc-test-cloneconfig
    lxc-test-ubuntu
    lxc-test-unpriv
6. Ubuntu made the same modifications to apparmor\overlayfs modules for Intel platform too.

For more detail information about lxc\lxc-tests, please refer to CARD-1852, you can get the detail work log from "Activity"-"Work Log" in this page:

Issues Description

With a large number of experiments and testing on different kernel tree and board, I got a variety of issues from them as follows.

Issues from pure kernel[2] on APM board

I've tried all lxc tests on pure kernel different versions from main line kernel tree[2], they are 3.19.0-rc4, 3.19.0, 4.0.0, but all of them can't pass all lxc test cases, the failed cases are listed as follows:

lxc-test-apparmor
lxc-test-apparmor-mount
lxc-test-autostart
lxc-test-cloneconfig
lxc-test-console
lxc-test-ubuntu
lxc-test-unpriv
lxc-test-usernic

All failed cases can't pass because of two main errors, take lxc-test-unpriv as example, you will get following two errors with do "sudo lxc-test-unpriv".

1. Start Container Error

The first error comes from starting container, the output information is showed as follows:

justin.zhao@r1-a13:~$ sudo lxc-test-unpriv 
/usr/sbin/deluser: The user `lxcunpriv' does not exist.
WARN: could not reopen tty: Permission denied
WARN: could not reopen tty: Permission denied
WARN: could not reopen tty: Permission denied
WARN: could not reopen tty: Permission denied
Using image from local cache
Unpacking the rootfs

---
You just created an Ubuntu container (release=trusty, arch=arm64, variant=default)

To enable sshd, run: apt-get install openssh-server

For security reason, container images ship without user accounts
and without a root password.

Use lxc-attach or chroot directly into the rootfs to set a root password
or create user accounts.
lxc-start: lxc_start.c: main: 344 The container failed to start.
lxc-start: lxc_start.c: main: 346 To get more details, run the container in foreground mode.
lxc-start: lxc_start.c: main: 348 Additional information can be obtained by setting the --logfile and --logpriority options.
Name:           c1
State:          STOPPED
lxc-attach: attach.c: lxc_attach: 632 failed to get the init pid
c2 is not running
c1 is not running
Removing user `lxcunpriv' ...
Warning: group `lxcunpriv' has no more members.
Done.
FAIL

The more detail error log is showed as follows:

lxc-start 1433406549.958 INFO     lxc_start_ui - lxc_start.c:main:264 - using rcfile /home/lxcunpriv/.local/share/lxc/c1/config
lxc-start 1433406549.958 WARN     lxc_confile - confile.c:config_pivotdir:1768 - lxc.pivotdir is ignored.  It will soon become an error.
lxc-start 1433406549.958 INFO     lxc_confile - confile.c:config_idmap:1376 - read uid map: type u nsid 0 hostid 910000 range 9999
lxc-start 1433406549.958 INFO     lxc_confile - confile.c:config_idmap:1376 - read uid map: type g nsid 0 hostid 910000 range 9999
lxc-start 1433406549.958 WARN     lxc_log - log.c:lxc_log_init:316 - lxc_log_init called with log already initialized
lxc-start 1433406549.960 WARN     lxc_cgmanager - cgmanager.c:cgm_get:963 - do_cgm_get exited with error
lxc-start 1433406549.960 INFO     lxc_start - start.c:lxc_check_inherited:221 - closed inherited fd 4
lxc-start 1433406549.963 INFO     lxc_container - lxccontainer.c:lxcapi_start:627 - Attempting to set proc title to [lxc monitor] /home/lxcunpriv/.local/share/lxc c1
lxc-start 1433406549.964 INFO     lxc_lsm - lsm/lsm.c:lsm_init:48 - LSM security driver AppArmor
lxc-start 1433406549.965 INFO     lxc_start - start.c:lxc_check_inherited:221 - closed inherited fd 4
lxc-start 1433406549.968 INFO     lxc_monitor - monitor.c:lxc_monitor_sock_name:177 - using monitor sock name lxc/df863b818eade8c1//home/lxcunpriv/.local/share/lxc
lxc-start 1433406550.128 INFO     lxc_start - start.c:lxc_init:451 - 'c1' is initialized
lxc-start 1433406550.129 INFO     lxc_start - start.c:resolve_clone_flags:848 - Cloning a new user namespace
lxc-start 1433406550.129 INFO     lxc_cgroup - cgroup.c:cgroup_init:65 - cgroup driver cgmanager initing for c1
lxc-start 1433406550.314 NOTICE   lxc_start - start.c:do_start:667 - switching to gid/uid 0 in new user namespace
lxc-start 1433406550.315 INFO     lxc_conf - conf.c:setup_utsname:908 - 'c1' hostname has been setup
lxc-start 1433406550.315 INFO     lxc_conf - conf.c:setup_network:2483 - network has been setup
lxc-start 1433406550.315 INFO     lxc_conf - conf.c:mount_autodev:1137 - Mounting /dev under /usr/lib/aarch64-linux-gnu/lxc
lxc-start 1433406550.315 INFO     lxc_conf - conf.c:mount_autodev:1158 - Mounted tmpfs onto /usr/lib/aarch64-linux-gnu/lxc/dev
lxc-start 1433406550.315 INFO     lxc_conf - conf.c:mount_autodev:1176 - Mounted /dev under /usr/lib/aarch64-linux-gnu/lxc
lxc-start 1433406550.316 INFO     lxc_conf - conf.c:mount_entry:1707 - failed to mount '/sys/fs/pstore' on '/usr/lib/aarch64-linux-gnu/lxc/sys/fs/pstore' (optional): No such file or directory
lxc-start 1433406550.316 INFO     lxc_conf - conf.c:mount_entry:1707 - failed to mount '/sys/firmware/efi/efivars' on '/usr/lib/aarch64-linux-gnu/lxc/sys/firmware/efi/efivars' (optional): No such file or directory
lxc-start 1433406550.316 INFO     lxc_conf - conf.c:mount_entry:1707 - failed to mount '/proc/sys/fs/binfmt_misc' on '/usr/lib/aarch64-linux-gnu/lxc/proc/sys/fs/binfmt_misc' (optional): No such file or directory
lxc-start 1433406550.316 INFO     lxc_conf - conf.c:mount_file_entries:2017 - mount points have been setup
lxc-start 1433406550.316 INFO     lxc_conf - conf.c:run_script_argv:345 - Executing script '/usr/share/lxcfs/lxc.mount.hook' for container 'c1', config section 'lxc'
lxc-start 1433406550.367 INFO     lxc_conf - conf.c:fill_autodev:1204 - Creating initial consoles under /usr/lib/aarch64-linux-gnu/lxc/dev
lxc-start 1433406550.367 INFO     lxc_conf - conf.c:fill_autodev:1215 - Populating /dev under /usr/lib/aarch64-linux-gnu/lxc
lxc-start 1433406550.367 INFO     lxc_conf - conf.c:fill_autodev:1247 - Populated /dev under /usr/lib/aarch64-linux-gnu/lxc
lxc-start 1433406550.367 INFO     lxc_conf - conf.c:setup_dev_console:1498 - console has been setup
lxc-start 1433406550.367 INFO     lxc_conf - conf.c:do_tmp_proc_mount:3558 - I am 1, /proc/self points to '1'
lxc-start 1433406550.418 INFO     lxc_conf - conf.c:lxc_create_tty:3357 - tty's configured
lxc-start 1433406550.418 INFO     lxc_conf - conf.c:setup_tty:1060 - 4 tty(s) has been setup
lxc-start 1433406550.418 NOTICE   lxc_conf - conf.c:lxc_setup:3937 - 'c1' is setup.
lxc-start 1433406550.418 WARN     lxc_apparmor - lsm/apparmor.c:apparmor_process_label_set:167 - Incomplete AppArmor support in your kernel
lxc-start 1433406550.418 ERROR    lxc_apparmor - lsm/apparmor.c:apparmor_process_label_set:169 - If you really want to start this container, set
lxc-start 1433406550.418 ERROR    lxc_apparmor - lsm/apparmor.c:apparmor_process_label_set:170 - lxc.aa_allow_incomplete = 1
lxc-start 1433406550.418 ERROR    lxc_apparmor - lsm/apparmor.c:apparmor_process_label_set:171 - in your container configuration file
lxc-start 1433406550.419 ERROR    lxc_sync - sync.c:__sync_wait:51 - invalid sequence number 1. expected 4
lxc-start 1433406550.419 ERROR    lxc_start - start.c:__lxc_start:1164 - failed to spawn 'c1'
lxc-start 1433406550.419 WARN     lxc_commands - commands.c:lxc_cmd_rsp_recv:172 - command get_init_pid failed to receive response
lxc-start 1433406550.420 WARN     lxc_cgmanager - cgmanager.c:cgm_get:963 - do_cgm_get exited with error
lxc-start 1433406550.420 ERROR    lxc_cgmanager - cgmanager.c:cgm_remove_cgroup:519 - call to cgmanager_remove_sync failed: invalid request
lxc-start 1433406550.420 ERROR    lxc_cgmanager - cgmanager.c:cgm_remove_cgroup:521 - Error removing all:lxc/c1-1
lxc-start 1433406555.425 ERROR    lxc_start_ui - lxc_start.c:main:344 - The container failed to start.
lxc-start 1433406555.425 ERROR    lxc_start_ui - lxc_start.c:main:346 - To get more details, run the container in foreground mode.
lxc-start 1433406555.425 ERROR    lxc_start_ui - lxc_start.c:main:348 - Additional information can be obtained by setting the --logfile and --logpriority options.

2. Mount Overlayfs Error

After I add "lxc.aa_allow_incomplete = 1" into lxc config file, then we get second error with following output information.

justin.zhao@r1-a13:~$ sudo lxc-test-unpriv 
/usr/sbin/deluser: The user `lxcunpriv' does not exist.
WARN: could not reopen tty: Permission denied
WARN: could not reopen tty: Permission denied
WARN: could not reopen tty: Permission denied
WARN: could not reopen tty: Permission denied
Using image from local cache
Unpacking the rootfs

---
You just created an Ubuntu container (release=trusty, arch=arm64, variant=default)

To enable sshd, run: apt-get install openssh-server

For security reason, container images ship without user accounts
and without a root password.

Use lxc-attach or chroot directly into the rootfs to set a root password
or create user accounts.
Name:           c1
State:          RUNNING
PID:            2877
CPU use:        0.13 seconds
Memory use:     2.43 MiB
KMem use:       0 bytes
Link:           vethWBKGAY
 TX bytes:      0 bytes
 RX bytes:      0 bytes
 Total bytes:   0 bytes
WARN: could not reopen tty: Permission denied
WARN: could not reopen tty: Permission denied
WARN: could not reopen tty: Permission denied
WARN: could not reopen tty: Permission denied
WARN: could not reopen tty: Permission denied
WARN: could not reopen tty: Permission denied
WARN: could not reopen tty: Permission denied
WARN: could not reopen tty: Permission denied
lxc_container: bdev.c: overlayfs_mount: 2253 Operation not permitted - overlayfs: error mounting /home/lxcunpriv/.local/share/lxc/c1/rootfs onto /usr/lib/aarch64-linux-gnu/lxc options upperdir=/home/lxcunpriv/.local/share/lxc/c2/delta0,lowerdir=/home/lxcunpriv/.local/share/lxc/c1/rootfs,workdir=/home/lxcunpriv/.local/share/lxc/c2/olwork
clone failed
c2 is not running
c1 is not running
Removing user `lxcunpriv' ...
Warning: group `lxcunpriv' has no more members.
Done.
FAIL

And the more detail log information is showed as follows:

lxc-start 1433407016.668 INFO     lxc_start_ui - lxc_start.c:main:264 - using rcfile /home/lxcunpriv/.local/share/lxc/c1/config
lxc-start 1433407016.668 WARN     lxc_confile - confile.c:config_pivotdir:1768 - lxc.pivotdir is ignored.  It will soon become an error.
lxc-start 1433407016.668 INFO     lxc_confile - confile.c:config_idmap:1376 - read uid map: type u nsid 0 hostid 910000 range 9999
lxc-start 1433407016.668 INFO     lxc_confile - confile.c:config_idmap:1376 - read uid map: type g nsid 0 hostid 910000 range 9999
lxc-start 1433407016.668 WARN     lxc_log - log.c:lxc_log_init:316 - lxc_log_init called with log already initialized
lxc-start 1433407016.669 WARN     lxc_cgmanager - cgmanager.c:cgm_get:963 - do_cgm_get exited with error
lxc-start 1433407016.670 INFO     lxc_start - start.c:lxc_check_inherited:221 - closed inherited fd 4
lxc-start 1433407016.673 INFO     lxc_container - lxccontainer.c:lxcapi_start:627 - Attempting to set proc title to [lxc monitor] /home/lxcunpriv/.local/share/lxc c1
lxc-start 1433407016.674 INFO     lxc_lsm - lsm/lsm.c:lsm_init:48 - LSM security driver AppArmor
lxc-start 1433407016.675 INFO     lxc_start - start.c:lxc_check_inherited:221 - closed inherited fd 4
lxc-start 1433407016.678 INFO     lxc_monitor - monitor.c:lxc_monitor_sock_name:177 - using monitor sock name lxc/df863b818eade8c1//home/lxcunpriv/.local/share/lxc
lxc-start 1433407016.808 INFO     lxc_start - start.c:lxc_init:451 - 'c1' is initialized
lxc-start 1433407016.809 INFO     lxc_start - start.c:resolve_clone_flags:848 - Cloning a new user namespace
lxc-start 1433407016.809 INFO     lxc_cgroup - cgroup.c:cgroup_init:65 - cgroup driver cgmanager initing for c1
lxc-start 1433407016.964 NOTICE   lxc_start - start.c:do_start:667 - switching to gid/uid 0 in new user namespace
lxc-start 1433407016.966 INFO     lxc_conf - conf.c:setup_utsname:908 - 'c1' hostname has been setup
lxc-start 1433407016.966 INFO     lxc_conf - conf.c:setup_network:2483 - network has been setup
lxc-start 1433407016.966 INFO     lxc_conf - conf.c:mount_autodev:1137 - Mounting /dev under /usr/lib/aarch64-linux-gnu/lxc
lxc-start 1433407016.966 INFO     lxc_conf - conf.c:mount_autodev:1158 - Mounted tmpfs onto /usr/lib/aarch64-linux-gnu/lxc/dev
lxc-start 1433407016.966 INFO     lxc_conf - conf.c:mount_autodev:1176 - Mounted /dev under /usr/lib/aarch64-linux-gnu/lxc
lxc-start 1433407016.966 INFO     lxc_conf - conf.c:mount_entry:1707 - failed to mount '/sys/fs/pstore' on '/usr/lib/aarch64-linux-gnu/lxc/sys/fs/pstore' (optional): No such file or directory
lxc-start 1433407016.966 INFO     lxc_conf - conf.c:mount_entry:1707 - failed to mount '/sys/firmware/efi/efivars' on '/usr/lib/aarch64-linux-gnu/lxc/sys/firmware/efi/efivars' (optional): No such file or directory
lxc-start 1433407016.967 INFO     lxc_conf - conf.c:mount_entry:1707 - failed to mount '/proc/sys/fs/binfmt_misc' on '/usr/lib/aarch64-linux-gnu/lxc/proc/sys/fs/binfmt_misc' (optional): No such file or directory
lxc-start 1433407016.967 INFO     lxc_conf - conf.c:mount_file_entries:2017 - mount points have been setup
lxc-start 1433407016.967 INFO     lxc_conf - conf.c:run_script_argv:345 - Executing script '/usr/share/lxcfs/lxc.mount.hook' for container 'c1', config section 'lxc'
lxc-start 1433407017.021 INFO     lxc_conf - conf.c:fill_autodev:1204 - Creating initial consoles under /usr/lib/aarch64-linux-gnu/lxc/dev
lxc-start 1433407017.021 INFO     lxc_conf - conf.c:fill_autodev:1215 - Populating /dev under /usr/lib/aarch64-linux-gnu/lxc
lxc-start 1433407017.021 INFO     lxc_conf - conf.c:fill_autodev:1247 - Populated /dev under /usr/lib/aarch64-linux-gnu/lxc
lxc-start 1433407017.021 INFO     lxc_conf - conf.c:setup_dev_console:1498 - console has been setup
lxc-start 1433407017.021 INFO     lxc_conf - conf.c:do_tmp_proc_mount:3558 - I am 1, /proc/self points to '1'
lxc-start 1433407017.078 INFO     lxc_conf - conf.c:lxc_create_tty:3357 - tty's configured
lxc-start 1433407017.078 INFO     lxc_conf - conf.c:setup_tty:1060 - 4 tty(s) has been setup
lxc-start 1433407017.078 NOTICE   lxc_conf - conf.c:lxc_setup:3937 - 'c1' is setup.
lxc-start 1433407017.078 WARN     lxc_apparmor - lsm/apparmor.c:apparmor_process_label_set:167 - Incomplete AppArmor support in your kernel
lxc-start 1433407017.079 INFO     lxc_apparmor - lsm/apparmor.c:apparmor_process_label_set:187 - changed apparmor profile to lxc-container-default
lxc-start 1433407017.079 NOTICE   lxc_start - start.c:start:1239 - exec'ing '/sbin/init'
lxc-start 1433407017.079 NOTICE   lxc_start - start.c:post_start:1250 - '/sbin/init' started with pid '2877'
lxc-start 1433407017.079 WARN     lxc_start - start.c:signal_handler:307 - invalid pid for SIGCHLD

From all above error's output and log information, we can conclude that both errors are related to kernel's apparmor and overlayfs modules. After I applied the patch[3] based on pure kernel 3.19.0, all lxc test cases can pass on this new Kernel image, and this patch[3] is just to update apparmor and overlayfs modules.

Issues from D02's kernel[4] on D02 board[1]

I also try to apply the same patch[3] into D02's kernel tree, and can build the kernel successfully on both utopic and vivid of Ubuntu. All lxc test cases can pass on Ubuntu vivid, but there are still some cases failed on Ubuntu utopic, take the lxc-test-unpriv as example, it will raises mounting overlayfs error as follows when I do "sudo lxc-test-unpriv".

  • Errors from Mounting Overlayfs

justin@linaro-developer:~$ sudo lxc-test-unpriv
/usr/sbin/deluser: The user `lxcunpriv' does not exist.
Path existed
WARN: could not reopen tty: Permission denied
WARN: could not reopen tty: Permission denied
WARN: could not reopen tty: Permission denied
WARN: could not reopen tty: Permission denied
Setting up the GPG keyring
Downloading the image index
Downloading the rootfs
Downloading the metadata
The image cache is now ready
Unpacking the rootfs

---
You just created an Ubuntu container (release=trusty, arch=arm64, variant=default)

To enable sshd, run: apt-get install openssh-server

For security reason, container images ship without user accounts
and without a root password.

Use lxc-attach or chroot directly into the rootfs to set a root password
or create user accounts.
Name:           c1
State:          RUNNING
PID:            2952
CPU use:        0.07 seconds
Memory use:     2.51 MiB
KMem use:       0 bytes
Link:           veth7AWHOE
 TX bytes:      0 bytes
 RX bytes:      0 bytes
 Total bytes:   0 bytes
WARN: could not reopen tty: Permission denied
WARN: could not reopen tty: Permission denied
WARN: could not reopen tty: Permission denied
WARN: could not reopen tty: Permission denied
WARN: could not reopen tty: Permission denied
WARN: could not reopen tty: Permission denied
WARN: could not reopen tty: Permission denied
lxc_container: bdev.c: overlayfs_mount: 2204 Invalid argument - overlayfs: error mounting /home/lxcunpriv/.local/share/lxc/c1/rootfs onto /usr/lib/aarch64-linux-gnu/lxc options upperdir=/home/lxcunpriv/.local/share/lxc/c2/delta0,lowerdir=/home/lxcunpriv/.local/share/lxc/c1/rootfs
clone failed
c2 is not running
c1 is not running
Removing user `lxcunpriv' ...
Warning: group `lxcunpriv' has no more members.
Done.
FAIL

There are still following lxc test cases can't pass with new kernel Image in Ubuntu utopic on D02 board[1], but it seems better than old kernel without the patch[3].

lxc-test-autostart
lxc-test-cloneconfig
lxc-test-ubuntu
lxc-test-unpriv

Reference

LEG/Engineering/LXC (last modified 2015-06-19 05:32:26)