Notes on booting ARM boards of the network

Requirements

Usually, systems booting of the network will follow these steps:

  • set the MAC address from EEPROM or from flash
  • do a BOOTP/DHCP request to get an IP address, optionally a TFTP server to boot from, and optionally some DNS servers to resolve hostnames
  • TFTP the required files, in the case of PXE boot this is a PXE file describing which files to load and how to use them
  • boot

There are variants to the above boot steps, such as using HTTP instead of TFTP (RedBoot supports directly loading http:// objects into memory), or having a hardcoded network configuration (IP address, netmask, perhaps gateway and DNS servers...). Some ARM boards like PandaBoard lack an EEPROM for the MAC address, and need one to be set by the bootloader if they want to use BOOTP/DHCP to get an IP address. Also, more things might be happening on the server side, such as provisioning, or registering the DHCP hostname into DNS.

Assuming the most classical use case, you will need:

  • a BOOTP/DHCP server
  • a TFTP server
  • some boot files (PXE configuration, kernel, initrd etc.)

Hardware

In an enterprise setup, network boot would likely be implemented on a dedicated server, likely with some redundancy if this is a critical piece of infrastructure (and not just used as a maintenance or recovery tool for instance).

For developers playing with network boot, it's advisable to set things up on a dedicated network, most importantly to not conflict with the regular DHCP server on your network. A good way to achieve this is to setup the tools to only listen on one interface of your development system, perhaps an USB-ethernet adapter if you don't have enough network interfaces.

Software

To implement a network boot solution, there are either all-in-one software solutions like Dnsmasq or Cobbler or you can build your own, in which case you get to pick your TFTP server, BOOTP/DHCP server, DNS server etc.

TFTP

In Debian/Ubuntu, multiple TFTP servers are available but in practice there might be limitations of this or that implementation or incompatibilities. Here are some sample packages providing TFTP server functionality that you can find in Debian/Ubuntu at the time of writing:

  • atftpd
  • tftpd
  • tftpd-hpa

But there's also a wealth of lesser known tools which might provide similar functionality and of course there are some bindings to write your own TFTP server. Also, some higher level programs ship with a builtin TFTP server.

In the author's experience, tftpd-hpa works well.

BOOTP/DHCP

Again, modern Linux distributions offer the choice between multiple DHCP servers such as ISC's DHCP server (in the isc-dhcp-server or dhcp3-server package under Debian/Ubuntu) or busybox' DHCP server implementation (in the udhcpd package under Debian/Ubuntu).

ISC's DHCP server stands out as the most commonly deployed DHCP server and shoudl be preferred if you're picking a standalone DHCP server.

BOOTP is essentially a subset of DHCP, and is less common nowadays. ISC's DHCP server can serve both BOOTP and DHCP clients, but DHCP is simpler, more common and more extensible. For instance, BOOTP clients usually don't pass a hostname, and don't support limited-time leases.

Most software supports only DHCP, but as a counter-example the RedBoot bootloader only supports BOOTP and not DHCP.

PXE

There are standalone servers for PXE, but essentially PXE is a standard covering how to do some BOOTP/DHCP requests and how to TFTP some files, so a typical network boot setup will not have a specialized PXE server but rather organize for files to be in the expected location in the TFTP hierarchy and for the DHCP server to return the right options.

PXELINUX does warrant a special mention: typically on x86, some network NICs will implement PXE in their ROM but with restrictions on what can be loaded and run; PXELINUX will act as a first stage payload which will load a PXELINUX-specific config over the network which allows prompting the user for which image to load, and more advanced setups like loading a kernel and and an initrd, or passing arguments on the kernel command-line. This is not ported to ARM. Etherboot/gPXE is another slightly more recent opensource implementation comparable to PXELINUX; again, it doesn't currently support ARM.

DNS

A recursive DNS server is not strictly required to achieve network booting, but it's usually needed on the deployed systems anyway and is usually setup in the DHCP server as to automatically pass the information to clients.

It is easier to deal with hostnames than with IP addresses, albeit that might also introduce new dependencies on your setup.

Rolling a recursive DNS server is a matter of installing and configuring popular software like BIND or Unbound, but it's usually fine to forward DNS requests to the recursive servers provided by your organization or your ISP. The main use case for deploying one's own DNS server is to register hostnames of DHCP clients in the DNS server along their attributed IP address.

Dnsmasq

Dnsmasq is a flexible all-in-one solution providing its own builtin DNS, DHCP and TFTP servers.

Cobbler

Cobbler is a higher level management software abstracting away the previously covered bits. Cobbler offers Python bindings, command-line and XML-RPC interfaces, a web UI, client side tools and allows managing multiple network boot profiles for one's own needs.

Recipes

The following recipes assume you're running a fairly recent Ubuntu setup. The instructions should be mostly applicable to other Linux distributions though.

The instructions also assume that you will use the network boot server's eth1 interface with IP address 172.16.0.254 and netmask 172.16.0.0/12.

Network setup

Setup eth1 with this /etc/network/interfaces snippet:

    auto eth1
    iface eth1 inet static
        address 172.16.0.254
        netmask 255.240.0.0

And issue sudo ifup eth1 to bring up the interface.

Deploying ISC's DHCP server

Install the isc-dhcp-server package.

In the default configuration, it should fail to start with following details in syslog:

    dhcpd: Internet Systems Consortium DHCP Server 4.1.1-P1
    dhcpd: Copyright 2004-2010 Internet Systems Consortium.
    dhcpd: All rights reserved.
    dhcpd: For info, please visit https://www.isc.org/software/dhcp/
    dhcpd: Internet Systems Consortium DHCP Server 4.1.1-P1
    dhcpd: Copyright 2004-2010 Internet Systems Consortium.
    dhcpd: All rights reserved.
    dhcpd: For info, please visit https://www.isc.org/software/dhcp/
    dhcpd: Wrote 0 leases to leases file.
    dhcpd: 
    dhcpd: No subnet declaration for eth0 (192.168.X.Y). 
    dhcpd: ** Ignoring requests on eth0.  If this is not what
    dhcpd:    you want, please write a subnet declaration
    dhcpd:    in your dhcpd.conf file for the network segment
    dhcpd:    to which interface eth0 is attached. **
    dhcpd: 
    dhcpd: 
    dhcpd: Not configured to listen on any interfaces! 

To only answer DHCP requests of eth1, set INTERFACES="eth1" in /etc/default/isc-dhcp-server.

Edit /etc/dhcp/dhcpd.conf and define some DHCP subnet; for instance:

    subnet 172.16.0.0 netmask 255.240.0.0 {
        range dynamic-bootp 172.17.0.0 172.17.255.255;
        option domain-name-servers google-public-dns-a.google.com, google-public-dns-b.google.com;
        option routers 172.16.0.254;
        option broadcast-address 172.31.255.255;
        default-lease-time 600;
        max-lease-time 7200;
        filename "xyz";
    }

If you only want to serve DHCP and not BOOTP clients, you can use the deny bootp directive.

Deploying HPA's TFTP server

Install the tftpd-hpa package. By default it listen on all interfaces and shares files out of /var/lib/tftpboot. You can change these defaults in /etc/default/tftpd-hpa.

U-Boot BOOTP/DHCP, TFTP and PXE implementations

Some U-Boot boards support network boot; this requires:

  • having support for your Ethernet adapter in U-Boot; if this adapter is wired over USB, your USB controller needs to be supported in U-Boot too
  • CONFIG_CMD_NET to be turned on for the bootp command; this is turned on in the default U-Boot configuration, so most boards should have it

  • CONFIG_CMD_DHCP to be turned on for the dhcp command

Note that turning on DHCP support changes the behavior of U-Boot to default to DHCP even when using the bootp command.

OfficeofCTO/NetworkBoot (last modified 2011-09-29 15:55:31)