Rolling your own minimal embedded Linux for the Raspberry Pi -- part three: services and remote access

Pi logo In this third article in my series on building a custom, minimal Linux for the Raspberry Pi, I describe how to move from a single-user, single-console system, to an installation with multiple consoles and the potential for remote access. Along the way we'll get WiFi working.

This is a long, and rather complicated article, even with a lot of the fine detail omitted for brevity. That's to be expected -- building a complete, custom Linux installation, even a simple one, requires a fair amount of work.

But first, let's recap.

Review

These articles are about building a minimal, fast-booting Linux for embedded applications on the Raspberry Pi (or similar). 'Embedded' is, of course, a vague term; I'm using it here to describe anything that could be considered an appliance, rather than a desktop computer. The main characteristics of an appliance are that should be, so far as possible, instant-on, and instant-off. The 'instant-on' (or, at least, fairly-quickly-on) requirement is met by initializing only minimal services at boot time. The 'instant-off' requirement is met by running with a read-only root filesytem -- one that will not be damaged simply by pulling the plug.

In the first part of this series I described how to create a bootable SD for the Raspberry Pi that would boot to a root shell prompt. The second part described how to set up volatile directories and install additional utilities, leading to a viable single-user Linux system. This additional set-up is necessary because the use of a read-only root filesystem creates constraints that will not be experienced with a desktop system. I pointed out then, and will do so again here, that the purpose of this exercise is to build an appliance that boots quickly and shuts down by pulling the plug. It isn't primarily about minimizing storage. To that end, I described how to create a script that would download Raspberry Pi binaries from Raspian along with their dependencies, and install them in an image of a root filesystem. I gave an example of such a script, and we will need it again in this part of the exercise. Obtaining binaries this way is not storage-efficient, because of the sprawling dependencies of repository-based software; but storage is cheap.

At this stage, you should have a system that boots in a few seconds to a single console with a root prompt. You won't be able to log in as a specific user, because no users exist, and you won't be able to switch virtual consoles. The virtual consoles exist, because they are managed by the kernel; but no process will be connected to them. You may have wired networking set up, because all this needs is a couple of utilities and the support that is built into the kernel; but WiFi is an order of magnitude more complex.

Before going much further, it's necessary to make an architectural decision about service management, because that decision will have far-reaching consequences.

What kind of service manager?

A Linux system will have a certain amount of one-time initialization -- loading kernel modules, mounting filesystems, setting a hostname, etc. We've discussed this already and, in fact, it's not very difficult -- a simple shell script will take care of most of this. However, in addition to one-time initialization, most Linux systems -- even in embedded applications -- will require services: processes that support system operation, and will run in the background so long as the system does.

What sorts of services will need to be started? That's impossible to answer in general terms, because it depends what the system is going to do. It's tempting to think that, in an embedded application, you don't need any services at all. However, you might need some or all of the following, if only for development and debugging:

In the desktop/server Linux world, there are two main strategies for service management: the old SysV-style init process, and the more modern systemd. Almost all mainstream Linux distributions now use systemd, for better or worse. It's a much more grown-up strategy than SysV init, because it manages boot-time dependencies; but it's overwhelmingly too complicated for any kind of appliance.

In the embedded Linux world, the choice is usually between one of the following:

It's certainly possible to code or script something that will start a few services and keep them running. However, I would argue that SysV init scripts are a better approach, even in an embedded system. init uses minimal memory, and isn't noticeably slower in operation than a hand-crafted solution. In the rest of this article I will assume that init will be the service manager, although the basic principles even if something else is used.

Installing and using SysV init in an embedded system

SysV init is in the Raspbian repository, under the name sysvinit-core. This package provides the init binary itself, a bunch of helper scripts and binaries, and a a selection of (mostly unhelpful in this situation) startup scripts.

Installing the init infrastructure

To use init you will need the following.

(1) The init binary itself, and a way for the kernel to run it. Running /sbin/init is, in fact, the Linux kernel's default boot mode but, if you've followed the previous articles, you may have changed that default by editing cmdline.txt. To use init, the init binary needs to be in the /sbin/ directory, and there needs to be no conflicting setting of init=... in cmdline.txt.

(2) The main configuration file for init, which is /etc/inittab. For an example, see below.

(3) A way to create the various scripts you need to initialize the system.

(4) If you want init to manage virtual consoles, you'll need a getty implementation. There are at least a half dozen available but, unless you need to handle real serial terminals, it's probably easiest to use agetty from the util-linux package. The getty binary is conventionally installed in /sbin. Installing util-linux will also get you /bin/login, which you'll need to authenticate users.

The iniitab file

This file, /etc/inittab, is read by init at boot time, and then used to coordinate all the other service management activities. There was a time when it would not have been necessary to explain how init worked to any experienced Linux user. These days, with the widespread adoption of systemd, a brief explanation might be in order.

init has a concept of run levels. Traditionally, a Linux system could, in principle, progress through the various run levels at it starts up, with run level 1 being single user mode, and run level 5 being full multi-user operation with graphical desktop. I would argue that this complexity is unnecessary in an embedded system -- there are really only two run levels: "on" and "off", which I denote by "0" and "1", since they have to be numeric.

Here is a sample inittab file, for an installation that has only two runlevels. 

si::sysinit:/etc/rc.d/startup.sh
id:1:initdefault:


1:1:respawn:/sbin/getty tty1
2:1:respawn:/sbin/getty tty2
3:1:respawn:/sbin/getty tty3

l0:0:wait:/etc/init.d/rc 0
l1:1:wait:/etc/init.d/rc 1

sd::shutdown:/etc/rc.d/shutdown
ca::ctrlaltdel:/sbin/reboot

The si:sysinit line indicates a script to run to perform one-time initialization. This script is the place to mount volatile file systems, set the hostname, etc -- things I described in the previous article in this series.

The initdefault entry indicates the run level to enter by default. Since there is only one run level other than "off", there isn't a lot of choice here.

The sd and ca lines define shutdown and reboot behaviour but, since this system will shut down by pulling the plug, these settings aren't particularly important.

The lines '1', '2', etc., indicate that getty should be executed in run level 1, and that it should continue to execute. The respawn setting is important here, because getty shuts down when a user has logged in. In order for a console to continue to be useful, getty has to be restarted when the virtual console has no process attached.

The r1 line indicates that init should run the rc script on the directory /etc/rc1.d at startup and shutdown (strictly speaking, when entering and leaving run level 1, but that amounts to the same thing in this configuration). The rc.1 directory will contain scripts with names 'SNNsomething' and `KNNsomething`, where the NN are numbers. The 'S' scripts are run in numerical order when entering run level 1 -- that is, at start-up; the 'K' scripts are run in reverse numerical order at shutdown.

In practice, the 'S' script and the 'K' script with a particular number can be links to the same script -- the rc process will run the script with the argument start at startup and stop at shutdown. It's conventional to make both the S and K scripts symbolic links to the 'real' scripts, which are in the directory /etc/init.d. This allows the start and stop code to be maintained in the same file, rather than two separate files. When a system had many run levels, and many different processes that had to be started and stopped when moving between run levels, some organization of this sort was essential. There is no actual harm in employing the same organization in an embedded system, and following the convention makes the start-up behaviour easier to follow than it might otherwise be.

A sample init script

Here is a very simple script for starting and stopping sshd; I would store this script as /etc/init.d/sshd. 

#!/bin/sh
case "$1" in
        stop)
          killall sshd
        ;;
        start)
          mkdir /run/sshd
          /usr/sbin/sshd -h /etc/ssh/id_rsa
        ;;
        *)
          echo 'Usage: /etc/init.d/sshd {start|stop}'
          exit 3
        ;;
esac
This script will be linked to an 'S' file and a 'K' file in rc1.d. From the running system, I see this: 
$ cd /etc/rc1.d/
kevin@pi:/etc/rc1.d$ ls -l *sshd*
lrwxrwxrwx 1 root root 14 Dec 11 11:41 K40sshd -> ../init.d/sshd
lrwxrwxrwx 1 root root 14 Dec 11 11:41 S40sshd -> ../init.d/sshd
kevin@pi:/etc/rc1.d$

At startup, init runs S40sshd with argument start, which runs /usr/bin/sshd with some necessary arguments. At shutdown, init runs killall sshd to stop sshd. This is pretty crude, and only really justifiable since the system will never actually go through an orderly shutdown.

Even without installing any init scripts, starting the system using init will make a difference -- rather than booting to a root shell, the system will now boot to a 'login:' prompt, provided by getty. Unfortunately, you won't be able to log in at the point, because you won't have any users to log in as -- see below for how to fix that.

Having explained how init runs scripts, we can look at how to use scripts to start various system services.

System log daemon

Most Linux systems will need a log daemon. The daemon traditionally listens for connections on a specific Unix socket, and writes data supplied by applications to a file /var/log/syslog. The log file may rotate when it gets too large.

This form of logging is not very appropriate in an embedded system, particularly with a read-only root filesystem. The directory /var/log can be made writable by, for example, linking it to a temporary filesystem in RAM. However, this isn't a very good use of limited RAM. What's required is a log daemon that maintains a fixed-size buffer in memory, and flushes the buffer out to a file (in a RAM filesystem) at periodic intervals. busybox package contains an implementation of a log daemon with a fixed size buffer, but it needs a specific utility to read the log, which I find a little inconvenient. So I wrote my own version, syslogd-lite, which is on GitHub. syslogd-lite looks like a traditional log daemon to applications that write logs, and the /var/log/syslog file looks like an ordinary file to administrators, except that it will never exceed a fixed size.

All the various lightweight log daemons can be started at boot time simply by running syslogd. There is no particular reason to provide an orderly way to stop them -- any data they are storing will disappear at shutdown anyway, since it's in RAM.

Users

At present, our Linux has no users. Although I've been talking about "booting to a root shell" in fact, unless the /etc/passwd file exists, we don't even have root as a user. If a system is to allow remote access, it makes sense to have at least one unprivileged user account to do it as. In fact, that makes sense even if the system doesn't allow remote access -- working as root as a matter of routine just feels icky.

If you set up init with getty as described above, then you should be able to see a log-in prompt in the various virtual consoles. However, to be able to log in, you need as a minimum a passed file. This should define at least two users -- root, and at least one unprivileged user. If you're going to allow remote access using ssh, then you'll need another user just for sshd, but more on that later.

A minimal passwd file for an embedded system might look like this: 

root:x:0:0:root:/root:/bin/false
user:[long encrypted password]:1000:100:User:/home/user:/bin/bash

In a Linux server installation it has become common practice to avoid putting passwords -- even encrypted passwords -- into passwd. These now go into a separate file, /etc/shadow, which is not world-readable. I don't feel that this complexity is necessary in an embedded system, but there's nothing to stop you setting things up that way if you prefer. Wherever you put the encrypted passwords, there's the problem of generating them.

A simple solution is just to create a user with the required username on some other, desktop Linux system, and just cut-and-paste the encrypted password between the systems. Nothing system-specific is encoded into the encrypted password.

Note that I've disallowed logging in as root completely. I will log in as user, and use sudo to change user if necessary. Arguably, this additional complexity is not necessary in an embedded system, but setting up sudo is simple enough. More on that later, though.

Along with a simple passwd file, you'll also need an /etc/group file to define groups for the users in passwd. It's trivially simple at this stage: 

root:x:0:
users:x:100:

Note that my passwd file defines the home directory of user as /home/user. This directory will need (ideally) to be created, and set to the ownership of the user user. If it's missing it won't prevent you logging in, but you'll have no place with permissions to store files, and you'll get a lot of irritating error messages.

At this stage -- with init and getty installed and set up, you should be able to log in as user at the login prompt, and you should be able to do this on multiple virtual consoles, switching between them using ctrl+alt+F1, ctrl+alt+F2, etc.

sudo

Since we have provided no way for root to log in, we need to enable either su or sudo so that the user can administer the system. I should point out that only limited administration is possible, because the root filesystem is read-only. I describe later how to switch the root filesystem between read-only and read-write modes.

sudo is in the Raspbian repository in the package sudo. This package includes not only the binary, but also a sample /etc/sudoers file. In the present case, you can probably delete the sample and replace it with this: 

root ALL=(ALL) ALL
user ALL=(ALL) ALL
With this change in place, you should be able to log in as user, and do 
$ sudo -i

to get a root shell. There's nothing to stop you adding additional user accounts at this stage, by editing /etc/password, /etc/group, and /etc/sudoers manually -- whether you need to do so depends, of course, on your application.

Switching between read-only and read-write filesystem

At present, the root filesystem is read-only and, if you're building a Linux appliance, I would expect it to stay that way through the whole production lifetime of the system. However, sometimes it is helpful to carry out experiments on the Pi itself, rather than building a new SD card image for each simple change.

Note:
If you switch to a read-write filesystem, it's all too easy to make changes to the live system and then not carry them back to your build environment. Then, when you build the SD card image again, the changes are overwritten. I don't know any solution to this problem except rigorous self-discipline. If you lack this, as I do, I would advise not switching to read-write mode often, if at all

In the previous article in this series I describe how to create a rudimentary /etc/fstab file. One of the entries in that file: 

/dev/mmcblk0p2 / auto defaults,noatime,ro 0 1
may not have had a clear purpose at that time. However, with this entry in place, you can now switch the root filesystem to read-write at the prompt: 
$ sudo mount -o rw,remount /
and to switch it back to read-only: 
$ sudo mount -o ro,remount /

This can be done when process are running, and have open files on the filesystem, if necessary.

Note:
Whenever the root filesystem is in read-write mode, it is vulnerable to damage if there is a disorderly shut-down or power-off. I would recommend switching back to read-only before switching off. Or just not using read-write mode at all, if practicable

WiFi networking

In the previous article I described how to obtain basic networking utilities and set a hostname. If you know how to use ipconfig you should be able to get a wired network connection up, at least with a fixed IP number. If you have an ethernet cable and a switch/router to plug it into, I would certainly recommend trying wired networking at this stage. Getting encrypted WiFi working (and it's always encrypted these days) is more complicated, and its worth checking that you know how to make wired networking operate first.

In this section I will explain the minimum needed to make WPA-encrypted WiFi work, with a static IP number and configuration. For dynamic configuration you would also need to install a DHCP client daemon, which isn't difficult, but that's a job for another day.

Prerequisites

You will need:

You should be able to get the various IP numbers from a Linux desktop system on the same network.

Starting wpa_supplicant

wpa_supplicant is the process that negotiates encryption in the background. You'll need it for WiFi operation on any network that uses encryption.

wpa_supplicant is started with the name of a network interface (usually wlan0, and a configuration file. The configuration file needs as a minimum to specify the location of the 'control interface' -- a directory where it can write a socket. This directory must exist, and be writeable; since we're working on a read-only filesystem, the script that starts wpa_supplicant will need to create this directory in some writeable area (that is, on a temporary filesytem). With these facts in mind, here is an init script to start and stop wpa_supplicant: 

#!/bin/sh
if [ -f /sbin/wpa_supplicant ]; then
    case "$1" in
        stop)
          killall wpa_supplicant
        ;;
        start)
          modprobe brcmutil
          modprobe brcmfmac
          mkdir -p /var/run/wpa_supplicant
          sleep 1
          wpa_supplicant -B -i wlan0 -c /etc/wpa_supplicant/wpa_supplicant.conf
        ;;
        *)
            echo 'Usage: /etc/init.d/wpa_supplicant {start|stop}'
            exit 3
            ;;
    esac
fi

The rather irritating one-second sleep is needed to give the kernel drivers time to initialize before starting wpa_supplicant. One second is probably far too long -- you might be able to tune this to a shorter value, or have the initialization of wpa_supplicant simply repeat until it succeeds.

The corresponding configuration file wpa_supplicant.conf is simply this: 

ctrl_interface=DIR=/var/run/wpa_supplicant

Setting up the WiFi adapter

Once wpa_supplicant is running, we can initialize the WiFi adapter, using static settings. Here is a sample init script: 

#!/bin/sh
  case "$1" in
      stop)
        ifconfig wlan0 down
      ;;
      start)
        wpa_cli -i wlan0 add_network
        wpa_cli -i wlan0 set_network 0 ssid [MY_SSID]
        wpa_cli -i wlan0 set_network 0 psk [MY_KEY]
        wpa_cli -i wlan0 enable_network 0
        ifconfig wlan0 $IP up
        route add default gw [MY_GATEWAY] 
      ;;
      *)
      echo 'Usage: /etc/init.d/wifi {start|stop}'
      exit 3
      ;;
esac
Note:
You'll find that network-related properties appear in many different configuration files as the system builds up, and it's better to store these in some sort of centralized configuration file, than hard-code them in a half-dozen different scripts. More on this point later.

If you want to get network access beyond the local network, you'll need to ensure you have a DNS server in /etc/resolv.conf 

nameserver [MY_DNS_SERVER] 
...

Running the scripts to start wpa_supplicant and the WiFi adapter should get you a working wireless network connection. If it doesn't, some troubleshooting will be in order. It's at this point where, if you haven't yet installed a system log daemon, you'll be regretting that decision, as all the diagnostic output from wpa_supplicant and wpa_cli goes to the system log.

Date and time

Not all applications will require correct date and time but, if yours does, you'll either have to buy/build a real-time-clock module, or get the system time from a network source.

The conventional way to synchronise time is to use an NTP daemon, although many Linux distributions have their own, usually simpler methods. An easy way to set the time on a one-off basis when the system starts is to make a request on a reliable webserver using curl (package curl in the Raspbian repository) and parse out the date from the response. Here is a script to do this. 

#!/bin/bash
URL=http://google.com
DELAY=5
tries=0
notset=true

while $notset && (( $tries < $MAX_TRIES ))
do
  logger --socket-errors=off -s $0: Getting time from $URL
  response=`curl -s --head $URL | grep ^Date: | sed 's/Date: //g'`
  if [ -z "$response" ]; then
    logger --socket-errors=off -s $0: No response -- sleeping
    sleep $DELAY
    tries=$(($tries+1))
  else
    notset=false
    date -s "$response" > /dev/null
  fi
done

if (( $tries == $MAX_TRIES )); then
  logger --socket-errors=off -s \
     $0: Could not get date from $URL after $MAX_TRIES attempts -- giving up
fi
Note that this script uses logger to write to the system log -- this utility is part of the util-linux package.

ssh client and server

Once networking is working, making outbound SSH connections from the Pi is straightforward enough -- just use the ssh utility from the openssh-client package. Installing an SSH server, to accept incoming connections, is a bit more fiddly -- and probably a lot more useful, in an embedded application.

The sshd daemon is in the openssh-client package in the Raspbian repository. In principle, running it amounts to nothing more than starting the process. Here is a simple init script to do that: 

#!/bin/sh
case "$1" in
   stop)
     killall sshd
     ;;
   start)
     mkdir /run/sshd
     /usr/sbin/sshd -h /etc/ssh/id_rsa
     ;;
   *)
     echo 'Usage: /etc/init.d/sshd {start|stop}'
     exit 3
     ;;
esac

There are a few things to note. As with many services, there needs to be a writable directory to store temporary files. In this case, it will be /run/sshd. The script will need to create this and, as ever, it will need to be in a temporary filesystem.

The openssh-server package provides various configuration files, some of which might need to be edited, to enable or disable particular features.

Although it isn't clear from the set-up, sshd needs to run as a particular user for security purposes, even though it starts as root. So we need to add to /etc/passwd: 

sshd:!:50:33:sshd:/:/bin/bash

This user need not be able to log in, and the account won't own any files, but it needs to exist.

The final, and most fiddly, problem is that sshd needs a server certificate. In the script above it was specified as /etc/ssh/id_rsa. Generating a simple, self-signed certificate is easy enough: 

$ ssh-keygen -b 1024 -t rsa -f /path/to/id_rsa

But this can't be done on the Pi -- not with a read-only filesystem, anyway. The certificate generation process is slow on a Pi -- it will take at least as long as the complete boot process so far. Because it's a read-only filesystem, we can't generate the certificate on the first boot, and then store it -- because there is nowhere to store it.

Instead, the certificate will need to be generated on the build host, when the root filesystem is being assembled, and then copied to the staging root filesystem for eventual transfer to an SD card.

Summary

The preceding sections have described various services that are likely to be needed when building up from a system that boots to single-user mode, to a full multi-user system with remote access.

On a Pi 3B+ -- which isn't the fastest single-board computer on earth -- the total cold boot time to a login prompt is about seven seconds. About five seconds of this is firmware and kernel initialization, which we have no control over, other than building a custom kernel. Once the system is started, I have the following processes running (plus init, which doesn't appear in this list): 

user@pi:~$ ps -ef|grep -v \\[
UID        PID  PPID  C STIME TTY          TIME CMD
root        91     1  0 12:27 ?        00:00:00 syslogd
root        94     1  0 12:27 tty1     00:00:00 /sbin/getty tty1
root        95     1  0 12:27 tty2     00:00:00 /sbin/getty tty2
root        96     1  0 12:27 tty3     00:00:00 /sbin/getty tty3
root       126     1  0 12:27 ?        00:00:00 wpa_supplicant -B -i wlan0 -c /wpa_supplicant.conf
root       148     1  0 12:27 ?        00:00:00 /usr/sbin/sshd -h /etc/ssh/id_rsa
user 173   162  0 12:28 ?              00:00:00 sshd: user@pts/0

It's interesting to consider what isn't running on what is, after all, now a workable, multi-user console workstation:

In other words, all the heavyweights of the standard Raspbian distribution (not including all the graphical desktop stuff, of course). Directly after boot, of the 1Gb or so of RAM on the Pi 3B+, 256Mb is used for the temporary filesystem in RAM, and 21Mb is used by the various processes. All the rest is free, or available for page caching.

This looks a lot more like the process and memory picture of an embedded Linux system than even the lightest-weight mainstream Raspberry Pi distribution does.

A note on configuration

A topic I have skimmed over in these articles so far is that of centralizing configuration. I'm assuming that the custom Linux installation will be used for lightweight applications, and might not have any user interface. Even if there is a simple user interface -- command-line console, for example -- the read-only nature of the installation makes configuration awkward. My intention is that all the necessary configuration is burned into the image on SD card at build time.

Since I will be using the same base image in a number of different installations, I need a way to customize the installation for each build. Parameters that will need to be changed include:

There is not normally a centralized configuration database for Linux, and these settings end up spread through various files on the system. Modifying the settings for multiple systems becomes a real problem.

My approach to configuration management -- which would only work on a simple, embedded Linux system -- is to have on the root filesytem one single configuration file that externalizes all the system-dependent settings into one file. All the init scripts, etc., load this file when they start, and use the settings in it. Nothing is hard-coded into any script.

Consequently, I can use the same SD card image for a number of different systems, and only have to edit a single file on the card to set it up. It takes a lot of extra work, and a good dose of self-discipline, to set things up this way, but it pays off in the long run.

Further work

In this article I've described how to set up init from scratch, and use it to manage services that I think many small Linux systems will need. However, that isn't the end of the job. Many systems will need some or all of the following.

And, of course, you'll need some software that actually does some useful work, rather than just keeping the system running.

Next: audio; or go to the series index.