2021-09-05
Note: this information is not meant as instructions, and I don't recommend you implement any of these changes without understanding what you're doing. I'm a lazy person and this method works reasonably well for my purposes, but these instructions won't be compatible with most people's needs. I'm writing it down for my own use and on the off-chance that someone else might find some part of it useful.
I use Void Linux on Raspberry Pi for a lot of small projects. In most cases I use Raspberry Pi Zero W for the convenient built-in 2.4GHz 802.11n wifi, and for a few projects that need image processing or other heavy processing tasks I use Raspberry Pi 4. My use cases are very specialized and simple and I have very few services running on these Raspberry Pis.
I do a couple of system modifications to hopefully make these devices more resilient against filesystem corruption.
MicroSD cards vary in quality. I take my MicroSD card recommendations from Jeff Geerling's blog and the only MicroSD failures that I've encountered in the last few years have all been my own fault.
I pop the MicroSD card into my Thinkpad and give it a typical Raspberry Pi partition table:
Device Boot Start End Sectors Size Id Type
/dev/mmcblk0p1 * 2048 206847 204800 100M b W95 FAT32
/dev/mmcblk0p2 206848 1983999 1777152 867.8M 83 LinuxThen I uncompress the premade musl-libc Raspberry Pi rootfs image to install Void.
Swich to bash for some conveniences:
chsh -s /bin/bashPut the hostname in the prompt:
-bash-5.1# vi ~/.profile
PS1="\H \w\\$ "
set -o vi
-bash-5.1# . ~/.profile
void-live ~# Write a hostname to /etc/hostname and set the system hostname with hostname(1). Make sure ntpd, wpa_supplicant, dhcpcd and sshd are enabled. Add a network to /etc/wpa_supplicant/wpa_supplicant.conf:
network = {
ssid="network ssid here"
psk="plaintext password here"
}Make sure the device gets an IP. Install socat.
Copy /etc/ssh/ssh_host_ed25519_key to key signing server and sign it with $rpi_host_key. Copy ssh_host_ed25519_key.cert and signing server's $user_key_pubkey rpi_user_key.pub back to /etc/ssh/. Add these lines to the bottom of /etc/ssh/sshd_config:
TrustedUserCAKeys /etc/ssh/rpi_user_ca.pub
HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pubConfirm that rpi_user_key can authenticate with the new host, and that the SSH client accepts its host key certificate.
Some of my Raspberry Pis automatically control devices and/or provide data sources. So I have a separate SSH key that's used by each service that needs to control/access each Raspberry Pi. If this new host needs to allow other hosts to automatically SSH into it to run certain commands, do the following:
Generate a new SSH keypair.
Install the pubkey onto the new host:'s ~/.ssh/authorized_keys:
airquality ~# vi /root/.ssh/authorized_keys
command="/root/bin/ssh_command $SSH_ORIGINAL_COMMAND" ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHgtXn3mDJ8RJFG0Rh35lVh6WBP7WXF9CGbIc95O8hfF unprivileged key for airquality (not the real public key)
airquality ~# mkdir ~/bin
airquality ~# vi ~/bin/ssh_command
#!/bin/sh
case "$1" in
on) ~/bin/on ;;
off) ~/bin/off ;;
get) ~/bin/get ;;
*) echo invalid ssh_command ;;
esac
airquality ~# chmod +x $_
airquality ~# vi ~/bin/on
#!/bin/sh
# Script to turn device 'on'
airquality ~# chmod +x $_Then copy the private key to the controlling host to be used for unprivileged SSH access to only the services listed in ~/bin/ssh_command.
The services that ship with Void all use symlinks to keep their supervise directories in /run/runit/:
airquality /etc/sv# file */supervise
agetty-tty1/supervise: symbolic link to /run/runit/supervise.agetty-tty1
agetty-ttyS0/supervise: broken symbolic link to /run/runit/supervise.agetty-ttyS0
aqi/supervise: directory
dhcpcd/supervise: symbolic link to /run/runit/supervise.dhcpcd
ntpd/supervise: symbolic link to /run/runit/supervise.chronyd
sshd/supervise: symbolic link to /run/runit/supervise.sshd
udevd/supervise: symbolic link to /run/runit/supervise.udevd
wpa_supplicant/supervise: symbolic link to /run/runit/supervise.wpa_supplicant
airquality /etc/sv# Note the broken symlink for agetty-ttyS0 which means it isn't running. Normally if you create a custom service runit will create a default ./supervise directory to track its status. But if our rootfs is readonly then that won't be possible and runit won't be able to start our service. So manually create symlinks for all custom services:
airquality /etc/sv/aqi# file *
run: POSIX shell script, ASCII text executable
supervise: directory
airquality /etc/sv/aqi# rm /var/service/aqi
airquality /etc/sv/aqi# ln -s /run/runit/supervise.aqi supervise
airquality /etc/sv/aqi# file *
run: POSIX shell script, ASCII text executable
supervise: broken symbolic link to /run/runit/supervise.aqi
airquality /etc/sv/aqi# ln -s /etc/sv/aqi /var/service
airquality /etc/sv/aqi# ln -s /tmp /var/log
ln -s /tmp /var/tmpIf this project needs any ARM device tree overlays, now's the time to add them.
For example, to enable the UART and disable the UART console:
airquality ~# mount /dev/mmcblk0p1 /boot
airquality ~# vi /boot/config.txt
enable_uart=1
airquality ~# vi /boot/cmdline.txt
root=/dev/mmcblk0p2 rw rootwait console=tty1 smsc95xx.turbo_mode=N dwc_otg.lpm_enable=0 loglevel=4 elevator=noop
airquality ~# umount /bootFor example, sending syslog to 10.0.0.2:514:
airquality ~# mkdir /etc/sv/logger
airquality ~# cd $_
airquality /etc/sv/logger# vi run
#!/bin/sh
exec socat unix-recv:/dev/log udp:10.0.0.2:514
airquality /etc/sv/logger# chmod +x run
airquality /etc/sv/logger# ln -s /run/runit/supervise.logger supervise
airquality /etc/sv/logger# ln -s /etc/sv/logger /var/service
airquality /etc/sv/logger# Additionally, to log kernel logs the same way:
airquality /etc/sv/logger# mkdir /etc/sv/klogger
airquality /etc/sv/logger# cd $_
airquality /etc/sv/klogger# vi run
#!/bin/sh
exec socat /proc/kmsg unix-send:/dev/log
airquality /etc/sv/klogger# chmod +x run
airquality /etc/sv/klogger# ln -s /run/runit/supervise.klogger supervise
airquality /etc/sv/klogger# ln -s /etc/sv/klogger /var/service
airquality /etc/sv/klogger# And on 10.0.0.2, the logging server:
logger /etc/sv/# mkdir /var/log/udp
logger /etc/sv/# mkdir -p socklog-udp/log
logger /etc/sv/# cd socklog-udp
logger /etc/sv/socklog-udp# vi run
#!/bin/sh
exec chpst -Unobody socklog inet 0 1337 2>&1
logger /etc/sv/socklog-udp# chmod +x run
logger /etc/sv/socklog-udp# cd log
logger /etc/sv/socklog-udp/log# vi run
#!/bin/sh
exec svlogd -t /var/log/udp
logger /etc/sv/socklog-udp/log# chmod +x run
logger /etc/sv/socklog-udp/log# ln -s /etc/sv/socklog-udp /var/service
logger /etc/sv/socklog-udp/log# This will enable a hardware watchdog to potentially recover the system in case it bogs way down. I only started doing this recently so I'm not sure if this will be useful or if it's just going to cause problems.
airquality ~# mkdir /etc/sv/watchdog
airquality ~# cd $_
airquality /etc/sv/watchdog# vi run
#!/bin/sh
modprobe watchdog
while true; do echo 1 > /dev/watchdog; sleep 14; done
airquality /etc/sv/watchdog# chmod +x run
airquality /etc/sv/watchdog# ln -s /run/runit/supervise.watchdog supervise
airquality /etc/sv/watchdog# ln -s /etc/sv/watchdog /var/service
airquality /etc/sv/watchdog# The watchdog will generate some noise in your kernel logs every 14 seconds, which gets noisy if you're doing network logging of kernel logs:
kern.crit: [1538830.858527] watchdog: watchdog0: watchdog did not stop!Edit /etc/runit/core-services/03-filesystems.sh and comment out these two lines near the bottom:
#msg "Mounting rootfs read-write..."
#mount -o remount,rw / || emergency_shellThen reboot and verify that all of your required services started properly.
You can still start and stop services as needed. If you need to modify something, you can temporarily remount the root fs read-write with:
airquality ~# mount -o remount,rw /Then make the changes and reboot. If it's not a good time to reboot, you can also just remount the root fs again as ro. On a stripped-down Void install there are so few services running that nothing will open any new files and obstruct the remount.