How to Manage Ruby Processes at Launch on Linux

For the past several months, we have been working on a software solution to run on embedded devices and recently found ourselves needing to solve the problem of how to run some of our software at determined times in the device lifecycle. In this post, we’ll explore the solutions that we explored, why we didn’t choose any of those solutions, and what we ended up with that has worked well.

manage ruby process

The Problem

Our goal is deceptively simple — we need to ensure that our software runs when a device is powered on. I say deceptively simple because we need to ensure that other supporting services are running. We wanted to wait until we had an active internet connection for instance, and most of our software depends upon a local redis instance being booted. We decided to write our software in ruby, but documentation on IoT (Internet of Things) ruby development or running ruby on an embedded device is relatively scarce. The good news is Linux has us covered!

The Linux community at large has many competing solutions to this problem, and reliable documentation has been hard to find. In our specific case, we have a collection of services that need to run on boot, comprised of scripts and long-running ruby processes. Additionally, our deployment device is a Raspberry Pi running Rasbian Linux, so our solution must work there.

Possible Solutions

We researched a few different solutions to manage ruby processes, outlined below. Ultimately we utilized Systemd.

System V (init.d) Script

Research how to run a bash script on boot in Linux, and the answer will most likely be creating a script in the /etc/init.d/ folder. The script itself is a bash script, which ends up being a lot of boilerplate code and very easy to get wrong. This is widely supported, although deprecated on most Linux distributions.

Pros

Already installed
Widely supported across Linux distros

Cons

Lots of boilerplate code
Difficult to read and write
No formal concept of load order to init scripts
Deprecated in favor of newer tools such as Upstart, Runit, or Systemd
Have to handle daemonization ourselves

Example:

#!/bin/sh
### BEGIN INIT INFO
# Provides:
# Required-Start:    $remote_fs $syslog
# Required-Stop:     $remote_fs $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Start daemon at boot time
# Description:       Enable service provided by daemon.
### END INIT INFO

dir=""
cmd=""
user=""

name=`basename $0`
pid_file="/var/run/$name.pid"
stdout_log="/var/log/$name.log"
stderr_log="/var/log/$name.err"

get_pid() {
    cat "$pid_file"
}

is_running() {
    [ -f "$pid_file" ] && ps `get_pid` > /dev/null 2>&1
}

case "$1" in
    start)
    if is_running; then
        echo "Already started"
    else
        echo "Starting $name"
        cd "$dir"
        if [ -z "$user" ]; then
            sudo $cmd >> "$stdout_log" 2>> "$stderr_log" &
        else
            sudo -u "$user" $cmd >> "$stdout_log" 2>> "$stderr_log" &
        fi
        echo $! > "$pid_file"
        if ! is_running; then
            echo "Unable to start, see $stdout_log and $stderr_log"
            exit 1
        fi
    fi
    ;;
    stop)
    if is_running; then
        echo -n "Stopping $name.."
        kill `get_pid`
        for i in {1..10}
        do
            if ! is_running; then
                break
            fi

            echo -n "."
            sleep 1
        done
        echo

        if is_running; then
            echo "Not stopped; may still be shutting down or shutdown may have failed"
            exit 1
        else
            echo "Stopped"
            if [ -f "$pid_file" ]; then
                rm "$pid_file"
            fi
        fi
    else
        echo "Not running"
    fi
    ;;
    restart)
    $0 stop
    if is_running; then
        echo "Unable to stop, will not attempt to start"
        exit 1
    fi
    $0 start
    ;;
    status)
    if is_running; then
        echo "Running"
    else
        echo "Stopped"
        exit 1
    fi
    ;;
    *)
    echo "Usage: $0 {start|stop|restart|status}"
    exit 1
    ;;
esac

exit 0

Above copied from this repo.

When starting a long-running process (daemon), it was up to us to write handle logging, PID file creation, signal handling and more. When writing a ruby daemon, we wrap our script using the daemons gem, but this still required extra effort and the creation of a “control” script. Additionally, System V doesn’t provide a way to monitor and ensure that a daemon is kept alive, meaning if our script fails it would not be restarted.

Monit

After passing on System V, Monit seems like a much better choice. It has a relatively easy configuration file syntax, can start our processes on boot, and can even ensure that the process is kept alive — if the process fails, monit can restart it automatically.

Monit itself is a daemon that can be installed via a package manager (in our case on Ubuntu and Rasbian) and is launched on boot. The monit process can be configured fairly easily to monitor and start scripts, although there are a few drawbacks that made monit our less than ideal choice.

Pros

Monit is launched on boot via System V
Easy configuration sytnax
Can restart a failed process automatically if desired
Can kill and restart a process based on parameters such as CPU usage and memory consumption
Does provide a formal way to declare that a monit service depends upon another monit service

Cons

Requires installation, such as apt-get on Raspian and Ubuntu
Still requires us to handle daemonization ourselves, with logging, PID files, and signal handling
Monit only provides a “sanitized” shell to launch processes, meaning a useless $PATH and more needs manipulating
Monit is a “polling” process, meaning a process that dies can be dead until Monit checks again

Ultimately we didn’t go with Monit because we needed to ensure that one of our processes was running always — and there will always be a delay with Monit polling for statuses of each process at an interval.

Runit

We did not do a lot of research into Runit, simply because Upstart and Systemd were what was available on Ubuntu. It really only popped on our radar because of Mike Perham of Sidekiq fame.

Cons

Not available on Ubuntu or Raspian

Upstart

Initially Upstart looked like the perfect solution, as it is the designated init system in Ubuntu 14.0 and has a clean syntax. However, the Linux community is attempting to converge on a standard, and with the release of Ubuntu 15.0 Systemd, and not Upstart, is the default init system. Combined with the fact that Raspian comes with Systemd instead of Upstart, and the choice was made for us.

Cons

Not available on Raspbian
Deprecated on Ubuntu 15.0 release

Finally — Systemd

The final solution we found involved using Systemd to manage our processes on boot and keep them alive reliably.

Pros

Installed and the default service manager on Ubuntu and Raspian
Simple configuration syntax
Can immediately restart a failed process
No need to daemonize our script!
Simple settings for user, working directory, and output
Integrated with syslog
Allows for “pre” and “post” run scripts before executing the process

Cons

Documentation didn’t make it clear that your service configuration file could exist anywhere on the filesystem.

Example

[Unit]
Description=udp-listener
After=multi-user.target

[Service]
Type=idle
WorkingDirectory=/var/apps/our-software
ExecStart=/var/apps/our-software/bin/udp-listener
User=pi
Group=pi

# if we crash, restart
RestartSec=1
Restart=always

# Logs go to syslog
# View with "sudo journalctl -u udp-listener"
StandardOutput=syslog
StandardError=syslog

SyslogIdentifier=udp-listener

[Install]
WantedBy=multi-user.target

In the end, Systemd provided the exact functionality we required, without requiring another dependency such as Monit to manage our services and hope this helps you with your next IoT ruby development project!