How to Manage Ruby Processes at Launch on Linux

How to Manage Ruby Processes at Launch on Linux

Feb 01, 2017 | By Matt |

Ruby, Linux, Raspberry Pi, IoT

For the past several months, we have been working on a software solution to run on embedded devices and recently found ourselves needing to solve the problem of how to run some of our software at determined times in the device lifecycle. In this post, we’ll explore the solutions that we explored, why we didn’t choose any of those solutions, and what we ended up with that has worked well.

manage ruby process

 

The Problem

Our goal is deceptively simple — we need to ensure that our software runs when a device is powered on. I say deceptively simple because we need to ensure that other supporting services are running. We wanted to wait until we had an active internet connection for instance, and most of our software depends upon a local redis instance being booted.  We decided to write our software in ruby, but documentation on IoT (Internet of Things) ruby development or running ruby on an embedded device is relatively scarce.  The good news is Linux has us covered!

The Linux community at large has many competing solutions to this problem, and reliable documentation has been hard to find. In our specific case, we have a collection of services that need to run on boot, comprised of scripts and long-running ruby processes. Additionally, our deployment device is a Raspberry Pi running Rasbian Linux, so our solution must work there.

Possible Solutions

We researched a few different solutions to manage ruby processes, outlined below. Ultimately we utilized Systemd.

System V (init.d) Script

Research how to run a bash script on boot in Linux, and the answer will most likely be creating a script in the /etc/init.d/ folder. The script itself is a bash script, which ends up being a lot of boilerplate code and very easy to get wrong. This is widely supported, although deprecated on most Linux distributions.

Pros

  • Already installed
  • Widely supported across Linux distros

Cons

  • Lots of boilerplate code
  • Difficult to read and write
  • No formal concept of load order to init scripts
  • Deprecated in favor of newer tools such as Upstart, Runit, or Systemd
  • Have to handle daemonization ourselves

Example:

#!/bin/sh
### BEGIN INIT INFO
# Provides:
# Required-Start:    $remote_fs $syslog
# Required-Stop:     $remote_fs $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Start daemon at boot time
# Description:       Enable service provided by daemon.
### END INIT INFO

dir=""
cmd=""
user=""

name=`basename $0`
pid_file="/var/run/$name.pid"
stdout_log="/var/log/$name.log"
stderr_log="/var/log/$name.err"

get_pid() {
    cat "$pid_file"
}

is_running() {
    [ -f "$pid_file" ] && ps `get_pid` > /dev/null 2>&1
}

case "$1" in
    start)
    if is_running; then
        echo "Already started"
    else
        echo "Starting $name"
        cd "$dir"
        if [ -z "$user" ]; then
            sudo $cmd >> "$stdout_log" 2>> "$stderr_log" &
        else
            sudo -u "$user" $cmd >> "$stdout_log" 2>> "$stderr_log" &
        fi
        echo $! > "$pid_file"
        if ! is_running; then
            echo "Unable to start, see $stdout_log and $stderr_log"
            exit 1
        fi
    fi
    ;;
    stop)
    if is_running; then
        echo -n "Stopping $name.."
        kill `get_pid`
        for i in {1..10}
        do
            if ! is_running; then
                break
            fi

            echo -n "."
            sleep 1
        done
        echo

        if is_running; then
            echo "Not stopped; may still be shutting down or shutdown may have failed"
            exit 1
        else
            echo "Stopped"
            if [ -f "$pid_file" ]; then
                rm "$pid_file"
            fi
        fi
    else
        echo "Not running"
    fi
    ;;
    restart)
    $0 stop
    if is_running; then
        echo "Unable to stop, will not attempt to start"
        exit 1
    fi
    $0 start
    ;;
    status)
    if is_running; then
        echo "Running"
    else
        echo "Stopped"
        exit 1
    fi
    ;;
    *)
    echo "Usage: $0 {start|stop|restart|status}"
    exit 1
    ;;
esac

exit 0

Above copied from this repo.

When starting a long-running process (daemon), it was up to us to write handle logging, PID file creation, signal handling and more. When writing a ruby daemon, we wrap our script using the daemons gem, but this still required extra effort and the creation of a “control” script. Additionally, System V doesn’t provide a way to monitor and ensure that a daemon is kept alive, meaning if our script fails it would not be restarted.

Monit

After passing on System V, Monit seems like a much better choice. It has a relatively easy configuration file syntax, can start our processes on boot, and can even ensure that the process is kept alive — if the process fails, monit can restart it automatically.

Monit itself is a daemon that can be installed via a package manager (in our case on Ubuntu and Rasbian) and is launched on boot. The monit process can be configured fairly easily to monitor and start scripts, although there are a few drawbacks that made monit our less than ideal choice.

Pros

  • Monit is launched on boot via System V
  • Easy configuration sytnax
  • Can restart a failed process automatically if desired
  • Can kill and restart a process based on parameters such as CPU usage and memory consumption
  • Does provide a formal way to declare that a monit service depends upon another monit service

Cons

  • Requires installation, such as apt-get on Raspian and Ubuntu
  • Still requires us to handle daemonization ourselves, with logging, PID files, and signal handling
  • Monit only provides a “sanitized” shell to launch processes, meaning a useless $PATH and more needs manipulating
  • Monit is a “polling” process, meaning a process that dies can be dead until Monit checks again

Ultimately we didn’t go with Monit because we needed to ensure that one of our processes was running always — and there will always be a delay with Monit polling for statuses of each process at an interval.

Runit

We did not do a lot of research into Runit, simply because Upstart and Systemd were what was available on Ubuntu. It really only popped on our radar because of Mike Perham of Sidekiq fame.

Cons

  • Not available on Ubuntu or Raspian

Upstart

Initially Upstart looked like the perfect solution, as it is the designated init system in Ubuntu 14.0 and has a clean syntax. However, the Linux community is attempting to converge on a standard, and with the release of Ubuntu 15.0 Systemd, and not Upstart, is the default init system. Combined with the fact that Raspian comes with Systemd instead of Upstart, and the choice was made for us.

Cons

  • Not available on Raspbian
  • Deprecated on Ubuntu 15.0 release

Finally — Systemd

The final solution we found involved using Systemd to manage our processes on boot and keep them alive reliably.

Pros

  • Installed and the default service manager on Ubuntu and Raspian
  • Simple configuration syntax
  • Can immediately restart a failed process
  • No need to daemonize our script!
  • Simple settings for user, working directory, and output
  • Integrated with syslog
  • Allows for “pre” and “post” run scripts before executing the process

Cons

  • Documentation didn’t make it clear that your service configuration file could exist anywhere on the filesystem.

Example

[Unit]
Description=udp-listener
After=multi-user.target

[Service]
Type=idle
WorkingDirectory=/var/apps/our-software
ExecStart=/var/apps/our-software/bin/udp-listener
User=pi
Group=pi

# if we crash, restart
RestartSec=1
Restart=always

# Logs go to syslog
# View with "sudo journalctl -u udp-listener"
StandardOutput=syslog
StandardError=syslog

SyslogIdentifier=udp-listener

[Install]
WantedBy=multi-user.target

In the end, Systemd provided the exact functionality we required, without requiring another dependency such as Monit to manage our services and hope this helps you with your next IoT ruby development project!

 

Free Prospector Trial

Written by Matt

Dad, husband, and Founder of Lotus Apps providing Rails and RubyMotion development out of Knoxville TN. Creator of http://www.gemprospector.com for #ruby devs.
Find me on:

Subscribe to Email Updates

Web vs Mobile software solutions
Pro
Why Should I Brand My Company?