In a recent project, we found ourselves in a situation where it made sense to introduce a background processor to the project while working on an embedded device, but we weren’t using Rails.  To use sidekiq without Rails, we needed to solve simple issues such as startup and process supervision, so that we could reliably ensure Sidekiq would be running in our production environment.

We needed a processor for the following reasons:

• We interacted with services via the internet
• We expected failures and wanted the ability to retry easily
• We were processing large amount of incoming data from connected hardware and wanted to offload that to the background

Installation without Rails

Installing Sidekiq in a plain ruby project using Bundler wasn’t anything extraordinary — you simply add the gem to your Gemfile, bundle and define your workers.

Running the actual Sidekiq process required passing several options to specify the root of our project, how to load our code, the path to our configuration file and more.

bundle exec sidekiq -r ./config/boot.rb -C ./config/sidekiq.yml

In our codebase, we had created a config/boot.rb to load bundler and require all of our classes to bootstrap our project. This was inspired by many other frameworks such as Sinatra or Rails. The config/sidekiq.yml file is standard, and since we had a config directory in our codebase, we shoved it in there to match Rails conventions.

Running Sidekiq via SystemD

Running locally in development is fine with the above command, but once deployed we needed a supervisor to ensure that Sidekiq was always running, and a way to start the Sidekiq process on boot. The answer to both questions is the wonderful SystemD.

You can read more about configuring ruby scripts, such as Sidekiq, in another post titled: "How to Managing Ruby Processes at Launch".

Our SystemD service unit definition looks like the following:

You’ll notice that we are set to reset within a second if Sidekiq crashes for any reason, and that we want to wait until after the redis-server (also managed via SystemD) is available before starting.

Daemons

Don’t daemonize! There is a “fork” type in SystemD that would have required us to specify a PID file location and allow starting the “sidekiq” executable with the “-D” flag to run as a daemon, but there’s no need for it.

SystemD will start and monitor the Sidekiq process itself, and supervise to make sure it’s running. Instead of worrying about log files, we’re sending all output to syslog, a centralized logging service.

Managing Sidekiq

With the service fine enabled via SystemD, restarting or stopping Sidekiq has become very easy:

sudo systemctl restart sidekiq
sudo systemctl status sidekiq

Sidekiq will launch during boot after redis-server is available, and automatically restart if it crashes. Awesome!

Logging

We hinted at logging with syslog already, but to view the logs you can use the journalctl program.

To view all entries from Sidekiq (even across reboots).

sudo journalctl -u sidekiq

Or to just view messages from the most recent boot:

sudo journalctl -u sidekiq -b

You can even pass options to filter to a specific time window, or do a live tail and grep to search for an occurrence of an error(great for debugging).

sudo journalctl -u sidekiq -f | grep SomeErrorMessageToExpect