DEV Community

Cover image for Rails, Sidekiq : full tutorial
David Boureau
David Boureau

Posted on • Edited on • Originally published at alsohelp.com

Rails, Sidekiq : full tutorial

#ruby #rails #sidekiq

Article orginally published here : https://alsohelp.com/blog/rails-sidekiq-tutorial/

Sidekiq allows Rails to launch any task in the background. Let's see how, from zero to production.

Motivation

At BootrAils, we are trying to keep the stack as simple as possible.

However, the need for the existence of background jobs arises very quickly when you create a new Rails application for business purposes. Think about the "forgot password ?" feature. An email needs to be sent. deliver_later is very recommended, and this method needs "background jobs" to work properly.

And you can't do that easily, neither in Ruby nor Ruby-on-Rails.

Rails Q: handling a request, is there a good utility or idiom to run a block of code asynchronously so it doesn't block the response but also doesn't incur the overhead of a job/queue system?

do_later { fire_and_forget() }

Just have a one-off call & don't want the user to wait.

— Justin Searls (@searls) January 18, 2022

In theory, ActiveJob (in charge of background jobs, as the name implies) is already included in every default Rails app.

But... the set up, dependencies, as well as production, test and development settings are entirely left to the developer.

There are two giants in this space : delayed_jobs (sometimes known as "DJ"), and Sidekiq. Sidekiq is more well-known, maintained, and documented, I would advise here to choose Sidekiq for any new Rails application - YMMV.

This tutorial is here to remove fear about installing background jobs into an application whose primary purpose is meant to be "just MCV".

Prerequisites

Here are the tools we will use in this tutorial :

$> ruby -v ruby 3.1.0p0 // you need at least version 3 here $> bundle -v Bundler version 2.2.11 $> npm -v 8.3.0 // you need at least version 7.1 here $> yarn -v 1.22.10 $> psql --version psql (PostgreSQL) 13.1 // let's use a production-ready database locally $> redis-cli ping // redis is a dependency of Sidekiq PONG $> foreman -v 0.87.2
Enter fullscreen mode Exit fullscreen mode

Create a fresh new Rails app - without Sidekiq first 

mkdir sidekiqrails && cd sidekiqrails echo "source 'https://rubygems.org'" > Gemfile echo "gem 'rails', '7.0.1'" >> Gemfile bundle install bundle exec rails new . --force -d=postgresql --minimal # Create a default controller echo "class WelcomeController < ApplicationController" > app/controllers/welcome_controller.rb echo "end" >> app/controllers/welcome_controller.rb # Create a default route echo "Rails.application.routes.draw do" > config/routes.rb echo ' get "welcome/index"' >> config/routes.rb echo ' root to: "welcome#index"' >> config/routes.rb echo 'end' >> config/routes.rb # Create a default view mkdir app/views/welcome echo '<h1>This is h1 title</h1>' > app/views/welcome/index.html.erb # Create database and schema.rb bin/rails db:create bin/rails db:migrate
Enter fullscreen mode Exit fullscreen mode

Then open application.rb and uncomment line 6 as follow :

# inside config/application.rb require_relative "boot" require "rails" # Pick the frameworks you want: require "active_model/railtie" require "active_job/railtie" # <== Uncomment # ... everything else remains the same
Enter fullscreen mode Exit fullscreen mode

Then create the parent Class of all jobs :

# inside app/jobs/application_job.rb class ApplicationJob < ActiveJob::Base end
Enter fullscreen mode Exit fullscreen mode

Side note We created the Rails app with the --minimal flag, so that you can discover clearly what is needed to run a job. With the default install, active_job/railtie is already uncommented, and the parent Class of all jobs already exists.

Add redis and sidekiq gems to your Rails app

Open your Gemfile, and add at the very bottom

gem 'redis' gem 'sidekiq'
Enter fullscreen mode Exit fullscreen mode

And in your terminal, run

bundle install
Enter fullscreen mode Exit fullscreen mode

Then add the following line inside application.rb

# inside config/application.rb # ... class Application < Rails::Application config.active_job.queue_adapter = :sidekiq # ...
Enter fullscreen mode Exit fullscreen mode

Create a hello world job

Create a new file under app/jobs/hello_world_job.rb

# inside app/jobs/hello_world_job.rb class HelloWorldJob < ApplicationJob queue_as :default def perform(*args) # Simulates a long, time-consuming task sleep 5 # Will display current time, milliseconds included p "hello from HelloWorldJob #{Time.now().strftime('%F - %H:%M:%S.%L')}" end end
Enter fullscreen mode Exit fullscreen mode

Nothing fancy. The purpose of this job is to be run asynchronously when called. Whereas the classic HTTP requests/responses will be handled inside the only available Thread. However any to HelloWorldJob can be initially triggered inside an HTTP request. Let's see how.

Call the hello world job from the view

Modify routes.rb as follow :

# inside config/routes.rb Rails.application.routes.draw do get "welcome/index" # route where any visitor require the helloWorldJob to be triggered post "welcome/trigger_job" # where visitor are redirected once job has been called get "other/job_done" root to: "welcome#index" end
Enter fullscreen mode Exit fullscreen mode

Create app/controllers/other_controller.rb

# inside app/controllers/other_controller.rb class OtherController < ApplicationController def job_done end end
Enter fullscreen mode Exit fullscreen mode

Create app/views/other/job_done.html.erb

<!-- inside app/views/other/job_done.html.erb --> <h1>Job was called</h1>
Enter fullscreen mode Exit fullscreen mode

Now our job is ready to be called from the initial view :

<%# inside app/views/welcome/index.html.erb %> <h1>This is h1 title</h1> <%= form_with url: welcome_trigger_job_path do |f| %> <%= f.submit 'Launch job' %> <% end %>
Enter fullscreen mode Exit fullscreen mode

And the call will happen inside the controller, like this :

# inside app/controllers/welcome_controller.rb class WelcomeController < ApplicationController def trigger_job HelloWorldJob.perform_later redirect_to other_job_done_path end end
Enter fullscreen mode Exit fullscreen mode

Add a Procfile.dev

You probably want to try everything right now, but wait a minute : our jobs must be sent from another thread. This is done locally by adding a Procfile.dev file at the root of our project :

web: bin/rails s worker: bundle exec sidekiq -C config/sidekiq.yml
Enter fullscreen mode Exit fullscreen mode

Add config/sidekiq.yml

# inside config/sidekiq.yml development: :concurrency: 5 production: :concurrency: 10 :max_retries: 1 :queues: - default
Enter fullscreen mode Exit fullscreen mode

And finally add the sidekiq initializer here : config/initializers/sidekiq.rb

# inside config/initializers/sidekiq.rb Sidekiq.configure_server do |config| config.redis = { url: ENV.fetch('REDIS_URL', 'redis://localhost:6379/1') } end Sidekiq.configure_client do |config| config.redis = { url: ENV.fetch('REDIS_URL', 'redis://localhost:6379/1') } end
Enter fullscreen mode Exit fullscreen mode

redis://localhost:6379/1 is the default URL of a locally installed redis database. If it's not available here, REDIS_URL is the environment variable that will come to the rescue.

Launch local app

Now it's time to see if everything is going well.

Launch your local server like this :

foreman start -f Procfile.dev
Enter fullscreen mode Exit fullscreen mode

Let's see what is displayed locally :

localhost

localhost

Click on the "Launch job" button, and immediately display logs in your terminal :

18:43:31 web.1 | Started POST "/welcome/trigger_job" for ::1 at 2022-01-22 18:43:31 +0100 18:43:31 web.1 | Processing by WelcomeController#trigger_job as HTML 18:43:31 web.1 | [ActiveJob] Enqueued HelloWorldJob (Job ID: 9b9aa325-d118-45ff-a74b-99cad73ecda8) to Sidekiq(default) 18:43:31 web.1 | Redirected to http://localhost:5000/other/job_done 18:43:31 web.1 | 18:43:31 web.1 | Started GET "/other/job_done" for ::1 at 2022-01-22 18:43:31 +0100 18:43:31 web.1 | Completed 200 OK in 4ms (Views: 3.3ms | ActiveRecord: 0.0ms | Allocations: 2644) 18:43:31 web.1 | 18:43:31 worker.1 | 2022-01-22T17:43:31.776Z pid=3002 tid=3ze class=HelloWorldJob jid=166d945e2b7cd15e37addc7c INFO: Performing HelloWorldJob (Job ID: 9b9aa325-d118-45ff-a74b-99cad73ecda8) from Sidekiq(default) enqueued at 2022-01-22T17:43:31Z 18:43:36 worker.1 | "hello from HelloWorldJob 2022-01-22 - 18:43:36.784" 18:43:36 worker.1 | 2022-01-22T17:43:36.785Z pid=3002 tid=3ze class=HelloWorldJob jid=166d945e2b7cd15e37addc7c INFO: Performed HelloWorldJob (Job ID: 9b9aa325-d118-45ff-a74b-99cad73ecda8) from Sidekiq(default) in 5008.15ms 18:43:36 worker.1 | 2022-01-22T17:43:36.786Z pid=3002 tid=3ze class=HelloWorldJob jid=166d945e2b7cd15e37addc7c elapsed=5.47 INFO: done
Enter fullscreen mode Exit fullscreen mode

We copy/pasted here a simplified version of the logs, so that you can see what is actually happening. The GET "/other/job_done" is triggered immediately, at 18:43:31; whereas the job is actually called 5 seconds later, at 18:43:36. This is what is called "background jobs".

Push Rails and Sidekiq to production

Follow step below :

# heroku is connected heroku login heroku create # This will modify local files echo "web: bundle exec puma -C config/puma.rb" > Procfile echo "worker: bundle exec sidekiq -e production -C config/sidekiq.yml" >> Procfile bundle lock --add-platform x86_64-linux # This will modify you heroku app heroku addons:create heroku-postgresql:hobby-dev heroku addons:create heroku-redis:hobby-dev heroku buildpacks:add heroku/ruby git add . && git commit -m 'ready for prod' git push heroku main # app works, but worker (for background jobs) is missing heroku ps:scale worker=1
Enter fullscreen mode Exit fullscreen mode

So far everything should work properly.

Wait 1 minute (maximum 2), so that every Heroku dependency is properly set up.

Open the application in your browser (the URL should contain heroku.com, so we're not trying localhost this time).

Click on the "launch" button again, and check your logs by typing

heroku logs
Enter fullscreen mode Exit fullscreen mode

Can you see the worker that correctly prints "Hello world" ? Sidekiq works !

A last word (of caution)

Background jobs tend to swallow exceptions without any complaint. Failed to connect to the Redis database ? It won't tell you. Bug appeared ? May not be shown. So our advice would be to make sure that everything is working properly by keeping the HelloWorldJob available (to quickly manually check if Sidekiq is working), and to install the Sidekiq web UI right from the start (not covered in this tutorial).

Top comments (0)