DEV Community

Takashi SAKAGUCHI
Takashi SAKAGUCHI

Posted on

Tame Your Flaky RSpec Tests by Fixing the Seed

#ruby #rails

About This Article

Have you ever experienced tests that occasionally fail when running your RSpec test suite on CI? These are what we call flaky tests.

Simply re-running the test until it passes and calling it a day is a missed opportunity. Let’s take a more sustainable approach to fixing them.

Enable config.order and Kernel.srand

The key idea here is simple: enable the following configuration in your RSpec settings file, which is generated by default but commented out.

# spec/spec_helper.rb # Run specs in random order to surface order dependencies. If you find an # order dependency and want to debug it, you can fix the order by providing # the seed, which is printed after each run. # --seed 1234 config.order = :random # Seed global randomization in this process using the `--seed` CLI option. # Setting this allows you to use `--seed` to deterministically reproduce # test failures related to randomization by passing the same `--seed` value # as the one that triggered the failure. Kernel.srand config.seed =end
Enter fullscreen mode Exit fullscreen mode

Once enabled, this lets you reproduce random test failures deterministically using the seed printed after each test run.

These lines are typically wrapped in =begin and =end comments when generated, so make sure to move them out to enable them.

Reference:
https://github.com/rspec/rspec-core/blob/v3.13.2/lib/rspec/core/project_initializer/spec/spec_helper.rb#L86-L97

=begin These lines # are all comments =end
Enter fullscreen mode Exit fullscreen mode

A Hands-On Example of Dealing with Flaky Tests

Let’s walk through an example to see how flaky tests can be identified and fixed.

Setup from rails new 

# Create a new Rails app $ rails new --skip-test rspec-rails-test $ cd rspec-rails-test/  # Add and set up rspec-rails $ bundle add rspec-rails --group development,test $ rails generate rspec:install
Enter fullscreen mode Exit fullscreen mode

Now activate the config.order and Kernel.srand settings by moving them outside of the comment block.

 config.profile_examples = 10 +=end   config.order = :random Kernel.srand config.seed -=end  end
Enter fullscreen mode Exit fullscreen mode

Confirm RSpec runs successfully with zero examples:

$ bundle exec rspec No examples found. Randomized with seed 37597
Enter fullscreen mode Exit fullscreen mode

Create a Model and Write a Test

Let’s create a simple model named YearMonth with year and month attributes, and some basic validations.

# app/models/year_month.rb class YearMonth include ActiveModel::Model include ActiveModel::Attributes attribute :year, :integer attribute :month, :integer validates :year, numericality: { only_integer: true } validates :month, numericality: { only_integer: true, in: (1..12) } end
Enter fullscreen mode Exit fullscreen mode

Now, add a basic test to confirm that the model is valid.

# spec/models/year_month_spec.rb require "rails_helper" RSpec.describe YearMonth do subject { YearMonth.new(attributes) } let(:attributes) { { year:, month: } } let(:year) { rand(2100) } let(:month) { rand(12) } it do expect(subject.valid?).to be_truthy end end
Enter fullscreen mode Exit fullscreen mode

Run the Test 

$ bundle exec rspec Randomized with seed 54242 .  Finished in 0.0191 seconds 1 example, 0 failures
Enter fullscreen mode Exit fullscreen mode

If you were unlucky and saw a failure like this, just re-run the test and confirm it eventually passes:

Randomized with seed 30925 F Failures: 1) YearMonth is expected to be truthy Failure/Error: expect(subject.valid?).to be_truthy expected: truthy value got: false
Enter fullscreen mode Exit fullscreen mode

Observe Occasional Failures

Actually, this test sometimes fails.
After several runs, you should see an abnormal termination as shown below.

It is like a CI that is terminating normally and cheerfully in the product code, but sometimes notifies you of a failure.

$ bundle exec rspec  Randomized with seed 30925 F Failures: 1) YearMonth is expected to be truthy Failure/Error: expect(subject.valid?).to be_truthy expected: truthy value got: false  # ./spec/models/year_month_spec.rb:11:in 'block (2 levels) in <top (required)>'  Finished in 0.01313 seconds (files took 0.50544 seconds to load) 1 example, 1 failure Failed examples: rspec ./spec/models/year_month_spec.rb:10 # YearMonth is expected to be truthy  Randomized with seed 30925 $
Enter fullscreen mode Exit fullscreen mode

In real-world projects, it's useful to automate the test run in a loop and go grab lunch. Here’s a sample shell script:

#!/bin/bash set -euxo pipefail i=0 while true do i=$((i + 1)) bundle exec rspec done
Enter fullscreen mode Exit fullscreen mode

Reproduce the Failure

The key detail is this line in the output:

Randomized with seed 30925
Enter fullscreen mode Exit fullscreen mode

You can reproduce the exact test order and failure using this seed:

$ bundle exec rspec --seed 30925
Enter fullscreen mode Exit fullscreen mode

Debug the Failure

Now that it’s reproducible, debugging is straightforward. Add a puts statement to inspect the attributes:

 it do + puts "attributes: #{subject.attributes.inspect}"  expect(subject.valid?).to be_truthy end
Enter fullscreen mode Exit fullscreen mode 
$ bundle exec rspec --seed 30925 attributes: {"year" => 1913, "month" => 0} F
Enter fullscreen mode Exit fullscreen mode

The month is 0 — that’s not valid. The issue is with rand(12), which generates values from 0 to 11.

let(:month) { rand(12) }
Enter fullscreen mode Exit fullscreen mode 
> 10000.times.map { rand(12) }.uniq.sort => [0, 1, ..., 11]
Enter fullscreen mode Exit fullscreen mode

Fix the Root Cause

Fix the range to 1..12 instead of 0..11:

- let(:month) { rand(12) } + let(:month) { rand(1..12) }
Enter fullscreen mode Exit fullscreen mode 
> 10000.times.map { rand(1..12) }.uniq.sort => [1, 2, ..., 12]
Enter fullscreen mode Exit fullscreen mode

Now commit and push your fix:

git commit -m 'fix flaky test'
Enter fullscreen mode Exit fullscreen mode

Summary

  • Enabling config.order = :random and Kernel.srand config.seed in spec/spec_helper.rb helps identify and debug flaky tests.
  • Use the printed seed value and --seed option to reliably reproduce failures.
  • Don’t leave flaky tests lying around — fix them to ensure your CI surfaces real issues, not noise.

Top comments (0)