Hey Howdy Hey!

I'm super appreciative of the refuse_coverage_drop option, but have two issues:

1. I cannot seem to get the last_run file generated.
2. Even if I could, is there a way to change the coverage path only when reading the last_run_path so I don't have to give our CI a write access token to commit these?

Missing last_run file

I suspect this is coming from the fact that we are collating. I generally used this tutorial from Knapsack Pro to help knapsack, simplecov, and semaphore get along nicely through these steps...

1. Knapsack parallelizes a full rspec run (bundle exec rake knapsack_pro:queue:rspec['--no-color --format progress --format RspecJunitFormatter --out tmp/rspec-junit/rspec.xml']) across 6 jobs. After each job completes (not all), this is run to push up the results into a semaphore artifact (will be used in the next step):

#!/bin/bash set -eu if [ -d "tmp/rspec-junit" ] then echo "Pushing rspec junit results" artifact push job tmp/rspec-junit --destination semaphore/test-results/ fi if [ -d "coverage" ] then echo "Pushing simplecov results" tar czf coverage_${SEMAPHORE_JOB_INDEX}.tgz -C coverage . artifact push workflow coverage_${SEMAPHORE_JOB_INDEX}.tgz fi 

2. Once all of the rspec jobs are complete, a temporary directory is made from the coverage files that were pushed to the workflow's artifacts:

#!/bin/bash set -euo pipefail for i in $(eval echo "{1..$SEMAPHORE_RAILS_JOB_COUNT}"); do artifact pull workflow coverage_$i.tgz; mkdir coverage_$i tar -xzf coverage_$i.tgz -C coverage_$i done 

3. A rake task is run that collates all those files together to output a total_coverage directory. This is where I attempted to add refuse_coverage_drop as I want to compare across everything together, not within the individual runs themselves (what is in each of these changes all the time and I presumed this would interfere with things).

None of these attempts generated a last_run file.

attempt one (setting refuse_coverage_drop IN the collate block:

 task report: :environment do require 'simplecov' require 'simplecov-json' SimpleCov.formatters = SimpleCov::Formatter::MultiFormatter.new([ SimpleCov::Formatter::HTMLFormatter, SimpleCov::Formatter::JSONFormatter, ]) SimpleCov.collate Dir['coverage_*/.resultset.json'] do refuse_coverage_drop end end

attempt two (setting refuse_coverage_drop AFTER the collate block):

 task report: :environment do require 'simplecov' require 'simplecov-json' SimpleCov.formatters = SimpleCov::Formatter::MultiFormatter.new([ SimpleCov::Formatter::HTMLFormatter, SimpleCov::Formatter::JSONFormatter, ]) SimpleCov.collate Dir['coverage_*/.resultset.json'] SimpleCov.refuse_coverage_drop end

attempt three (setting refuse_coverage_drop BEFORE the collate block):

 task report: :environment do require 'simplecov' require 'simplecov-json' SimpleCov.formatters = SimpleCov::Formatter::MultiFormatter.new([ SimpleCov::Formatter::HTMLFormatter, SimpleCov::Formatter::JSONFormatter, ]) SimpleCov.refuse_coverage_drop SimpleCov.collate Dir['coverage_*/.resultset.json'] end

attempt four (setting refuse_coverage_drop in .simplecov):

require 'knapsack_pro' SimpleCov.start 'rails' do add_filter [ "lib/analytics/", "lib/data_migrations/", "lib/data_restoration/", "lib/tasks/turnstone.rake", "spec/", ] refuse_coverage_drop end KnapsackPro::Hooks::Queue.before_queue do SimpleCov.command_name("rspec_ci_node_#{KnapsackPro::Config::Env.ci_node_index}") end 

The artifacts always output as such:

For each of the above attempts, I checked both...

total_coverage

coverage_1

Reading/writing last_run in CI

More of a general "how is everyone else doing this?" than anything. I'd liked to dynamically generate the last_run file as part of our CI workflows as a step in the deploy process (meaning only when the branch being tested is master) without giving write access to Semaphore (so it could commit the file each time).

It seems plausible that if I can figure out the first issue, then I can easily push the last_run file to our project's artifacts on Semaphore (overwriting when it is the master branch only).

I'm more confused about reading the last run when checking the differences for maximum coverage drop.

I see that LastRun uses the coverage_path to fetch the file. I'm aware that coverage_path is configurable, but that's an all-or-nothing when I only want to "look elsewhere" for the last_run file - outside the standard coverage_path and instead to the last_run file in project's artifact.

simplecov (0.21.2)
ruby 2.5.1p57 (2018-03-29 revision 63029) [x86_64-darwin19]
rails (~> 5.1)

spec/spec_helper.rb if it is helpful:

# This file is copied to spec/ when you run 'rails generate rspec:install' ENV["RAILS_ENV"] ||= 'test' require 'simplecov' require File.expand_path("../../config/environment", __FILE__) require 'knapsack_pro' require 'rspec/rails' require 'shoulda/matchers' require 'vcr' require 'stripe_mock' require 'sidekiq/testing' KnapsackPro::Adapters::RSpecAdapter.bind