Skip to content

Conversation

@pdobacz
Copy link
Member

@pdobacz pdobacz commented Oct 7, 2025

Tackles the point (3.) from github.com/ethereum/execution-spec-tests/pull/2243 (consume direct using evmone in EEST).

Here we switch statetest and blockchaintest runners to discover gtests by parsing the JSON files and recognizing each test.

The downside is that the invocation we use in CI (just ...evmone-statetest ... fixtures/state_tests) became very chatty, so --gtest_brief=1 was added. There is an option to only go inside the JSONs when a single .json file path is given (as consume direct does), but this would complicate things quite a bit, requiring us to have more code, but not terribly much more.

Makes consume direct for evmone version in PR mentioned 👇 work correctly

@codecov
Copy link

codecov bot commented Oct 7, 2025

Codecov Report

❌ Patch coverage is 92.06349% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.09%. Comparing base (0591d81) to head (70f14a6).
⚠️ Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
test/blockchaintest/blockchaintest.cpp 87.87% 2 Missing and 2 partials ⚠️
test/statetest/statetest.cpp 96.66% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@ Coverage Diff @@ ## master #1331 +/- ## ========================================== + Coverage 87.07% 87.09% +0.02%  ========================================== Files 169 169 Lines 24820 24864 +44 Branches 4076 4081 +5 ========================================== + Hits 21611 21656 +45  + Misses 548 546 -2  - Partials 2661 2662 +1 
Flag Coverage Δ
eest-develop 84.17% <ø> (ø)
eest-develop-gmp 15.83% <19.04%> (-0.02%) ⬇️
eest-legacy 11.10% <15.87%> (-0.01%) ⬇️
eest-legacy-silkpre 17.56% <20.63%> (-0.02%) ⬇️
evmone-unittests 83.89% <77.77%> (+0.19%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
core 93.55% <ø> (ø)
tooling 86.50% <92.06%> (+0.36%) ⬆️
tests 84.13% <ø> (ø)
Files with missing lines Coverage Δ
test/statetest/statetest.cpp 92.85% <96.66%> (-2.02%) ⬇️
test/blockchaintest/blockchaintest.cpp 87.05% <87.87%> (+8.37%) ⬆️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
@pdobacz
Copy link
Member Author

pdobacz commented Oct 8, 2025

It seems that discovering tests individually has more problems than I thought initially, if you apply it always. Problems which arise:

  1. When calling with a dir with many tests it becomes quite unresponsive, as it discovers the tests for a long time before giving any output
  2. It becomes problematic to filter out tests

Would it work for us if discover was done as it used to be (per file) when provided with a dir path (e.g. like we do in our CI) and discover per test when provided with a single .json file path (e.g. like what consume direct uses)? @chfast

EDIT: We seem to have decided it would work, this PR takes this approach

@pdobacz pdobacz force-pushed the granular-test branch 4 times, most recently from 97e3872 to b8468ed Compare October 8, 2025 15:44
@pdobacz pdobacz marked this pull request as ready for review October 8, 2025 15:45
@pdobacz pdobacz requested a review from chfast October 8, 2025 15:45
@pdobacz pdobacz requested a review from chfast October 9, 2025 12:37
Co-authored-by: Paweł Bylica <pawel@hepcolgum.band>
@pdobacz pdobacz enabled auto-merge (squash) October 9, 2025 12:52
@pdobacz pdobacz merged commit 89bfa23 into master Oct 9, 2025
21 checks passed
@pdobacz pdobacz deleted the granular-test branch October 9, 2025 13:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

3 participants