Skip to content

Conversation

@vkarak
Copy link
Contributor

@vkarak vkarak commented May 15, 2025

The main elements introduced by this PR are:

  1. The @xfail decorator to mark a test as an expected sanity failure.
  2. The xfail() builtin to mark reference tuples as expected performance failures.

Two new test states are introduced which are also logged: XFAIL for expected failures and XPASS for unexpected passes.

Design-wise the concept is the same as the one explained in #2378.

Implementation-wise there are two major changes:

  1. The execution policies code is refactored by eliminating code duplication between the serial and asynchronous policies.
  2. A new utility function reference_bounds() is introduced which is used to validate and calculate the absolute lower/upper bounds. This used to be internal to assert_reference() and now it's made public (maybe the docs there need a slight reorganization).
  3. The treatment of performance failures is rewritten as now we have to handle much more complex cases, were the performance variables of a test can simultaneously be pass, fail, xpass or xfail.

For more details on how feature works macroscopically check the tutorial and the reference docs.

Closes #2378.

@vkarak vkarak added this to the ReFrame 4.9 milestone May 15, 2025
@vkarak vkarak requested review from ekouts, teojgo and victorusu May 15, 2025 14:08
@vkarak vkarak self-assigned this May 15, 2025
@vkarak vkarak force-pushed the feat/expected-failures branch from 7bbb18a to 0d5b5e4 Compare May 15, 2025 14:10
@vkarak vkarak requested a review from jack-morrison June 6, 2025 08:32
Copy link
Contributor

@victorusu victorusu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm.

@github-project-automation github-project-automation bot moved this from Todo to In Progress in ReFrame Backlog Jun 18, 2025
@vkarak vkarak force-pushed the feat/expected-failures branch from 8cb57af to 489e3b0 Compare June 18, 2025 19:17
@vkarak vkarak requested a review from teojgo June 18, 2025 19:18
@codecov
Copy link

codecov bot commented Jun 18, 2025

Codecov Report

Attention: Patch coverage is 97.77778% with 5 lines in your changes missing coverage. Please review.

Project coverage is 91.16%. Comparing base (2a2dd82) to head (17d6122).
Report is 5 commits behind head on develop.

Files with missing lines Patch % Lines
reframe/frontend/executors/__init__.py 93.02% 3 Missing ⚠️
reframe/core/decorators.py 90.90% 1 Missing ⚠️
reframe/frontend/executors/policies.py 98.61% 1 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@ ## develop #3481 +/- ## =========================================== + Coverage 91.13% 91.16% +0.02%  =========================================== Files 62 62 Lines 13132 13253 +121 =========================================== + Hits 11968 12082 +114  - Misses 1164 1171 +7 

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
@vkarak vkarak force-pushed the feat/expected-failures branch from 5615f79 to 17d6122 Compare June 26, 2025 16:18
@vkarak vkarak merged commit b57f957 into reframe-hpc:develop Jun 27, 2025
39 checks passed
@github-project-automation github-project-automation bot moved this from In Progress to Done in ReFrame Backlog Jun 27, 2025
@vkarak vkarak deleted the feat/expected-failures branch June 27, 2025 15:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment