Use check-run-results for run-fail test stderr

[clubby789]

Fixes #65865

[rustbot]

r? @estebank

rustbot has assigned @estebank.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

[clubby789]

oops, non anonymized panic strings

[rustbot]

Some changes occurred in tests/ui/sanitizer

cc @rust-lang/project-exploit-mitigations, @rcvalle

[clubby789]

Reverted a few tests that have more variable output

[rustbot]

Some changes occurred in src/tools/compiletest

cc @jieyouxu

[clubby789]

Hmm, we can't normalize $CARGO_HOME naively like other paths, because we have tests with paths containing .../cargo... - and /cargo is the CI $CARGO_HOME

[bors]

⌛ Testing commit e1a9aff with merge 2a2f01f...

[bors]

💔 Test failed - checks-actions

[clubby789]

Looks spurious

@bors retry

[clubby789]

@bors r-
Misread the error, sanitizer tests are fragile due to addresses and should not have stderr checked

[jieyouxu]

Also leaving a comment here so discussions don't get lost elsewhere:

We should be a bit careful with replacing //@ error-pattern with //@ check-run-results, because they are not mutually exclusive and they aren't really substitutes (if anything, they are complementary):

• //@ error-pattern will look for a regex match in the run stderr (and stdout combined, IIRC) if the test is //@ run-fail.
• //@ check-run-results instructs compiletest to snapshot run stderr/stdout, and compare a run's stderr/stdout against those exactly.

Concretely, this means:

• //@ check-run-results is for when you really want the exact run stderr/stdout to not change. This will be especially relevant to tests that may include things that change between runs.
  • If a test needs such exact snapshot comparisons, you can still have error-pattern to enforce the check on the pattern that is critical, to prevent you from accidentally blessing the important thing away.
• //@ error-pattern is good for checking the critical parts of the run stderr/stdout, cannot be easily blessed away, and is typically less sensitive than full stderr/stdout comparisons.

[clubby789]

Going to close; I think a blanket approach is unhelpful as it's going to be flaky (especially when stderr contains line numbers from external crates or stdlib), and doesn't provide much value

[jieyouxu]

cc Audit {check-run-results, error-pattern, check-stdout, normalize-*, dont-check-compiler-*, run-rustfix, rustfix-only-machine-applicable, forbid-output} family of directives.

[jieyouxu]

cc Audit run-fail ui tests to see if they need exact run stderr/stdout #134889.