[bugfix] Do not wipe the output dir of a test if run with --dry-run

[vkarak]

Closes #3551.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.22%. Comparing base (ff41341) to head (a332dec).
⚠️ Report is 759 commits behind head on master.

Additional details and impacted files

@@ Coverage Diff @@ ## master #3553 +/- ## ========================================== + Coverage 86.66% 91.22% +4.55%  ========================================== Files 61 62 +1 Lines 12140 13465 +1325 ========================================== + Hits 10521 12283 +1762  + Misses 1619 1182 -437 

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[ekouts]

For me it's not clear what * should * happen with the old output directory. People may be confused by looking at the output of old runs when they are testing --dry-run. On the other hand I understand that removing the old results with a "dry run" option can also be unwanted behaviour, so lgtm!

[vkarak]

For me it's not clear what * should * happen with the old output directory. People may be confused by looking at the output of old runs when they are testing --dry-run. On the other hand I understand that removing the old results with a "dry run" option can also be unwanted behaviour, so lgtm!

Removing the output directory is a totally undesired side effect, as this is supposed to hold scripts and data from what it has been actually executed. In the docs we state that --dry-run generates the scripts in the stage directory and, also, in dry-run mode we do not copy back to the output directory anything from the test, because what is in keep_files is likely not produced. So, essentially, the --dry-run mode was literally just wiping out the output directory.