[RFC] New command line options for controlling llvm-lit output

LLVM Project
1

This was initially sparked by a discussion in [llvm-lit] Show test output when combining -v or -a with -q · Issue #106643 · llvm/llvm-project · GitHub about how the various flags controlling output in llvm-lit should behave when combined.

At the moment -q is the only flag that can silence notes and warnings emitted by llvm-lit, but -q also completely overrides the other flags (-v, -a, -s) – no matter the order of the flags. This means it’s impossible to silence notes and warnings while also seeing the output of a command.

After thinking a bit, I think the best approach would be to restructure the flags into smaller, more focused options. The current ones would remain as a shorthand for setting multiple options with a single flag. The behaviour when combining existing flags would be whatever the behaviour is when combining the options they alias. I also propose that flags occurring later in the lit invokation override earlier flags, rather than the current behaviour of some flags always overriding other flags. This proposal would have the unfortunate side effect that e.g. -s -a and -a -s would result in different behaviour in non-obvious ways, but this can be alleviated by emitting warnings, if necessary.

The new flags I propose are:

  • --test-output=[off,failed,all]
    • whether the executed commands and their outputs are printed after each test
    • default: off
  • --test-progress=[off,failed,all]
    • whether the result of each test is printed during testing (as opposed to the summary at the end)
    • default: all
    • I don’t love this name, but naming things is hard :-/
  • --progress-bar (complementing the existing --no-progress-bar)
    • whether a progress bar is shown during testing
    • default: disabled
  • --diagnostic-level=[error,warning,note,debug]
    • whether to only emit lit diagnostics of levels fatal and error, or additionally warning, or additionally note, or additionally debug output
    • default: note
    • I don’t feel strongly about incorporating debug into this option

There is some crossover between --test-output and --test-progress: --test-output requires, and implies, --test-progress of at least the same level. That is to say, there’s no way of printing the output of test foo/bar.c, if we aren’t even printing the fact that foo/bar.c was executed. So --test-output=all implies --test-progress=all, and --test-output=failed sets --test-progress=failed if the current mode is --test-progress=off. Conversely, --test-output=all --test-progress=off results in no test output. This is why -s -a and -a -s give different results, as the current options would be aliases as follows:

  • -v → --test-output=failed
  • -a → --test-output=all
  • -s → --progress-bar --test-progress=failed
  • -q → --diagnostic-level=error --test-output=off
  • --debug → --diagnostic-level=debug

Happy to hear what the community has to say!

3 Likes
2

what about –-test-itemized? I also don’t think –-test-progress is a good name

In general, I’m supportive of this proposal.

1 Like
3

I personally think --test-progress is fine, or even just --progress. If I were skimming the man pages for some long-running tool and looking for options to adjust how it reports its progress, “progress” is the keyword I would search for first.

4

I believe the functionality you described is –-progress-bar in LIT. And that’s the exact reason I don’t think –-test-progress is a good name here – it’s really easy to get confused with progress bar

5

They are somewhat related, in the sense that --test-progress communicates “progress” after each individual test, with information about what test was executed and what the result was. It isn’t great at communicating how much is left to test, whereas --progress-bar doesn’t tell you much about which tests have been run and how they went. But either of them will show you that progress is being made (unlike -q, which is indistinguishable from a halted state until it finishes).

It is unfortunate that “progress” is highly connoted with progress bars specifically though… Would something like --incremental-results be too similar to the completely unrelated (and deprecated) --incremental flag?

1 Like
6

If –-incremental had been deprecated and removed for a while then I think that name is fine. Unfortunately at this moment it’s still there so yeah, IMHO –-incremental-results might be too closed to that.

What about something like –-print-result-after=[all,failed,never]? It’s inspired by the –-print-after-all/--print-after=<pass name> flags from opt and llc.

1 Like
7

Works for me!