Feature Request

Describe the problem you need a feature to resolve.

I'm writing some automation that calls out to operator-sdk scorecard, and have noticed that it will return the same exit code (1) if the tool either encountered a fatal error, or succeeded in execution but the bundle being tested did not pass a test.

Effectively, this means that I must parse stdout/stderr to determine if the tool was able to parse the bundle at all, or encountered a fatal error in the process. I'm assuming that having nothing in stderr implies the tool did not encounter an error in execution, so I can then parse stdout which should contain the results.

As an example, I'm running something like this

# operator-sdk scorecard --output json --selector=suite=olm my-bundle-image-here:latest *** JSON OUTPUT OMITTED, but the tool completed and the output reflected that my bundle failed three tests *** # echo $? 1 

If I ran that with a bogus bundle image, it would also return 1, but I would see something a Fatal log line in stderr.

Describe the solution you'd like.

The easiest thing conceptually is to return a different non-zero exit code if it's a test failure vs. if it's a fatal error. So if any test failed, return 2, vs. returning 1 if the tool encountered a fatal error preventing it from executing.

As a semantic, I do feel like the tool should return 0 even if a test fails (to tool itself worked!). But I also understand how that might make certain automation scenarios a bit trickier (now the developer has to parse the output instead of just relying on the exit code to tell them that something needs their attention). With that in mind, the above solution feels more beneficial.