[Modify Feature]: Have pipelines accept non-zero exit codes below 1000(?) as passing instead of failing.

[stephengillie]

Description of the new feature/enhancement

During validation, the pipelines run every executable added to the filesystem during the install process, then close each and record the exit code. If none give a zero exit code, I believe, then the PR gets the Validation-Executable-Error label. Then a moderator is supposed to install the package into a sandbox or VM and verify.

It's something of a known issue with our pipelines that they always error with normal CLI application design. This is because the pipeline runs the CLI application without any arguments, and expects a zero exit code. But most CLI applications are designed to return a non-zero exit code when run without arguments. Frequently, this number is single-digit, so 10 could be a better value than 1000. But I've seen applications with exit codes as high as 127 that ran without issue in a VM.

Proposed technical implementation details (optional)

If a lower value than 1000 would be better, that would be great. I might not have thought through all edge cases, and I'm not an expert on exit codes, so please be gentle.

I've generally been approving PRs with exit codes below 1000 without verifying, and haven't heard of any issues from users. So I would like to have this added into the pipelines, to facilitate better CLI application acceptance into the repo.

This might interoperate well with the suggestion for an ApplicationType of GUI, CLI, etc, where these policies could be applied solely to the CLI applications.

[Trenly]

I wonder if there is a need to extend ExpectedReturnCodes here into the portable application space. When the ExpectedReturnCodes are used with an installer, it indicates why the installer may have failed. However, since WinGet is the installer for portable applications, it may make sense for the ExpectedReturnCodes for a portable package to be what the application may return. This would enable the pipelines to check BaseInstallerType == Portable && returnCode in ExpectedReturnCodes to see if the application ran successfully. This would allow manifest authors to specify something like:

InstallerType: portable
ExpectedReturnCodes:
  - InstallerReturnCode: 1
    ReturnResponse: invalidParameter

[mdanish-kh]

Since parameters like --help, --version (or similarly top level commands cli.exe help, cli.exe version) are very common in CLI design and expected to return a zero return code, what if the pipelines behavior is extended to check against these params/commands by default. If invoking the CLI gives a non-zero return code, the pipelines can then test by passing in these params/commands and see if they return a success code

I think this can cover a lot of portable apps in the repo. I don't have the data to back my claim but in my experience reviewing portable PRs, this is something that I've noticed as a pattern