Feature request: Run the same tests with different environment variables, in parallel

[svix-jplatte]

Motivation

We've got a test script over here that runs the same test suite a bunch of times with different environment variables set. It already got sped up a bunch by using nextest, so thanks for that!

However, we can only use nextest's high level of parallelization within one invocation of the test suite with a particular set of environment variables. Before we start the next invocation with a different set of environment variables, we wait for all previously-started tests to finish. We could fix this by adding more parallelism at the shell level, but then the output of the various nextest invocations would get mangled.

Proposal

Since nextest already runs every test in a separate process, it seems like it shouldn't be too hard for it to also support including the same test multiple times in a run, with different environment variables set. A potential interface for this might be a JSON or TOML input file you can feed to cargo nextest run that defines a list of test configurations to run, something like

[[test-configuration]]
filter = "<optional filterset>"
env = { FOO = "foo1" }

[[test-configuration]]
run-ignored = "only" # same possible values as on the command line

[[test-configuration.env]]
FOO = "foo2"
BAR = "xyz"

Alternatives

No response

Additional context

No response