Feature request: Allow tests from different groups to not be interspersed

[cyphar]

Motivation

As part of my safe path library it is necessary to run some race tests that ensure that attackers cannot trick the library into walking outside of the rootfs.

Unfortunately, having the racing threads running as the same time as the other tests can cause spurious test failures (the short explanation is that openat2 will return -EAGAIN if it detected a race, and there is a hard limit on the number of retries the library does, so if you have a program doing a rename-exchange attack some tests in the suite will randomly fail).

Ideally it would be possible to tell nextest to only schedule the race tests in parallel with one another, and the other tests in parallel with one another, but not to intersperse the tests in the two groups. I was hoping that test groups let you do this but it appears they don't.

Proposal

Allow a test group to be marked as "exclusive" or something, which means that other test groups cannot be run at the same time as it (but running tests in the same group in parallel is still allowed). The configuration could be as simple as

[test-groups]
exclusive = { exclusive = true, max-threads = 16 }

[[profile.default.overrides]]
filter = "test(tests::test_race_resolve_partial::)"
test-group = "exclusive"

I'm not familiar with nextest's implementation, so I don't know how hard it would be to make that work. The ideal solution would be for this to be done through something like #[nextest(...)] which would let this kind of important information live with the tests themselves, but it seems (from #101) that this is not an acceptable solution.

Alternatives

The only solution I've found so far that can ensure that these tests are not run at the same time as other tests is to make them be run serially. max-threads doesn't work, but if you use threads-required it will make these tests hog all of the thread slots and thus ensure that they are the only thing running:

[[profile.default.overrides]]
filter = "test(tests::test_race_resolve_partial::)"
threads-required = "num-test-threads"

However, this significantly increases the test run time because all of the tests in the group need to be run in serial now, when running them in parallel is totally fine (it's just that they can't run in parallel with tests from outside their group).

The other option is to do two separate test runs, but this runs into issues where a naive cargo nextest run will fail while the Makefile-configured one won't. I already have workarounds in my Makefile (such as CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_RUNNER='sudo -E') but adding more is quite ugly.

Additional context

No response

[sunshowers]

Thank you for the feature request!

I think this is somewhat complicated, especially due to the fact that tests within the same test group are not necessarily scheduled together today. This will likely be a substantial amount of design and implementation work and will require working through these issues, which I don't expect to do unless a use case for this arises at my workplace.

I think if you'd like to pursue this, it's probably worth writing up a design document describing how you expect this to work:

• Should tests within a test group be scheduled together?
• Should exclusivity be pairwise between test groups (possibly including the global test group), or should it be a binary true/false?

In your case, can you use a separate temp directory per test? It's not clear from the description why the tests must share state.

[cyphar]

In your case, can you use a separate temp directory per test? It's not clear from the description why the tests must share state.

The problem is not that they share state in the test itself, it's that the race tests are exercising an attack (swapping two directories inside a per-test directory) and this attack will cause openat2(RESOLVE_IN_ROOT) system calls on the entire system to return spurious -EAGAINs. The "shared state" is in the kernel (specifically, rename_lock is a global seqlock that applies to all filesystems on the entire system) and there's no way to avoid this other than ensuring the tests don't run at the same time (or adding a bunch of retry loops in the tests).

Should tests within a test group be scheduled together?

Yes.

Should exclusivity be pairwise between test groups (possibly including the global test group), or should it be a binary true/false?

I'm not quite sure what you mean by pairwise, but I would imagine marking a test group as exclusive would mean that only that test group would run. In other words, with a config like this:

[test-groups]
exclusive1 = { exclusive = true, max-threads = 16 }
exclusive2 = { exclusive = true, max-threads = 16 }
exclusive3 = { exclusive = true, max-threads = 16 }

[[profile.default.overrides]]
filter = "test(tests::foo::)"
test-group = "exclusive1"

[[profile.default.overrides]]
filter = "test(tests::bar::)"
test-group = "exclusive2"

[[profile.default.overrides]]
filter = "test(tests::baz::)"
test-group = "exclusive3"

cargo nextest run would be equivalent to:

cargo nextest run "test(tests::foo::)"
cargo nextest run "test(tests::bar::)"
cargo nextest run "test(tests::baz::)"
cargo nextest run "not (test(tests::foo::) or test(tests::bar::) or test(tests::baz::))"

And so on for more complicated configurations.

[sunshowers]

By pairwise I mean should we let you say group A and group B are exclusive, while group A and group C aren't? B here can be the global group i.e. exclusive with all other tests.

Maybe we can start with globally exclusive and expand to pairwise exclusive later.

[cyphar]

Ah okay, I thought so but wasn't sure. Yeah, binary would be simpler to implement but obviously pairwise would be more flexible.

Though I suspect that most users would want to specify which pairings are allowed rather than disallowed (if you had to specify all pairings that were disallowed, each time you add a new group you'd need to update the configuration for the rest) -- so I suspect that this would need to be done with a separate field anyway (exclusive = { enabled = true, except = ["foo", "bar"] } or something) so this can always be added later.

[ziimakc]

Maybe we can start with globally exclusive.

This will be more then enough.