test: merge profdata after test runs #246
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cyphar
force-pushed
the
test-reduce-output
branch
18 times, most recently
from
7409766 to
58fef6b
Compare
cyphar
changed the title
cyphar
force-pushed
the
test-reduce-output
branch
2 times, most recently
from
e5bb8c3 to
09e296f
Compare
It turns out that the the llvm-cov test data ends up taking up ~22GB after a single run, causing disk exhaustion when writing profiles: > Error: No space left on device (os error 28) at path "/tmp/.tmptg1faV" > LLVM Profile Error: Failed to write file > "/home/runner/work/libpathrs/libpathrs/target/llvm-cov-target/libpathrs-2591742-9389327919420937493_2.profraw": > No space left on device or when writing the GHA log: > System.IO.IOException: No space left on device : > '/home/runner/actions-runner/cached/_diag/Worker_20250925-173427-utc.log' Commit bd44fd3 ("gha: clear unused packages for CI") tried to fix this by using an action to clear unused packages, but this job takes several minutes to run and isn't guaranteed to solve the problem as we add more tests. Instead, sparse-merging the coverage data with llvm-profdata should resolve this issue. cargo-llvm-cov already does this internally and searches for any .profraw files in the target directory, so this should just work without any real downsides. However, only sparse-merging after each complete run is not sufficient -- we also need to partition the runs when in CI in order to do this merge reduction step before the coverage files fill the disk. Ideally we would use the hash partitioning of nextest but due to some flawed flag parsing behaviour of cargo-hack we can only partition at the cargo-hack level for now. We could also try compressing it, but this would make simple one-shot runs not as effective. The data is not text so while it does seem to compress reasonably well it's probably not worth persuing that yet. Signed-off-by: Aleksa Sarai <[email protected]>
cyphar
force-pushed
the
test-reduce-output
branch
from
09e296f to
97acf90
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
It turns out that the the llvm-cov test data ends up taking up ~22GB
after a single run, causing disk exhaustion when writing profiles:
or when writing the GHA log:
Commit bd44fd3 ("gha: clear unused packages for CI") tried to fix
this by using an action to clear unused packages, but this job takes
several minutes to run and isn't guaranteed to solve the problem as we
add more tests.
Instead, sparse-merging the coverage data with llvm-profdata should
resolve this issue. cargo-llvm-cov already does this internally and
searches for any .profraw files in the target directory, so this should
just work without any real downsides.
However, only sparse-merging after each complete run is not sufficient
-- we also need to partition the runs when in CI in order to do this
merge reduction step before the coverage files fill the disk. Ideally we
would use the hash partitioning of nextest but due to some flawed flag
parsing behaviour of cargo-hack we can only partition at the cargo-hack
level for now.
We could also try compressing it, but this would make simple one-shot
runs not as effective. The data is not text so while it does seem to
compress reasonably well it's probably not worth persuing that yet.
Signed-off-by: Aleksa Sarai [email protected]