Skip to content

Conversation

strtgbb
Copy link
Collaborator

@strtgbb strtgbb commented Sep 2, 2025

Changelog category (leave one):

  • CI Fix or Improvement (changelog entry is not required)

Add a measure of test instability to the workflow report to help triage test failures.

Instability is broken down by check_name and test_name as some tests are only unstable under sanitizers.
Only the last 4 weeks of data is looked at to allow the metric to decay once a test is fixed.

Exclude tests:

  • Fast test
  • Integration Tests
  • Stateless tests
  • Stateful tests
  • Performance tests
  • All with ASAN
  • All with TSAN
  • All with MSAN
  • All with UBSAN
  • All with Coverage
  • All with Aarch64
  • All Regression
  • Disable CI Cache

@strtgbb strtgbb added cicd Improvements and fixes to the CICD process antalya-25.6 antalya-25.6.5 labels Sep 2, 2025
Copy link

github-actions bot commented Sep 2, 2025

Workflow [PR], commit [ad3281d]

@strtgbb
Copy link
Collaborator Author

strtgbb commented Sep 2, 2025

Issues with the current implementation:

  • Adding tests changes what group they are in and thus check_name, this results in an inaccurate calculation.
  • It is slow, report generation time increased from 15-20s to 30-50s. Maybe it should be a CLI option.

I could track just test_name, this solves the changing check_name issue, but we lose info about the effects of sanitizers.
Alternatively, I could make the code smart enough to ignore the check_name group number, at the cost of complexity and runtime.

@strtgbb strtgbb marked this pull request as ready for review September 2, 2025 18:15
@strtgbb strtgbb added port-everywhere PRs that must be ported to every other version branch developer enhancement Features to support the development experience labels Sep 4, 2025
@strtgbb
Copy link
Collaborator Author

strtgbb commented Sep 8, 2025

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cicd Improvements and fixes to the CICD process developer enhancement Features to support the development experience port-everywhere PRs that must be ported to every other version branch
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants