Skip to content

Conversation

@Liam-DeVoe
Copy link
Member

Closes #4387.

  • Disable coverage by default
    • IMO this is too much data (and too high risk for collision with sys.monitoring tools / slowdowns, but that's a weaker argument) to enable by default, but I'm open to discussion
  • @settings(observability=...) takes three possible values:
    • observability=True enables observability
    • observability=False disables observability
    • observability=[option1, option2], like observability=["coverage"], enables observability, and also enables whatever options are specified
  • if @settings(observability=...) is not set, inherit from HYPOTHESIS_OBSERVABILITY envvar
  • HYPOTHESIS_EXPERIMENTAL_OBSERVABILITY_NOCOVER removed with no deprecation period
  • HYPOTHESIS_EXPERIMENTAL_OBSERVABILITY and HYPOTHESIS_EXPERIMENTAL_OBSERVABILITY_CHOICES kept for backwards-compat because it was easy, but are now-undocumented and will be removed at some point

I'd like to keep a very clear separation between @settings(observability=...), and add_observability_callback. IMO the former should always write to/be related to .hypothesis/observed, and the latter should never be (ie it's sent in-memory).

This brings up a non-trivial issue, probably for a future PR: when adding observability callbacks, there's no way to control the coverage and choices options. You don't want to set @settings(observability=["choices"]) on the test/profile, because this would write to .hypothesis/observed, and you just want the observations in-memory. Previously this was done with observability.OBSERVABILITY_COLLECT_COVERAGE and observability.OBSERVABILITY_CHOICES. I'd like to get rid of those vars eventually, and add add_observability_callback(options=["coverage"]). But semantic questions like "what if two callbacks where only one sets options=["coverage"] have to be answered. I do think this can be done, I just don't want to do it right now. So I've left those vars in, though now un-documented.

[cc @hgoldstein95]


I'd also love to write a hypothesis.works blog post about this, at some point..

@Liam-DeVoe Liam-DeVoe force-pushed the stabilize-observability branch from c224f58 to 8d25f1c Compare October 14, 2025 02:54
@Liam-DeVoe Liam-DeVoe force-pushed the stabilize-observability branch from 2459070 to a1bc62b Compare October 23, 2025 04:37
@Liam-DeVoe Liam-DeVoe force-pushed the stabilize-observability branch from a1bc62b to d4a3459 Compare October 23, 2025 04:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

New option to include choice sequence and spans in observability output

1 participant