stabilize observability

Author: Liam-DeVoe

hypothesis-python/RELEASE.rst

@@ -0,0 +1,12 @@
+RELEASE_TYPE: minor
+
+This release stabilizes our :ref:`observability interface <observability>`, which was previously marked as experimental. When observability is enabled, Hypothesis writes data about each test case to ``.hypothesis/observed`` in an analysis-ready `jsonlines <https://jsonlines.org/>`_ format, intended to help you understand the performance of your Hypothesis tests.
+
+Observability can be controlled in two ways:
+
+* via the new |settings.observability| argument,
+* or via the ``HYPOTHESIS_OBSERVABILITY`` environment variable.
+
+See :ref:`Configuring observability <observability-configuration>` for details.
+
+If you use VSCode, we recommend the `Tyche <https://github.com/tyche-pbt/tyche-extension>`__ extension, a PBT-specific visualization tool designed for Hypothesis's observability interface.

hypothesis-python/docs/changelog.rst

@@ -668,11 +668,11 @@ Further improve the performance of the constants-collection feature introduced i
 6.135.3 - 2025-06-08
 --------------------
 
-This release adds the experimental and unstable |OBSERVABILITY_CHOICES| option for :ref:`observability <observability>`. If set, the choice sequence is included in ``metadata.choice_nodes``, and choice sequence spans are included in ``metadata.choice_spans``.
+This release adds the experimental and unstable ``OBSERVABILITY_CHOICES`` option for :ref:`observability <observability>`. If set, the choice sequence is included in ``metadata.choice_nodes``, and choice sequence spans are included in ``metadata.choice_spans``.
 
 These are relatively low-level implementation detail of Hypothesis, and are exposed in observability for users building tools or research on top of Hypothesis. See |PrimitiveProvider| for more details about the choice sequence and choice spans.
 
-We are actively working towards a better interface for this. Feel free to use |OBSERVABILITY_CHOICES| to experiment, but don't rely on it yet!
+We are actively working towards a better interface for this. Feel free to use ``OBSERVABILITY_CHOICES`` to experiment, but don't rely on it yet!
 
 .. _v6.135.2:
 

hypothesis-python/docs/prolog.rst

@@ -26,6 +26,7 @@
 .. |settings.suppress_health_check| replace:: :obj:`settings.suppress_health_check <hypothesis.settings.suppress_health_check>`
 .. |settings.stateful_step_count| replace:: :obj:`settings.stateful_step_count <hypothesis.settings.stateful_step_count>`
 .. |settings.backend| replace:: :obj:`settings.backend <hypothesis.settings.backend>`
+.. |settings.observability| replace:: :obj:`settings.observability <hypothesis.settings.observability>`
 
 .. |~settings.max_examples| replace:: :obj:`~hypothesis.settings.max_examples`
 .. |~settings.database| replace:: :obj:`~hypothesis.settings.database`
@@ -38,6 +39,7 @@
 .. |~settings.suppress_health_check| replace:: :obj:`~hypothesis.settings.suppress_health_check`
 .. |~settings.stateful_step_count| replace:: :obj:`~hypothesis.settings.stateful_step_count`
 .. |~settings.backend| replace:: :obj:`~hypothesis.settings.backend`
+.. |~settings.observability| replace:: :obj:`settings.observability <hypothesis.settings.observability>`
 
 .. |settings.register_profile| replace:: :func:`~hypothesis.settings.register_profile`
 .. |settings.get_profile| replace:: :func:`~hypothesis.settings.get_profile`
@@ -147,7 +149,6 @@
 .. |with_observability_callback| replace:: :data:`~hypothesis.internal.observability.with_observability_callback`
 .. |observability_enabled| replace:: :data:`~hypothesis.internal.observability.observability_enabled`
 .. |TESTCASE_CALLBACKS| replace:: :data:`~hypothesis.internal.observability.TESTCASE_CALLBACKS`
-.. |OBSERVABILITY_CHOICES| replace:: :data:`~hypothesis.internal.observability.OBSERVABILITY_CHOICES`
 .. |BUFFER_SIZE| replace:: :data:`~hypothesis.internal.conjecture.engine.BUFFER_SIZE`
 .. |MAX_SHRINKS| replace:: :data:`~hypothesis.internal.conjecture.engine.MAX_SHRINKS`
 .. |MAX_SHRINKING_SECONDS| replace:: :data:`~hypothesis.internal.conjecture.engine.MAX_SHRINKING_SECONDS`

hypothesis-python/docs/reference/integrations.rst

@@ -75,17 +75,10 @@ If you're interested in similar questions, `drop me an email`_!
 Observability
 -------------
 
-.. note::
+.. tip::
 
   The `Tyche <https://github.com/tyche-pbt/tyche-extension>`__ VSCode extension provides an in-editor UI for observability results generated by Hypothesis. If you want to *view* observability results, rather than programmatically consume or display them, we recommend using Tyche.
 
-.. warning::
-
-    This feature is experimental, and could have breaking changes or even be removed
-    without notice.  Try it out, let us know what you think, but don't rely on it
-    just yet!
-
-
 Motivation
 ~~~~~~~~~~
 
@@ -108,24 +101,36 @@ debuggers such as `rr <https://rr-project.org/>`__ or `pytrace <https://pytrace.
 because there's no good way to compare multiple traces from these tools and their
 Python support is relatively immature.
 
+.. _observability-configuration:
+
+Configuring observability
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+There are two ways to configure observability:
 
-Configuration
-~~~~~~~~~~~~~
+* Setting the ``HYPOTHESIS_OBSERVABILITY`` environment variable. ``HYPOTHESIS_OBSERVABILITY`` may take one of the following values:
 
-If you set the ``HYPOTHESIS_EXPERIMENTAL_OBSERVABILITY`` environment variable,
-Hypothesis will log various observations to jsonlines files in the
+  * ``True``, ``true``, or ``1`` enables observability.
+  * ``False``, ``false``, or ``0`` disables observability (the default).
+  * ``HYPOTHESIS_OBSERVABILITY`` can be further customized with specific string values.
+
+    * ``HYPOTHESIS_OBSERVABILITY=coverage`` is equivalent to :obj:`settings(observability=["coverage"]) <hypothesis.settings.observability>`.
+    * ``HYPOTHESIS_OBSERVABILITY=choices`` is equivalent to :obj:`settings(observability=["choices"]) <hypothesis.settings.observability>`.
+    * ``HYPOTHESIS_OBSERVABILITY=choices,coverage`` is equivalent to :obj:`settings(observability=["choices", "coverage"]) <hypothesis.settings.observability>`.
+    * and so on.
+
+* Setting |settings.observability| on a specific test. See the docs there for details. |settings.observability| can be configured in the same way as ``HYPOTHESIS_OBSERVABILITY``.
+
+If neither ``HYPOTHESIS_OBSERVABILITY`` or |settings.observability| are set, observability is disabled.
+
+When observability is configured, Hypothesis will log various observations to jsonlines files in the
 ``.hypothesis/observed/`` directory.  You can load and explore these with e.g.
 :func:`pd.read_json(".hypothesis/observed/*_testcases.jsonl", lines=True) <pandas.read_json>`,
 or by using the :pypi:`sqlite-utils` and :pypi:`datasette` libraries::
 
     sqlite-utils insert testcases.db testcases .hypothesis/observed/*_testcases.jsonl --nl --flatten
     datasette serve testcases.db
 
-If you are experiencing a significant slow-down, you can try setting
-``HYPOTHESIS_EXPERIMENTAL_OBSERVABILITY_NOCOVER`` instead; this will disable coverage information
-collection. This should not be necessary on Python 3.12 or later, where coverage collection is very fast.
-
-
 Collecting more information
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -181,7 +186,7 @@ While the observability format is agnostic to the property-based testing library
 Choices metadata
 ++++++++++++++++
 
-These additional metadata elements are included in ``metadata`` (as e.g. ``metadata["choice_nodes"]`` or ``metadata["choice_spans"]``), if and only if |OBSERVABILITY_CHOICES| is set.
+These additional metadata elements are included in ``metadata`` (as e.g. ``metadata["choice_nodes"]`` or ``metadata["choice_spans"]``), if and only if observability is configured to include choices (see :ref:`observability-configuration`).
 
 .. jsonschema:: ./schema_metadata_choices.json
    :hide_key: /additionalProperties, /type

hypothesis-python/docs/reference/internals.rst

@@ -36,8 +36,6 @@ Observability
 .. autofunction:: hypothesis.internal.observability.observability_enabled
 
 .. autodata:: hypothesis.internal.observability.TESTCASE_CALLBACKS
-.. autodata:: hypothesis.internal.observability.OBSERVABILITY_COLLECT_COVERAGE
-.. autodata:: hypothesis.internal.observability.OBSERVABILITY_CHOICES
 
 Engine constants
 ----------------

hypothesis-python/docs/reference/schema_metadata_choices.json

@@ -3,7 +3,7 @@
     "properties": {
         "choice_nodes": {
             "type": ["array", "null"],
-            "description": ".. warning::\n\n  EXPERIMENTAL AND UNSTABLE. This attribute may change format or disappear without warning.\n\nThe sequence of choices made during this test case. This includes the choice value, as well as its constraints and whether it was forced or not.\n\nOnly present if |OBSERVABILITY_CHOICES| is ``True``.\n\n.. note::\n\n  The choice sequence is a relatively low-level implementation detail of Hypothesis, and is exposed in observability for users building tools or research on top of Hypothesis. See |PrimitiveProvider| for more details about the choice sequence.",
+            "description": ".. warning::\n\n  EXPERIMENTAL AND UNSTABLE. This attribute may change format without warning.\n\nThe sequence of choices made during this test case. This includes the choice value, as well as its constraints and whether it was forced or not.\n\n.. note::\n\n  Only present if observability is configured to include choices (see :ref:`observability-configuration`).\n\n.. note::\n\n  The choice sequence is a relatively low-level implementation detail of Hypothesis, and is exposed in observability for users building tools or research on top of Hypothesis. See |PrimitiveProvider| for more details about the choice sequence.",
             "items": {
                 "type": "object",
                 "properties": {
@@ -31,7 +31,7 @@
         "choice_spans": {
             "type": "array",
             "items": {"type": "array"},
-            "description": ".. warning::\n\n  EXPERIMENTAL AND UNSTABLE. This attribute may change format or disappear without warning.\n\nThe semantically-meaningful spans of the choice sequence of this test case.\n\nEach span has the format ``[label, start, end, discarded]``, where:\n\n* ``label`` is an opaque integer-value string shared by all spans drawn from a particular strategy.\n* ``start`` and ``end`` are indices into the choice sequence for this span, such that ``choices[start:end]`` are the corresponding choices.\n* ``discarded`` is a boolean indicating whether this span was discarded (see |PrimitiveProvider.span_end|).\n\nOnly present if |OBSERVABILITY_CHOICES| is ``True``.\n\n.. note::\n\n  Spans are a relatively low-level implementation detail of Hypothesis, and are exposed in observability for users building tools or research on top of Hypothesis. See |PrimitiveProvider| (and particularly |PrimitiveProvider.span_start| and |PrimitiveProvider.span_end|) for more details about spans."
+            "description": ".. warning::\n\n  EXPERIMENTAL AND UNSTABLE. This attribute may change format without warning.\n\nThe semantically-meaningful spans of the choice sequence of this test case.\n\nEach span has the format ``[label, start, end, discarded]``, where:\n\n* ``label`` is an opaque integer-value string shared by all spans drawn from a particular strategy.\n* ``start`` and ``end`` are indices into the choice sequence for this span, such that ``choices[start:end]`` are the corresponding choices.\n* ``discarded`` is a boolean indicating whether this span was discarded (see |PrimitiveProvider.span_end|).\n\n.. note::\n\n  Only present if observability is configured to include choices (see :ref:`observability-configuration`).\n\n.. note::\n\n  Spans are a relatively low-level implementation detail of Hypothesis, and are exposed in observability for users building tools or research on top of Hypothesis. See |PrimitiveProvider| (and particularly |PrimitiveProvider.span_start| and |PrimitiveProvider.span_end|) for more details about spans."
         }
     },
     "required": ["traceback", "reproduction_decorator", "predicates", "backend", "sys.argv", "os.getpid()", "imported_at", "data_status", "interesting_origin", "choice_nodes", "choice_spans"],

hypothesis-python/docs/reference/schema_observations.json

@@ -36,7 +36,7 @@
                 },
                 "coverage": {
                     "type": ["object", "null"],
-                    "description": "Mapping of filename to list of covered line numbers, if coverage information is available, or None if not.  Hypothesis deliberately omits stdlib and site-packages code.",
+                    "description": "Mapping of filename to list of covered line numbers, if coverage information is available, or None if not.  Hypothesis deliberately omits stdlib and site-packages code.\n\n.. note::\n\n  Only present if observability is configured to include coverage (see :ref:`observability-configuration`).",
                     "additionalProperties": {
                         "type": "array",
                         "items": {"type": "integer", "minimum": 1},

hypothesis-python/src/hypothesis/_settings.py

@@ -34,6 +34,7 @@
     InvalidArgument,
 )
 from hypothesis.internal.conjecture.providers import AVAILABLE_PROVIDERS
+from hypothesis.internal.observability import ObservabilityOption, envvar_observability
 from hypothesis.internal.reflection import get_pretty_function_description
 from hypothesis.internal.validation import check_type, try_convert
 from hypothesis.utils.conventions import not_set
@@ -57,6 +58,7 @@
     "deadline",
     "print_blob",
     "backend",
+    "observability",
 ]
 
 
@@ -427,6 +429,18 @@ def _validate_backend(backend: str) -> str:
     return backend
 
 
+def _validate_observability(
+    observability: Any,
+) -> bool | Collection[ObservabilityOption]:
+    if isinstance(observability, bool):
+        return observability
+    observability = try_convert(tuple, observability, "observability")
+    return tuple(
+        try_convert(ObservabilityOption, option, "observability")
+        for option in observability
+    )
+
+
 class settingsMeta(type):
     def __init__(cls, *args, **kwargs):
         super().__init__(*args, **kwargs)
@@ -467,7 +481,8 @@ class settings(metaclass=settingsMeta):
     |~settings.max_examples|, |~settings.derandomize|, |~settings.database|,
     |~settings.verbosity|, |~settings.phases|, |~settings.stateful_step_count|,
     |~settings.report_multiple_bugs|, |~settings.suppress_health_check|,
-    |~settings.deadline|, |~settings.print_blob|, and |~settings.backend|.
+    |~settings.deadline|, |~settings.print_blob|, |~settings.backend|, and
+    |~settings.observability|.
 
     A settings object can be applied as a decorator to a test function, in which
     case that test function will use those settings. A test may only have one
@@ -516,7 +531,7 @@ class settings(metaclass=settingsMeta):
             "default",
             max_examples=100,
             derandomize=False,
-            database=not_set,  # see settings.database for the default database
+            database=not_set,  # see settings.database for default behavior
             verbosity=Verbosity.normal,
             phases=tuple(Phase),
             stateful_step_count=50,
@@ -525,6 +540,7 @@ class settings(metaclass=settingsMeta):
             deadline=duration(milliseconds=200),
             print_blob=False,
             backend="hypothesis",
+            observability=not_set,  # see settings.observability for default behavior
         )
 
         ci = settings.register_profile(
@@ -571,6 +587,7 @@ def __init__(
         deadline: int | float | datetime.timedelta | None = not_set,  # type: ignore
         print_blob: bool = not_set,  # type: ignore
         backend: str = not_set,  # type: ignore
+        observability: bool | Collection[str] = not_set,  # type: ignore
     ) -> None:
         self._in_definition = True
 
@@ -597,10 +614,12 @@ def __init__(
             if derandomize is not_set  # type: ignore
             else _validate_choices("derandomize", derandomize, choices=[True, False])
         )
+
         if database is not not_set:  # type: ignore
             database = _validate_database(database)
         self._database = database
         self._cached_database = None
+
         self._verbosity = (
             self._fallback.verbosity  # type: ignore
             if verbosity is not_set  # type: ignore
@@ -644,6 +663,15 @@ def __init__(
             else _validate_backend(backend)
         )
 
+        if observability is not_set:  # type: ignore
+            self._observability = (
+                envvar_observability
+                if self._fallback is None
+                else self._fallback.observability
+            )
+        else:
+            self._observability = _validate_observability(observability)
+
         self._in_definition = False
 
     @property
@@ -901,6 +929,45 @@ def backend(self):
         """
         return self._backend
 
+    @property
+    def observability(self):
+        """
+        Controls the :ref:`observability <observability>` behavior of Hypothesis.
+
+        Observability may be enabled or disabled by passing ``True`` or ``False``.
+
+        Observability behavior can be further customized by passing a collection
+        of valid string options instead. If a collection of string options is passed,
+        observability is enabled, and will additionally include output corresponding
+        to each option. The valid options are:
+
+        * ``"coverage"``: include the ``coverage`` field in test case observations.
+        * ``"choices"``: include the ``metadata.choice_nodes`` and
+          ``metadata.choice_spans`` fields in test case observations.
+
+        For example, observations generated under the following settings will include
+        both coverage data and choice sequence data:
+
+        .. code-block:: python
+
+            from hypothesis import settings
+            s = settings(observability=["coverage", "choices"])
+
+        ``observability=[]`` is treated as ``observability=False``.
+
+        If not set, the ``observability`` will be inherited from the
+        :ref:`HYPOTHESIS_OBSERVABILITY <observability-configuration>`
+        environment variable. If that environment variable is also not set,
+        Hypothesis defaults to leaving observability disabled.
+        """
+        return self._observability
+
+    @property
+    def _observability_options(self) -> tuple[ObservabilityOption, ...]:
+        if not isinstance(self.observability, tuple):
+            return ()
+        return self.observability
+
     def __call__(self, test: T) -> T:
         """Make the settings object (self) an attribute of the test.
 
@@ -1073,6 +1140,7 @@ def note_deprecation(
     deadline=duration(milliseconds=200),
     print_blob=False,
     backend="hypothesis",
+    observability=not_set,  # type: ignore
 )
 settings.register_profile("default", default)
 settings.load_profile("default")