From a6718d0f6c148b6356c87b2f76881df25b480b46 Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Sun, 3 Aug 2025 11:48:49 -0400 Subject: [PATCH 01/10] Remove reference to old title. --- peps/pep-0788.rst | 4 ---- 1 file changed, 4 deletions(-) diff --git a/peps/pep-0788.rst b/peps/pep-0788.rst index 55ffac8594e..f23cca202a9 100644 --- a/peps/pep-0788.rst +++ b/peps/pep-0788.rst @@ -325,10 +325,6 @@ in newer versions with the recent acceptance of :pep:`734`. Rationale ========= -So, how do we address all of this? The best way seems to be starting from -scratch and "reimagining" how to create, acquire and attach -:term:`thread states ` in the C API. - Preventing Interpreter Shutdown with Reference Counting ------------------------------------------------------- From 483557b3931f2b644a9115822b15afb1aa86375f Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Sun, 3 Aug 2025 11:57:04 -0400 Subject: [PATCH 02/10] Use 0 for the failure value. --- peps/pep-0788.rst | 43 +++++++++++++++++++++++++------------------ 1 file changed, 25 insertions(+), 18 deletions(-) diff --git a/peps/pep-0788.rst b/peps/pep-0788.rst index f23cca202a9..1909d39d513 100644 --- a/peps/pep-0788.rst +++ b/peps/pep-0788.rst @@ -417,25 +417,25 @@ Strong Interpreter References .. c:type:: PyInterpreterRef An opaque, strong reference to an interpreter. + The interpreter will wait until a strong reference has been released before shutting down. This type is guaranteed to be pointer-sized. -.. c:function:: int PyInterpreterRef_Get(PyInterpreterRef *ref) +.. c:function:: PyInterpreterRef PyInterpreterRef_Get(void) Acquire a strong reference to the current interpreter. - On success, this function returns ``0`` and sets *ref* - to a strong reference to the interpreter, and returns ``-1`` - with an exception set on failure. + On success, this function returns a strong reference to the current + interpreter, and returns ``0`` with an exception set on failure. - Failure typically indicates that the interpreter has - already finished waiting on strong references. + Failure typically indicates that the interpreter has already finished + waiting on strong references. The caller must hold an :term:`attached thread state`. -.. c:function:: int PyInterpreterRef_Main(PyInterpreterRef *ref) +.. c:function:: PyInterpreterRef PyInterpreterRef_Main(PyInterpreterRef *ref) Acquire a strong reference to the main interpreter. @@ -443,8 +443,8 @@ Strong Interpreter References can't be saved. Prefer safely acquiring a reference through :c:func:`PyInterpreterRef_Get` whenever possible. - On success, this function will return ``0`` and set *ref* to a strong - reference, and on failure, this function will return ``-1``. + On success, this function returns a strong reference to the main + interpreter, and returns ``0`` without an exception set on failure. Failure typically indicates that the main interpreter has already finished waiting on its reference count. @@ -462,8 +462,10 @@ Strong Interpreter References Duplicate a strong reference to an interpreter. - This function cannot fail, and the caller doesn't need to hold an - :term:`attached thread state`. + On success, this function returns a strong reference to the interpreter + denoted by *ref*, and returns ``0`` without an exception set on failure. + + The caller does not need to hold an :term:`attached thread state`. .. c:function:: void PyInterpreterRef_Close(PyInterpreterRef ref) @@ -479,6 +481,7 @@ Weak Interpreter References .. c:type:: PyInterpreterWeakRef An opaque, weak reference to an interpreter. + The interpreter will *not* wait for the reference to be released before shutting down. @@ -491,9 +494,8 @@ Weak Interpreter References This function is generally meant to be used in tandem with :c:func:`PyInterpreterWeakRef_AsStrong`. - On success, this function returns ``0`` and sets *wref* to a - weak reference to the interpreter, and returns ``-1`` with an exception - set on failure. + On success, this function returns a weak reference to the current + interpreter, and returns ``0`` with an exception set on failure. The caller must hold an :term:`attached thread state`. @@ -501,18 +503,23 @@ Weak Interpreter References Duplicate a weak reference to an interpreter. + On success, this function returns a non-zero weak reference to the + interpreter denoted by *wref*, and returns ``0`` without an exception set + on failure. + This function cannot fail, and the caller doesn't need to hold an :term:`attached thread state`. -.. c:function:: int PyInterpreterWeakRef_AsStrong(PyInterpreterWeakRef wref, PyInterpreterRef *ref) +.. c:function:: PyInterpreterRef PyInterpreterWeakRef_AsStrong(PyInterpreterWeakRef wref) Acquire a strong reference to an interpreter through a weak reference. - On success, this function returns ``0`` and sets *ref* to a strong - reference to the interpreter denoted by *wref*. + On success, this function returns a strong reference to the interpreter + denoted by *wref*. The weak reference is still valid after calling this + function. If the interpreter no longer exists or has already finished waiting - for its reference count to reach zero, then this function returns ``-1`` + for its reference count to reach zero, then this function returns ``0`` without an exception set. This function is not safe to call in a re-entrant signal handler. From 9a12f4445ae6bd2e406e973f8463283be8e9f2b5 Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Sun, 3 Aug 2025 12:02:16 -0400 Subject: [PATCH 03/10] Use 0 for the failure value in the examples. --- peps/pep-0788.rst | 35 ++++++++++++++++++----------------- 1 file changed, 18 insertions(+), 17 deletions(-) diff --git a/peps/pep-0788.rst b/peps/pep-0788.rst index 1909d39d513..ab51a441acf 100644 --- a/peps/pep-0788.rst +++ b/peps/pep-0788.rst @@ -651,8 +651,8 @@ With this PEP, you'd implement it like this: PyObject *file, const char *text) { - PyInterpreterRef ref; - if (PyInterpreterWeakRef_AsStrong(wref, &ref) < 0) { + PyInterpreterRef ref = PyInterpreterWeakRef_AsStrong(wref); + if (ref == 0) { /* Python interpreter has shut down */ return -1; } @@ -660,7 +660,7 @@ With this PEP, you'd implement it like this: PyThreadRef thread_ref; if (PyThreadState_Ensure(ref, &thread_ref) < 0) { PyInterpreterRef_Close(ref); - puts("Out of memory.\n", stderr); + fputs("Cannot call Python.\n", stderr); return -1; } @@ -696,8 +696,8 @@ held. Any future finalizer that wanted to acquire the lock would be deadlocked! my_critical_operation(PyObject *self, PyObject *unused) { assert(PyThreadState_GetUnchecked() != NULL); - PyInterpreterRef ref; - if (PyInterpreterRef_Get(&ref) < 0) { + PyInterpreterRef ref = PyInterpreterRef_Get(&ref); + if (ref == 0) { /* Python interpreter has shut down */ return NULL; } @@ -779,8 +779,8 @@ This is the same code, rewritten to use the new functions: PyThread_handle_t handle; PyThead_indent_t indent; - PyInterpreterRef ref; - if (PyInterpreterRef_Get(&ref) < 0) { + PyInterpreterRef ref = PyInterpreterRef_Get(); + if (ref == 0) { return NULL; } @@ -800,7 +800,8 @@ Example: A Daemon Thread With this PEP, daemon threads are very similar to how non-Python threads work in the C API today. After calling :c:func:`PyThreadState_Ensure`, simply -release the interpreter reference, allowing the interpreter to shut down. +release the interpreter reference to allow the interpreter to shut down (and +hang the current thread forever). .. code-block:: c @@ -829,8 +830,8 @@ release the interpreter reference, allowing the interpreter to shut down. PyThread_handle_t handle; PyThead_indent_t indent; - PyInterpreterRef ref; - if (PyInterpreterRef_Get(&ref) < 0) { + PyInterpreterRef ref = PyInterpreterRef_Get(); + if (ref == 0) { return NULL; } @@ -859,8 +860,8 @@ deadlock the interpreter if it's not released. { ThreadData *data = (ThreadData *)arg; PyInterpreterWeakRef wref = data->wref; - PyInterpreterRef ref; - if (PyInterpreterWeakRef_AsStrong(wref, &ref) < 0) { + PyInterpreterRef ref = PyInterpreterWeakRef_AsStrong(wref); + if (ref == 0) { fputs("Python has shut down!\n", stderr); return -1; } @@ -888,8 +889,8 @@ deadlock the interpreter if it's not released. PyErr_NoMemory(); return NULL; } - PyInterpreterWeakRef wref; - if (PyInterpreterWeakRef_Get(&wref) < 0) { + PyInterpreterWeakRef wref = PyInterpreterWeakRef_Get(); + if (wref == 0) { PyMem_RawFree(tdata); return NULL; } @@ -918,9 +919,9 @@ interpreter here. static void call_python(void) { - PyInterpreterRef ref; - if (PyInterpreterRef_Main(&ref) < 0) { - fputs("Python has shut down!", stderr); + PyInterpreterRef ref = PyInterpreterRef_Main(); + if (ref == 0) { + fputs("Python has shut down.", stderr); return; } From f83e180326b196890b50fefb49cb4c1c6032b5fc Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Sun, 3 Aug 2025 12:06:25 -0400 Subject: [PATCH 04/10] Rename PyInterpreterRef_AsInterpreter to PyInterpreterRef_GetInterpreter. --- peps/pep-0788.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/peps/pep-0788.rst b/peps/pep-0788.rst index ab51a441acf..e03fb4bafab 100644 --- a/peps/pep-0788.rst +++ b/peps/pep-0788.rst @@ -451,9 +451,9 @@ Strong Interpreter References The caller does not need to hold an :term:`attached thread state`. -.. c:function:: PyInterpreterState *PyInterpreterRef_AsInterpreter(PyInterpreterRef ref) +.. c:function:: PyInterpreterState *PyInterpreterRef_GetInterpreter(PyInterpreterRef ref) - Return the interpreter denoted by *ref*. + Return the :c:type:`PyInterpreterState` pointer denoted by *ref*. This function cannot fail, and the caller doesn't need to hold an :term:`attached thread state`. From b1818bce217e4a7ac3d737be750ed494955cfd28 Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Sun, 3 Aug 2025 12:07:16 -0400 Subject: [PATCH 05/10] Rename PyInterpreterRef_Main to PyUnstable_InterpreterRef_GetMain. This is subject to change, depending on what the C API working group agrees on. --- peps/pep-0788.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/peps/pep-0788.rst b/peps/pep-0788.rst index e03fb4bafab..bc2e93d4656 100644 --- a/peps/pep-0788.rst +++ b/peps/pep-0788.rst @@ -435,7 +435,7 @@ Strong Interpreter References The caller must hold an :term:`attached thread state`. -.. c:function:: PyInterpreterRef PyInterpreterRef_Main(PyInterpreterRef *ref) +.. c:function:: PyInterpreterRef PyUnstable_InterpreterRef_GetMain(PyInterpreterRef *ref) Acquire a strong reference to the main interpreter. @@ -906,7 +906,7 @@ Example: Calling Python Without a Callback Parameter There are a few cases where callback functions don't take a callback parameter (``void *arg``), so it's impossible to acquire a reference to any specific interpreter. The solution to this problem is to acquire a reference to the main -interpreter through :c:func:`PyInterpreterRef_Main`. +interpreter through :c:func:`PyUnstable_InterpreterRef_GetMain`. But wait, won't that break with subinterpreters, per :ref:`pep-788-subinterpreters-gilstate`? Fortunately, since the callback has @@ -919,7 +919,7 @@ interpreter here. static void call_python(void) { - PyInterpreterRef ref = PyInterpreterRef_Main(); + PyInterpreterRef ref = PyUnstable_InterpreterRef_GetMain(); if (ref == 0) { fputs("Python has shut down.", stderr); return; From 47b6444d396bc10238bb2c2ce7e9b131d1b3de7c Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Sun, 3 Aug 2025 12:08:29 -0400 Subject: [PATCH 06/10] Rename PyInterpreterWeakRef_AsStrong to PyInterpreterWeakRef_Promote. --- peps/pep-0788.rst | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/peps/pep-0788.rst b/peps/pep-0788.rst index bc2e93d4656..2747edd24a9 100644 --- a/peps/pep-0788.rst +++ b/peps/pep-0788.rst @@ -402,13 +402,13 @@ note that this *is not* the same as joining the thread; the interpreter will only wait until the reference count is zero, and then proceed. After the reference count has reached zero, threads can no longer prevent the interpreter from shutting down (thus :c:func:`PyInterpreterRef_Get` and -:c:func:`PyInterpreterWeakRef_AsStrong` will fail). +:c:func:`PyInterpreterWeakRef_Promote` will fail). A weak reference to an interpreter won't prevent it from finalizing, and can be safely accessed after the interpreter no longer supports creating strong references, and even after the interpreter-state has been deleted. Deletion and duplication of the weak reference will always be allowed, but promotion -(:c:func:`PyInterpreterWeakRef_AsStrong`) will always fail after the +(:c:func:`PyInterpreterWeakRef_Promote`) will always fail after the interpreter reaches a point where strong references have been waited on. Strong Interpreter References @@ -492,7 +492,7 @@ Weak Interpreter References Acquire a weak reference to the current interpreter. This function is generally meant to be used in tandem with - :c:func:`PyInterpreterWeakRef_AsStrong`. + :c:func:`PyInterpreterWeakRef_Promote`. On success, this function returns a weak reference to the current interpreter, and returns ``0`` with an exception set on failure. @@ -510,7 +510,7 @@ Weak Interpreter References This function cannot fail, and the caller doesn't need to hold an :term:`attached thread state`. -.. c:function:: PyInterpreterRef PyInterpreterWeakRef_AsStrong(PyInterpreterWeakRef wref) +.. c:function:: PyInterpreterRef PyInterpreterWeakRef_Promote(PyInterpreterWeakRef wref) Acquire a strong reference to an interpreter through a weak reference. @@ -651,7 +651,7 @@ With this PEP, you'd implement it like this: PyObject *file, const char *text) { - PyInterpreterRef ref = PyInterpreterWeakRef_AsStrong(wref); + PyInterpreterRef ref = PyInterpreterWeakRef_Promote(wref); if (ref == 0) { /* Python interpreter has shut down */ return -1; @@ -860,7 +860,7 @@ deadlock the interpreter if it's not released. { ThreadData *data = (ThreadData *)arg; PyInterpreterWeakRef wref = data->wref; - PyInterpreterRef ref = PyInterpreterWeakRef_AsStrong(wref); + PyInterpreterRef ref = PyInterpreterWeakRef_Promote(wref); if (ref == 0) { fputs("Python has shut down!\n", stderr); return -1; From 754045e80ad2d717366f0d1990b55b8c6fe3b14f Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Sun, 3 Aug 2025 12:10:07 -0400 Subject: [PATCH 07/10] Rename PyInterpreter[Weak]Ref_Get to PyInterpreter[Weak]Ref_FromCurrent. --- peps/pep-0788.rst | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/peps/pep-0788.rst b/peps/pep-0788.rst index 2747edd24a9..24542a0fba1 100644 --- a/peps/pep-0788.rst +++ b/peps/pep-0788.rst @@ -401,7 +401,7 @@ around the same time when :class:`threading.Thread` objects are joined, but note that this *is not* the same as joining the thread; the interpreter will only wait until the reference count is zero, and then proceed. After the reference count has reached zero, threads can no longer prevent the -interpreter from shutting down (thus :c:func:`PyInterpreterRef_Get` and +interpreter from shutting down (thus :c:func:`PyInterpreterRef_FromCurrent` and :c:func:`PyInterpreterWeakRef_Promote` will fail). A weak reference to an interpreter won't prevent it from finalizing, and can @@ -423,7 +423,7 @@ Strong Interpreter References This type is guaranteed to be pointer-sized. -.. c:function:: PyInterpreterRef PyInterpreterRef_Get(void) +.. c:function:: PyInterpreterRef PyInterpreterRef_FromCurrent(void) Acquire a strong reference to the current interpreter. @@ -441,7 +441,7 @@ Strong Interpreter References This function only exists for special cases where a specific interpreter can't be saved. Prefer safely acquiring a reference through - :c:func:`PyInterpreterRef_Get` whenever possible. + :c:func:`PyInterpreterRef_FromCurrent` whenever possible. On success, this function returns a strong reference to the main interpreter, and returns ``0`` without an exception set on failure. @@ -487,7 +487,7 @@ Weak Interpreter References This type is guaranteed to be pointer-sized. -.. c:function:: int PyInterpreterWeakRef_Get(PyInterpreterWeakRef *wref) +.. c:function:: int PyInterpreterWeakRef_FromCurrent(PyInterpreterWeakRef *wref) Acquire a weak reference to the current interpreter. @@ -696,7 +696,7 @@ held. Any future finalizer that wanted to acquire the lock would be deadlocked! my_critical_operation(PyObject *self, PyObject *unused) { assert(PyThreadState_GetUnchecked() != NULL); - PyInterpreterRef ref = PyInterpreterRef_Get(&ref); + PyInterpreterRef ref = PyInterpreterRef_FromCurrent(&ref); if (ref == 0) { /* Python interpreter has shut down */ return NULL; @@ -779,7 +779,7 @@ This is the same code, rewritten to use the new functions: PyThread_handle_t handle; PyThead_indent_t indent; - PyInterpreterRef ref = PyInterpreterRef_Get(); + PyInterpreterRef ref = PyInterpreterRef_FromCurrent(); if (ref == 0) { return NULL; } @@ -830,7 +830,7 @@ hang the current thread forever). PyThread_handle_t handle; PyThead_indent_t indent; - PyInterpreterRef ref = PyInterpreterRef_Get(); + PyInterpreterRef ref = PyInterpreterRef_FromCurrent(); if (ref == 0) { return NULL; } @@ -889,7 +889,7 @@ deadlock the interpreter if it's not released. PyErr_NoMemory(); return NULL; } - PyInterpreterWeakRef wref = PyInterpreterWeakRef_Get(); + PyInterpreterWeakRef wref = PyInterpreterWeakRef_FromCurrent(); if (wref == 0) { PyMem_RawFree(tdata); return NULL; @@ -1013,7 +1013,7 @@ of requiring less magic: the non-Python thread gets a chance to attach. The problem with using an interpreter ID is that the reference count has to be "invisible"; it must be tracked elsewhere in the interpreter, likely being *more* - complex than :c:func:`PyInterpreterRef_Get`. There's also a lack + complex than :c:func:`PyInterpreterRef_FromCurrent`. There's also a lack of intuition that a standalone integer could have such a thing as a reference count. From 4ed9e36655476ea2457c5a21cc3df252e4237a86 Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Mon, 4 Aug 2025 08:47:51 -0400 Subject: [PATCH 08/10] Fix incorrect usage of PyInterpreterRef_FromCurrent(). --- peps/pep-0788.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0788.rst b/peps/pep-0788.rst index 24542a0fba1..e90da409a74 100644 --- a/peps/pep-0788.rst +++ b/peps/pep-0788.rst @@ -696,7 +696,7 @@ held. Any future finalizer that wanted to acquire the lock would be deadlocked! my_critical_operation(PyObject *self, PyObject *unused) { assert(PyThreadState_GetUnchecked() != NULL); - PyInterpreterRef ref = PyInterpreterRef_FromCurrent(&ref); + PyInterpreterRef ref = PyInterpreterRef_FromCurrent(); if (ref == 0) { /* Python interpreter has shut down */ return NULL; From 100d850d49b93386bdb5afa8eee2566aea605f2b Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Thu, 7 Aug 2025 08:15:17 -0400 Subject: [PATCH 09/10] Rename PyUnstable_InterpreterRef_GetMain to PyUnstable_GetDefaultInterpreterRef. Per discussion on DPO. The idea is that this is a general runtime/lifecycle API, not an operation executing on an interpreter reference. --- peps/pep-0788.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0788.rst b/peps/pep-0788.rst index e90da409a74..c756457b2b3 100644 --- a/peps/pep-0788.rst +++ b/peps/pep-0788.rst @@ -435,7 +435,7 @@ Strong Interpreter References The caller must hold an :term:`attached thread state`. -.. c:function:: PyInterpreterRef PyUnstable_InterpreterRef_GetMain(PyInterpreterRef *ref) +.. c:function:: PyInterpreterRef PyUnstable_GetDefaultInterpreterRef(PyInterpreterRef *ref) Acquire a strong reference to the main interpreter. From 4455670c902049e38d3f9201517eb9c6f6cb16eb Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Thu, 7 Aug 2025 08:24:17 -0400 Subject: [PATCH 10/10] Add a section describing PyUnstable_GetDefaultInterpreterRef's motivation. --- peps/pep-0788.rst | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/peps/pep-0788.rst b/peps/pep-0788.rst index c756457b2b3..cd2b5a8a70a 100644 --- a/peps/pep-0788.rst +++ b/peps/pep-0788.rst @@ -387,6 +387,21 @@ The exact details of this deprecation aren't too clear. It's likely that the usual five-year deprecation (as specificed by :pep:`387`) will be too short, so for now, these functions will have no specific removal date. +Compatibility Shim for ``PyGILState_Ensure`` +-------------------------------------------- + +This proposal comes with :c:func:`PyUnstable_GetDefaultInterpreterRef` as a +compatibility hack for some users of :c:func:`PyGILState_Ensure`. It is a +thread-safe way to acquire a strong reference to the main (or "default") +interpreter. + +The main drawback to porting new code to :c:func:`PyThreadState_Ensure` is that +it isn't a drop-in replacement for :c:func:`!PyGILState_Ensure`, as it needs +an interpreter reference argument. In some large applications, refactoring to +use a :c:type:`PyInterpreterRef` everywhere might be tricky; so, this function +acts as a silver bullet for users who explicitly want to disallow support for +subinterpreters. + Specification ============= @@ -906,7 +921,7 @@ Example: Calling Python Without a Callback Parameter There are a few cases where callback functions don't take a callback parameter (``void *arg``), so it's impossible to acquire a reference to any specific interpreter. The solution to this problem is to acquire a reference to the main -interpreter through :c:func:`PyUnstable_InterpreterRef_GetMain`. +interpreter through :c:func:`PyUnstable_GetDefaultInterpreterRef`. But wait, won't that break with subinterpreters, per :ref:`pep-788-subinterpreters-gilstate`? Fortunately, since the callback has @@ -919,7 +934,7 @@ interpreter here. static void call_python(void) { - PyInterpreterRef ref = PyUnstable_InterpreterRef_GetMain(); + PyInterpreterRef ref = PyUnstable_GetDefaultInterpreterRef(); if (ref == 0) { fputs("Python has shut down.", stderr); return;