Opinions wanted for PEP 793 – PyModExport: A new entry point for C extension modules

[encukou]

Hello,
I'd like to hear opinions from nanobind contributors (and other binding maintainers) on PEP 793

What is this? I propose a new way to initialize C extension modules: rather than the current PyInit_* function, you would define PyModExport_*. Rather than returning a module object or PyModuleDef, this function would return a C array of slots (same info as in a PyModuleDef, but laid out differently).
This change is needed to support stable ABI for free-threaded CPython builds without hacks. See the PEP for the rationale, and all the details.

For nanobind, it looks like this would mean a change to the PyInit_ definitions -- annoying but localized -- in return for single-source compatibility with python 3.15+ and (as a separate extension/wheel) non-free-threaded stable ABI. (I'd be willing to collaborate on making that a single build if someone's interested, and the PEP is accepted.)

What do you think?
Please reply here, or on the PEP's discussion thread: https://discuss.python.org/t/93444

Thanks for your time,
Petr

[hpkfft]

Speaking only for myself, and as one who is not an expert in C extension modules, this PEP looks like a good thing.
Doing work one time (especially if it is localized) is well worth the effort in order to have fewer builds thereafter.
(I assume one build means one wheel, i.e., my extension module, which uses nanobind, has to distribute only one wheel.)
I would be willing to collaborate on making that happen.

To be sure I'm understanding correctly, let me use my extension as an example. Also, let me ignore AArch64 and only list wheels for Linux x86_64. (Note that nanobind does not support the stable ABI until Python 3.12.) I currently have:

• hpk-0.5.1-cp311-cp311-linux_x86_64.whl for Python 3.11
• hpk-0.5.1-cp312-abi3-linux_x86_64.whl for Python 3.12, 3.13, 3.14, all non-free-threaded
• hpk-0.5.1-cp314-cp314t-linux_x86_64.whl for Python 3.14 free-threaded

If your PEP is accepted and nanobind does as recommended, will it be possible to only add one more wheel:

• hpk-0.X.Y-cp315-abiZ-linux_x86_64.whl for Python 3.15, 3.16, 3.17, ..., non-free-threaded or free-threaded

[encukou]

Yes, that's pretty much what I'm proposing in PEP 803, which looks at a bigger picture. (PEP 793 only proposes the C API change that's necessary for schemes like this).

[nicholasjng]

I'm also interested in collaborating on the unified wheel builds if this gets accepted.

Maybe I overlooked it in the proposal, but does the token API help with accessing constructed modules in a thread-safe way for the case of multiple interpreters? I've seen a few PRs in different nanobind-using repos lately essentially building a thread-safe import cache for their modules, and I'm wondering if PyType_GetModuleByToken et al. could help with that.

[encukou]

No: there's no 1-to-1 mapping of extension DLLs to extension module objects, and this API does not change that.
Python uses a cache in sys.modules, which users can clear (or poison) from Python code so it's not the tool to use if you need memory safety guarantees.
There's also a cache for modules that use single-phase initialization, but the semantics of that are very ... surprising.

Tokens are for the case where you already have a module (or type) object and want to check if it was defined by “your” module. PyType_GetModuleByToken gets you the superclass that was defined in “your” module.

So, PEP 793 won't help here. Do you want to point to a few of these “caching” repos? Maybe we can design something for them.

[nicholasjng]

Thanks, that helps.

There's this PR in MLIR that caches an important nanobind::module_ import in a way that's compatible with free-threading.

Another example is duckdb (currently not using nanobind, but they are experimenting with free-threading IIUC), who use an import cache on the C++ side, which would also have to be made thread-safe.

[encukou]

These techniques look like they use a process-global cache, so they'll probably break with multiple interpreters.

Like @wjakob, I'm not sure why this is necessary, given Python's sys.modules cache. (Users can clear that, but it should not be a performance concern...)

[wjakob]

Hi @encukou, this sounds very useful. If you provide such an API, nanobind will use it.