PEP 798: Adjust Generator Expression Semantics

[adqm]

The main change here is adjusting the proposed semantics of unpacking in generator expressions and adding a section briefly outlining the pros and cons of the alternatives, but I also:

• added a link to a Reddit thread I stumbled upon, which was quite positive about the PEP (not sure if this is actually worth including or not), and
• made several small wording/grammar/spelling changes.

---

📚 Documentation preview 📚: https://pep-previews--4547.org.readthedocs.build/

[JelleZijlstra]

It might be worth showing some examples of what the actual difference in semantics is between yield from and the current approach with for x in ...: yield x. I think it matters if you .send something into the genexp?

[adqm]

Yep, my understanding is that using yield from would mean that, if the thing we're unpacking is itself a generator, then anything we send to our top-level generator would actually go to the one we're unpacking. With an explicit loop, that doesn't happen. My gut feeling is that wanting to delegate to the thing being unpacked is probably a pretty niche situation (I'm not sure I know of a good practical use case for it), though I could certainly be wrong there.

Either way, I agree it's worth making that difference more explicit. I'll add a paragraph here, but it might not be until tomorrow. Let's hold off on merging until then.

[adqm]

Just a heads-up that I probably won't have my updates in tonight, since I'm still going back and forth in my head about what the right decision is here...

In case it's of interest, here's what I'm currently thinking:

• Even though we're likely talking about a niche use case, one thing I'd like to make sure is that we're setting things up for reasonable future upgrades to be backward-compatible with the details being proposed here.
• Right now, the version with yield from and the version with the explicit loop both potentially produce output even when we .send to them, but different outputs; so a future change to add this delegation functionality would change the behavior of any code using .send on one of these things. An error would be preferable from the perspective of future-proofing.

So I think I might actually be slowly convincing myself that the option you voted for in the poll might be the right choice 😄. That way we could use yield from in the sync case, add an explicit error message when someone tries to use unpacking in an async genexp, and let someone else decide what to do about the async situation later on (ideally, once we have proper delegation for async generators).

But, tl;dr: I'm going to need to think about this a bit more (and probably follow up in the Discourse thread, too).

[JelleZijlstra]

There's no hurry! The only sort-of-deadline is the Python 3.15 feature freeze next May, and we have plenty of time before that.

Here's some thoughts from me, but I don't have firm views either.
As you implied, the thing about picking one set of semantics is that backwards compatibility will mean that we'll have a hard time changing the behavior later. That's what makes it appealing to make cases where we're not sure of the right semantics into an error, since we can change our mind later about code that currently throws an error.

yield from vs. for ... in ...: yield matters here only if you call .send() or .throw() on the generator object created by the genexp. As far as I can tell, there's currently (pre-PEP 798) no good reason to call those methods on a generator created for a genexp: the value sent by .send() is ignored, and the exception thrown by .throw() is just thrown back into the calling scope.

With PEP 798 unpacking and yield from semantics, that changes and you could do theoretically useful things with .send() on genexps, like this:

>>> def make_gen(i):
...         val = yield "input:"
...         yield val * i
...         
>>> g = (*make_gen(i) for i in range(5, 10))
>>> 
>>> g.send(None)
'input:'
>>> g.send(1)
5
>>> g.send(None)
'input:'
>>> g.send(3)
18
>>> g.send(None)
'input:'
>>> g.send(4)
28

Well, not that practically useful, but maybe you'd be able to come up with a way to use this that is actually good for something useful.

This provides an argument that it's reasonable to go with for ... in: yield semantics now (either just for async genexps or for both), and switch to yield from later: the only behavior we'd change is around things that nobody has a good reason to do right now anyway. Still, people may do things even if they don't have a good reason to (or there might be a good reason that I'm not clever enough to think of), so compatibility might still be an issue.

[adqm]

Thanks for the input! I agree on all fronts.

And actually, the need to repeatedly send Nones at the right times to kickstart the internal generators (which I somehow hadn't thought of) seems to make the delegation even less useful in this context than I had thought it might be (and I already thought it wasn't likely to be terribly useful). So maybe it's the case the case that anyone wanting that kind of delegation should (and/or would want to) write it out long-hand instead of using a genexp anyway...

[JelleZijlstra]

the need to repeatedly send Nones at the right times to kickstart the internal generators

I think you don't actually need to send Nones for every one of them, e.g. this worked:

>>> g.send(4)
28
>>> g.send(42)
'input:'
>>> g.send(43)
344

But it's definitely a complicated interface to work with.

[adqm]

I think you don't actually need to send Nones for every one of them

Ah, sorry, yes, but every so often one thing that you send will effectively be ignored, right? That is, the output above would have been the same with any other object in place of 42 there.

[JelleZijlstra]

True, though I could have gotten it out if I had made the inner generator do something with the result of the second yield.

[adqm]

Oh, interesting. Yeah, you're right. Sorry; I was misunderstanding what was actually happening there!