Proposal: Add optimization to ignore "duplicate" navigations

[rakina]

What problem are you trying to solve?

We observe users sometimes navigate to the same URL in quick succession, likely by accident. Because new navigations take precedent over an older one, this means it will waste the earlier navigation that's already in progress, potentially wasting a response that is already in flight for the navigation and causing the user to wait longer (from the time the first navigation kicks off).

What solutions exist today?

Maybe sites can check for e.g. double clicks with a click handler, but it won't help when other sites are linking to that site, and also other methods (e.g. clicking reload button)

How would you solve it?

Ignore the duplicate navigation and let the first navigation continue. Add a carveout in navigate that checks if a navigation can be considered a duplicate of an ongoing navigation in the navigable, and if so, just return early. Criteria for "duplicate" includes (ideas welcome):

• The navigations start time difference is under a certain threshold (e.g. 3 seconds)
• The navigations are both GET, no form data
• The URL, initiator origin, referrer policy, headers (including cookies), userInvolvement, historyHandling of the navigations are the same

The ignoring will be mostly silent, but can be observable in e.g. the NavigationTiming API showing the navigation start time of the first navigation instead of the second navigation. If navigate() API is used, the ignored duplicate navigation will cause the Promise to reject. We probably want to also trigger WebDriver BiDi navigation failed so that automations won't get confused that a navigation just gets silently dropped.

Any suggestions of other things to change / be careful about is very welcome.

Anything else?

Chromium has been experimenting with this optimization with a partner and saw significant improvements in latency and also capacity (since we won't need to serve the same resource twice for a duplicated navigation). We want to standardize this behavior.

cc @fergald @llannasatoll @noamr @domfarolino

[annevk]

I can understand wanting to deduplicate end user actions, but it seems a bit less expected to deduplicate API calls and such as well. Maybe it's okay.

I also don't think we have the cookies (or many headers) available as part of the navigation algorithm currently. Fetch ends up adding those.

[achristensen07]

This proposal assumes a static network topology. At risk of opening an even bigger can of worms, wouldn't it be better to race the two requests in case the second request finds a faster path through the network? Imagine a case where the user navigates on poor cell data, then 1 second later successfully reconnects to good wifi, then 1 second later reclicks the same navigation.

[rakina]

FYI we have a draft spec PR now, let us know if we're missing anything. I think we need to add a way to [inform the navigation API about the aborted navigation][(https://html.spec.whatwg.org/multipage/nav-history-apis.html#inform-the-navigation-api-about-aborting-navigation) without it being owned by the navigable yet, but I'll ask about it in the PR later.

I can understand wanting to deduplicate end user actions, but it seems a bit less expected to deduplicate API calls and such as well. Maybe it's okay.

It is a little unexpected, but I'm not sure if there's an actual use case for triggering two practically identical navigation right after another outside of the user requesting it through e.g. double click on some button or something. If there are, I guess we can make carveout to specific cases (maybe just the navigate() API)?

I also don't think we have the cookies (or many headers) available as part of the navigation algorithm currently. Fetch ends up adding those.

Oh interesting. Do you think we can just refer to the request's header list when comparing the navigations? At least the request's header list seems to be referred in step 6.3 of "create navigation params by fetching" here

This proposal assumes a static network topology. At risk of opening an even bigger can of worms, wouldn't it be better to race the two requests in case the second request finds a faster path through the network? Imagine a case where the user navigates on poor cell data, then 1 second later successfully reconnects to good wifi, then 1 second later reclicks the same navigation.

Yeah it's definitely possible that a second navigation might actually get a response faster. However we've been experimenting and it looks like it's more common for this to help with latency overall. If we get really unlucky, the threshold is low enough (it's 1-3s in our impl) that another retry would actually trigger a new navigation. I guess we can make an exception to when the network actually changed, although I'm not sure how that looks like in spec. Any suggestions?

[annevk]

@rakina you can refer to it for sure, but at that point the header list is essentially empty, so I don't think it's useful, unless you mean to compare Content-Type specifically.

[rakina]

Oh that's a good point. Hmm, it looks like the cookie header and other headers are added quite deep into the fetch algorithm, where it's probably hard to back out without disrupting things. So maybe instead we should compare specific properties that are important? Since we are restricting this to GET maybe we only need to check the cookies via retrieve cookies (and then compare it with the cookies in the header of the existing navigation). Will that be enough?

[annevk]

I think if you are really intent on comparing cookies you have to do the comparison in Fetch somehow, as it's conceptually (and practically in some implementations) a completely different layer. That seems rather involved however. But it also seems that comparing cookies would end up exposing some details, such as whether an HTTPOnly cookie was set since the initial navigation. Not sure that's information we'd want to give to an attacker.

[rakina]

Hmm, so maybe it's better to just not compare the cookies at all, with some risk of ignoring actually-different navigations? Is that acceptable, if the time limit is low enough? (Comparing only non-http cookies would be a bit incomplete for the purposes, and not sure if that's possible).

[noamr]

Hmm, so maybe it's better to just not compare the cookies at all, with some risk of ignoring actually-different navigations? Is that acceptable, if the time limit is low enough? (Comparing only non-http cookies would be a bit incomplete for the purposes, and not sure if that's possible).

If this becomes an issue we could change this behavior based on whether the document itself changes anything that would affect the navigation after the first one (e.g. sets cookies).

[rakina]

If this becomes an issue we could change this behavior based on whether the document itself changes anything that would affect the navigation after the first one (e.g. sets cookies).

Oh that's a good idea. Can we just make a wrapper for setting the cookie attribute and make that set a "last cookie update timestamp" or something and then only ignore if the last modification time is less than the original navigation's start time? Are there other API calls that can trigger cookie modifications etc to cover outside of that?

[noamr]

If this becomes an issue we could change this behavior based on whether the document itself changes anything that would affect the navigation after the first one (e.g. sets cookies).

Oh that's a good idea. Can we just make a wrapper for setting the cookie attribute and make that set a "last cookie update timestamp" or something and then only ignore if the last modification time is less than the original navigation's start time? Are there other API calls that can trigger cookie modifications etc to cover outside of that?

I think it can be a boolean "navigation conditions potentially changed" or some such, associated with the navigation when it starts and initialized at false, and becomes true if any API that might affect those conditions is called. If it's true, a second navigation can abort the first one; We can be pretty generous about setting it to true.

[annevk]

There's also the Cookie Store API, but any fetch that happens between the navigations can in theory impact the cookies.