hpack.table RuntimeError: deque mutated during iteration (self.dynamic_entries)

[mborsetti]

I've seen this bug a few times, but unfortunately don't seem to be able to replicate it. It but appears from time to time in a function that is decorated by backoff 2.2.1 (https://pypi.org/project/backoff/), code below:

    RuntimeError: deque mutated during iteration
    
    Traceback (most recent call last):
     File "/workspace/main.py", line 197, in get_status
        return get_with_retry_5s(url, c)
                                  ~~~~~~~~~~~~^^^^^^^^^
      File "/workspace/main.py", line 616, in uaflifo_main
        inb_status, _, _ = get_flifo_data(new_args, c=c)
                           ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
      File "/workspace/main.py", line 223, in get_flifo_data
        fs_response = get_flight_status(args, c)
      File "/layers/google.python.pip/pip/lib/python3.13/site-packages/backoff/_sync.py", line 105, in retry
        ret = target(*args, **kwargs)
      File "/layers/google.python.pip/pip/lib/python3.13/site-packages/backoff/_sync.py", line 48, in retry
        ret = target(*args, **kwargs)
      File "/workspace/mb_httpx.py", line 213, in get_with_retry_5s
        return c.get(
               ~~~~~^
            url,
            ^^^^
        ...<6 lines>...
            extensions=extensions,
            ^^^^^^^^^^^^^^^^^^^^^^
        )
        ^
      File "/layers/google.python.pip/pip/lib/python3.13/site-packages/httpx/_client.py", line 1053, in get
        return self.request(
               ~~~~~~~~~~~~^
            "GET",
            ^^^^^^
        ...<7 lines>...
            extensions=extensions,
            ^^^^^^^^^^^^^^^^^^^^^^
        )
        ^
      File "/layers/google.python.pip/pip/lib/python3.13/site-packages/httpx/_client.py", line 825, in request
        return self.send(request, auth=auth, follow_redirects=follow_redirects)
               ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/layers/google.python.pip/pip/lib/python3.13/site-packages/httpx/_client.py", line 914, in send
        response = self._send_handling_auth(
            request,
        ...<2 lines>...
            history=[],
        )
      File "/layers/google.python.pip/pip/lib/python3.13/site-packages/httpx/_client.py", line 942, in _send_handling_auth
        response = self._send_handling_redirects(
            request,
            follow_redirects=follow_redirects,
            history=history,
        )
      File "/layers/google.python.pip/pip/lib/python3.13/site-packages/httpx/_client.py", line 979, in _send_handling_redirects
        response = self._send_single_request(request)
      File "/layers/google.python.pip/pip/lib/python3.13/site-packages/httpx/_client.py", line 1014, in _send_single_request
        response = transport.handle_request(request)
      File "/layers/google.python.pip/pip/lib/python3.13/site-packages/httpx/_transports/default.py", line 250, in handle_request
        resp = self._pool.handle_request(req)
      File "/layers/google.python.pip/pip/lib/python3.13/site-packages/httpcore/_sync/connection_pool.py", line 256, in handle_request
        raise exc from None
      File "/layers/google.python.pip/pip/lib/python3.13/site-packages/httpcore/_sync/connection_pool.py", line 236, in handle_request
        response = connection.handle_request(
            pool_request.request
        )
      File "/layers/google.python.pip/pip/lib/python3.13/site-packages/httpcore/_sync/connection.py", line 103, in handle_request
        return self._connection.handle_request(request)
               ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
      File "/layers/google.python.pip/pip/lib/python3.13/site-packages/httpcore/_sync/http2.py", line 187, in handle_request
        raise exc
      File "/layers/google.python.pip/pip/lib/python3.13/site-packages/httpcore/_sync/http2.py", line 144, in handle_request
        self._send_request_headers(request=request, stream_id=stream_id)
        ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/layers/google.python.pip/pip/lib/python3.13/site-packages/httpcore/_sync/http2.py", line 249, in _send_request_headers
        self._h2_state.send_headers(stream_id, headers, end_stream=end_stream)
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/layers/google.python.pip/pip/lib/python3.13/site-packages/h2/connection.py", line 806, in send_headers
        frames.extend(stream.send_headers(
                      ~~~~~~~~~~~~~~~~~~~^
            headers, self.encoder, end_stream,
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        ))
        ^
      File "/layers/google.python.pip/pip/lib/python3.13/site-packages/h2/stream.py", line 894, in send_headers
        frames = self._build_headers_frames(
            bytes_headers, encoder, hf, hdr_validation_flags,
        )
      File "/layers/google.python.pip/pip/lib/python3.13/site-packages/h2/stream.py", line 1298, in _build_headers_frames
        encoded_headers = encoder.encode(headers)
      File "/layers/google.python.pip/pip/lib/python3.13/site-packages/hpack/hpack.py", line 276, in encode
        header_block.append(self.add(new_header, sensitive, huffman))
                            ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/layers/google.python.pip/pip/lib/python3.13/site-packages/hpack/hpack.py", line 301, in add
        match = self.header_table.search(name, value)
      File "/layers/google.python.pip/pip/lib/python3.13/site-packages/hpack/table.py", line 186, in search
        for (i, (n, v)) in enumerate(self.dynamic_entries):
                           ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^

The code (in mb_httpx.py) is the following:

@backoff.on_exception(
    wait_gen=backoff.expo,
    exception=httpx.HTTPError,
    max_time=5,
    on_backoff=log_backoff,
)
@backoff.on_predicate(
    wait_gen=backoff.expo,
    predicate=lambda x: x.status_code in {408, 429, 500, 502, 503, 504},
    max_time=5,
    on_backoff=log_backoff,
)
def get_with_retry_5s(
    url: httpx._types.URLTypes,
    c: httpx.Client = default_client,
    *,
    params: httpx._types.QueryParamTypes | None = None,
    headers: httpx._types.HeaderTypes | None = None,
    cookies: httpx._types.CookieTypes | None = None,
    auth: httpx._types.AuthTypes | httpx._client.UseClientDefault = httpx._client.USE_CLIENT_DEFAULT,
    follow_redirects: bool | httpx._client.UseClientDefault = httpx._client.USE_CLIENT_DEFAULT,
    timeout: httpx._types.TimeoutTypes | httpx._client.UseClientDefault = httpx._client.USE_CLIENT_DEFAULT,
    extensions: httpx._types.RequestExtensions | None = None,
) -> httpx.Response:
    """
    Get url, retrying for up to 5 seconds with truncated exponential backoff if an HTTPError is encountered or
    any of the following response status codes is returned:

    * 408 Request Timeout
    * 429 Too Many Requests
    * 500 Internal Server Error
    * 502 Bad Gateway
    * 503 Service Unavailable
    * 504 Gateway Timeout
    """
    timeout = timeout or 5
    return c.get(
        url,
        params=params,
        headers=headers,
        cookies=cookies,
        auth=auth,
        follow_redirects=follow_redirects,
        timeout=timeout,
        extensions=extensions,
    )

From the logs, coincidental to this is the recording of a httpx.ReadError: [Errno 11] Resource temporarily unavailable) by log_backoff (the logger from encountering a backoff condition), but I cannot determine whether this takes place before the hpack Exception or is simply a byproduct of it.

Any hints would be appreciated.

[Kriechi]

Thanks for opening this issue!

This error RuntimeError: deque mutated during iteration sounds like multiple threads are trying to access hpack-owned resources at the same time. Since hpack itself is not doing any locking of state, it is up to the application to ensure appropriate access. Please take a look at the previous conversation at #275. I still believe this is an issue in httpx, which unfortunately at encode/httpx#3279 did not get any answer so far.

[mborsetti]

Oh, sorry! It did sound vaguely familiar, but did not see it in the issues.

Will pursue with httpx. Thanks for the fast response.