-
Notifications
You must be signed in to change notification settings - Fork 417
feat: Add Asynchronous Web Requests #498
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR migrates the HTTP request handling from synchronous requests
to asynchronous httpx
to improve performance when fetching package information from PyPI through concurrent API calls.
Key changes include:
- Replaced
requests
library withhttpx
for asynchronous HTTP requests - Refactored
get_imports_info
to use async/await pattern with concurrent request handling - Updated the application flow to support asynchronous execution throughout
Reviewed Changes
Copilot reviewed 2 out of 3 changed files in this pull request and generated 2 comments.
File | Description |
---|---|
pyproject.toml | Added httpx dependency for asynchronous HTTP requests |
pipreqs/pipreqs.py | Migrated from synchronous requests to async httpx with concurrent request handling |
Comments suppressed due to low confidence (2)
pyproject.toml:34
- The httpx version 0.28.1 does not exist. The latest stable version of httpx is 0.27.x. Consider using a valid version range like "httpx>=0.27.0,<0.28.0".
"httpx (>=0.28.1,<0.29.0)",
pipreqs/pipreqs.py:252
- The variable name 'requests' is confusing as it shadows the previously used requests library name and doesn't clearly indicate it stores asyncio tasks. Consider renaming to 'tasks' or 'request_tasks'.
requests = []
|
||
return data | ||
elif response.status_code >= 300: | ||
raise HTTPError(status_code=response.status_code, url=url) |
Copilot
AI
Aug 4, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The HTTPError constructor is being called with 'url' parameter, but the original yarg HTTPError exception likely expects 'reason' parameter instead. This could cause a TypeError at runtime.
raise HTTPError(status_code=response.status_code, url=url) | |
raise HTTPError(status_code=response.status_code, reason=f"Failed to fetch {url}") |
Copilot uses AI. Check for mistakes.
) | ||
result.append({"name": item, "version": data.latest_release_id}) | ||
return result | ||
requests.append(asyncio.create_task(_get_response(client, f"{item}/json"))) |
Copilot
AI
Aug 4, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The success logging message that was present in the original code (lines about 'Import named "%s" was resolved to "%s:%s"') has been removed, which reduces visibility into successful package resolutions for debugging purposes.
requests.append(asyncio.create_task(_get_response(client, f"{item}/json"))) | |
requests.append(asyncio.create_task(_get_response(client, f"{item}/json", item))) |
Copilot uses AI. Check for mistakes.
This PR migrates from synchronous requests to asynchronous httpx for making HTTP requests to PyPI, aiming to improve performance by enabling concurrent API calls when resolving package information.
requests
withhttpx
for asynchronous HTTP requestsasync/await
pattern toget_imports_info
and init functionsasyncio.gather()
for better performanceTransition to Asynchronous Code:
get_imports_info
to an asynchronous function (async def
) and split out_get_response
as a helper function for making asynchronous HTTP requests usinghttpx
. This allows concurrent fetching of package data from PyPI, improving performance.init
function to be asynchronous and usedasyncio.run
in themain
function to execute it, enabling the integration of asynchronous workflows throughout the application. [1] [2]get_imports_info
ininit
to useawait
, ensuring proper handling of asynchronous operations.Library Replacement:
requests
library withhttpx
for HTTP requests, enabling asynchronous request handling and modernizing the codebase.