Skip to content

[Storage] Decompression for Binary Response in Download APIs #43587

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

[Storage] Decompression for Binary Response in Download APIs #43587

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions sdk/storage/azure-storage-blob/azure/storage/blob/_blob_client.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -732,6 +732,8 @@ def download_blob(
function(current: int, total: int) where current is the number of bytes transferred
so far, and total is the total size of the download.
:paramtype progress_hook: Callable[[int, int], None]
:keyword bool decompress: If True, any compressed content, identified by the Content-Type header, will be
Copy link

Copilot AI Oct 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The documentation states that compression is identified by the Content-Type header, but based on the test cases which use 'content_encoding', it appears compression is actually identified by the Content-Encoding header, not Content-Type. The documentation should be corrected to reference Content-Encoding.

Suggested change
:keyword bool decompress: If True, any compressed content, identified by the Content-Type header, will be
:keyword bool decompress: If True, any compressed content, identified by the Content-Encoding header, will be

Copilot uses AI. Check for mistakes.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not quite right. The Content-Encoding header specifies gzip if we are trying to download the content in compressed format. Technically the header is x-ms-blob-content-md5 that contains the content...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot is right here

decompressed automatically before being returned. Default value is True.
:keyword int timeout:
Sets the server-side timeout for the operation in seconds. For more details see
https://learn.microsoft.com/rest/api/storageservices/setting-timeouts-for-blob-service-operations.
Expand Down
3 changes: 3 additions & 0 deletions sdk/storage/azure-storage-blob/azure/storage/blob/_blob_client.pyi
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -206,6 +206,7 @@ class BlobClient(StorageAccountHostsMixin, StorageEncryptionMixin):
max_concurrency: int = 1,
encoding: str,
progress_hook: Optional[Callable[[int, int], None]] = None,
decompress: Optional[bool] = None,
timeout: Optional[int] = None,
**kwargs: Any
) -> StorageStreamDownloader[str]: ...
Expand All @@ -227,6 +228,7 @@ class BlobClient(StorageAccountHostsMixin, StorageEncryptionMixin):
max_concurrency: int = 1,
encoding: None = None,
progress_hook: Optional[Callable[[int, int], None]] = None,
decompress: Optional[bool] = None,
timeout: Optional[int] = None,
**kwargs: Any
) -> StorageStreamDownloader[bytes]: ...
Expand All @@ -248,6 +250,7 @@ class BlobClient(StorageAccountHostsMixin, StorageEncryptionMixin):
max_concurrency: int = 1,
encoding: Optional[str] = None,
progress_hook: Optional[Callable[[int, int], None]] = None,
decompress: Optional[bool] = None,
timeout: Optional[int] = None,
**kwargs: Any
) -> Union[StorageStreamDownloader[str], StorageStreamDownloader[bytes]]: ...
Expand Down
7 changes: 7 additions & 0 deletions sdk/storage/azure-storage-blob/azure/storage/blob/_download.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -422,6 +422,10 @@ def __len__(self):
def _get_encryption_data_request(self) -> None:
# Save current request cls
download_cls = self._request_options.pop('cls', None)

# Temporarily removing this for the get properties request
decompress = self._request_options.pop('decompress', None)

# Adjust cls for get_properties
self._request_options['cls'] = deserialize_blob_properties

Expand All @@ -434,6 +438,9 @@ def _get_encryption_data_request(self) -> None:
# Restore cls for download
self._request_options['cls'] = download_cls

if decompress is not None:
self._request_options['decompress'] = decompress

@property
def _download_complete(self):
if is_encryption_v2(self._encryption_data):
Expand Down
2 changes: 2 additions & 0 deletions sdk/storage/azure-storage-blob/azure/storage/blob/aio/_blob_client_async.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -745,6 +745,8 @@ async def download_blob(
function(current: int, total: int) where current is the number of bytes transferred
so far, and total is the total size of the download.
:paramtype progress_hook: Callable[[int, int], Awaitable[None]]
:keyword bool decompress: If True, any compressed content, identified by the Content-Type header, will be
Copy link

Copilot AI Oct 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The documentation states that compression is identified by the Content-Type header, but based on the test cases which use 'content_encoding', it appears compression is actually identified by the Content-Encoding header, not Content-Type. The documentation should be corrected to reference Content-Encoding.

Suggested change
:keyword bool decompress: If True, any compressed content, identified by the Content-Type header, will be
:keyword bool decompress: If True, any compressed content, identified by the Content-Encoding header, will be

Copilot uses AI. Check for mistakes.
decompressed automatically before being returned. Default value is True.
:keyword int timeout:
Sets the server-side timeout for the operation in seconds. For more details see
https://learn.microsoft.com/rest/api/storageservices/setting-timeouts-for-blob-service-operations.
Expand Down
3 changes: 3 additions & 0 deletions sdk/storage/azure-storage-blob/azure/storage/blob/aio/_blob_client_async.pyi
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -208,6 +208,7 @@ class BlobClient( # type: ignore[misc]
max_concurrency: int = 1,
encoding: str,
progress_hook: Optional[Callable[[int, int], Awaitable[None]]] = None,
decompress: Optional[bool] = None,
timeout: Optional[int] = None,
**kwargs: Any
) -> StorageStreamDownloader[str]: ...
Expand All @@ -229,6 +230,7 @@ class BlobClient( # type: ignore[misc]
max_concurrency: int = 1,
encoding: None = None,
progress_hook: Optional[Callable[[int, int], Awaitable[None]]] = None,
decompress: Optional[bool] = None,
timeout: Optional[int] = None,
**kwargs: Any
) -> StorageStreamDownloader[bytes]: ...
Expand All @@ -250,6 +252,7 @@ class BlobClient( # type: ignore[misc]
max_concurrency: int = 1,
encoding: Optional[str] = None,
progress_hook: Optional[Callable[[int, int], Awaitable[None]]] = None,
decompress: Optional[bool] = None,
timeout: Optional[int] = None,
**kwargs: Any
) -> Union[StorageStreamDownloader[str], StorageStreamDownloader[bytes]]: ...
Expand Down
7 changes: 7 additions & 0 deletions sdk/storage/azure-storage-blob/azure/storage/blob/aio/_download_async.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -292,6 +292,10 @@ def __len__(self):
async def _get_encryption_data_request(self) -> None:
# Save current request cls
download_cls = self._request_options.pop('cls', None)

# Temporarily removing this for the get properties request
decompress = self._request_options.pop('decompress', None)

# Adjust cls for get_properties
self._request_options['cls'] = deserialize_blob_properties

Expand All @@ -304,6 +308,9 @@ async def _get_encryption_data_request(self) -> None:
# Restore cls for download
self._request_options['cls'] = download_cls

if decompress is not None:
self._request_options['decompress'] = decompress

async def _setup(self) -> None:
if self._encryption_options.get("key") is not None or self._encryption_options.get("resolver") is not None:
await self._get_encryption_data_request()
Expand Down
23 changes: 23 additions & 0 deletions sdk/storage/azure-storage-blob/tests/test_common_blob.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3595,4 +3595,27 @@ def test_upload_blob_partial_stream_chunked(self, **kwargs):
result = blob.download_blob().readall()
assert result == data[:length]

@pytest.mark.live_test_only
@BlobPreparer()
def test_download_blob_decompress(self, **kwargs):
storage_account_name = kwargs.pop("storage_account_name")
storage_account_key = kwargs.pop("storage_account_key")

# Arrange
self._setup(storage_account_name, storage_account_key)
blob_name = self._get_blob_reference()
blob = self.bsc.get_blob_client(self.container_name, blob_name)
compressed_data = b'\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\xff\xcaH\xcd\xc9\xc9WH+\xca\xcfUH\xaf\xca,\x00\x00\x00\x00\xff\xff\x03\x00d\xaa\x8e\xb5\x0f\x00\x00\x00'
decompressed_data = b"hello from gzip"
content_settings = ContentSettings(content_encoding='gzip')

# Act / Assert
blob.upload_blob(data=compressed_data, content_settings=content_settings)

result = blob.download_blob(decompress=True).readall()
assert result == decompressed_data

result = blob.download_blob(decompress=False).readall()
assert result == compressed_data

# ------------------------------------------------------------------------------
25 changes: 25 additions & 0 deletions sdk/storage/azure-storage-blob/tests/test_common_blob_async.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3526,4 +3526,29 @@ async def test_upload_blob_partial_stream_chunked(self, **kwargs):
result = await (await blob.download_blob()).readall()
assert result == data[:length]

@pytest.mark.live_test_only
@BlobPreparer()
async def test_download_blob_decompress(self, **kwargs):
storage_account_name = kwargs.pop("storage_account_name")
storage_account_key = kwargs.pop("storage_account_key")

# Arrange
await self._setup(storage_account_name, storage_account_key)
blob_name = self._get_blob_reference()
blob = self.bsc.get_blob_client(self.container_name, blob_name)
compressed_data = b'\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\xff\xcaH\xcd\xc9\xc9WH+\xca\xcfUH\xaf\xca,\x00\x00\x00\x00\xff\xff\x03\x00d\xaa\x8e\xb5\x0f\x00\x00\x00'
decompressed_data = b"hello from gzip"
content_settings = ContentSettings(content_encoding='gzip')

# Act / Assert
await blob.upload_blob(data=compressed_data, content_settings=content_settings)

downloaded = await blob.download_blob(decompress=True)
result = await downloaded.readall()
assert result == decompressed_data

downloaded = await blob.download_blob(decompress=False)
result = await downloaded.readall()
assert result == compressed_data

# ------------------------------------------------------------------------------