-
Couldn't load subscription status.
- Fork 628
Description
Describe the bug
When using SSE & S3 bucket IAM resource policies that enforce KMS SSE headers on write actions, block compaction fails with IAM error:
level=info ts=2025-10-03T08:34:52.296662765Z caller=retention.go:80 msg="marking block for deletion" blockID=9ed07faa-275d-495d-b211-61ed48b0f271 tenantID=single-tenant
level=error ts=2025-10-03T08:34:52.351880904Z caller=retention.go:83 msg="failed to mark block compacted during retention" blockID=9ed07faa-275d-495d-b211-61ed48b0f271 tenantID=single-tenant err="error copying obj meta to compacted obj meta: User: arn:aws:sts::123456789:assumed-role/role-name/session is not authorized to perform: s3:PutObject on resource: \"arn:aws:s3:::bucket-name/single-tenant/9ed07faa-275d-495d-b211-61ed48b0f271/meta.compacted.json\" with an explicit deny in a resource-based policy"
Other Tempo bucket write actions work successfully. When the bucket IAM resource policy is removed, the error clears up.
I believe it may be down to the minio CopyObject method not including SSE headers like it does for PutObject, but I am not super familiar with Go, so not completely sure.
To Reproduce
Steps to reproduce the behavior:
- Start Tempo (Version: 2.8.2)
- Configure
storageconfig for S3 with KMS SSE:
storage:
trace:
backend: s3
s3:
bucket: bucket-name
endpoint: s3.amazonaws.com
region: region
access_key: ""
secret_key: ""
insecure: false
sse:
type: SSE-KMS
kms_key_id: 1234-1234-1234-1234-1234
tags:
creator: tempo
wal:
path: /tmp/tempo/wal
- Set IAM resource policy on S3 bucket:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "DenyUnencryptedUploads",
"Effect": "Deny",
"Principal": "*",
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::bucket-name/*",
"Condition": {
"StringNotEquals": {
"s3:x-amz-server-side-encryption": "aws:kms"
}
}
},
{
"Sid": "DenyWrongKey",
"Effect": "Deny",
"Principal": "*",
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::bucket-name/*",
"Condition": {
"StringNotEquals": {
"s3:x-amz-server-side-encryption-aws-kms-key-id": "arn:aws:kms:region:123456789:key/1234-1234-1234-1234-1234"
}
}
}
]
}
Expected behavior
Tempo sends CopyObject requests to S3 with required SSE headers
Environment:
- Infrastructure: Kubernetes
- Deployment tool: helm
Additional Context
Here are some links to the associated code:
SSE headers not passed for copyObject (sad scenario)
https://github.com/grafana/tempo/blob/main/tempodb/backend/s3/compactor.go#L15-L43
https://github.com/minio/minio-go/blob/master/core.go#L58-L61
https://github.com/minio/minio-go/blob/master/api-compose-object.go#L232-L319
SSE headers passed for putObject (happy scenario)
https://github.com/minio/minio-go/blob/master/api-put-object.go#L196-L198
https://github.com/grafana/tempo/blob/main/tempodb/backend/s3/s3.go#L158-L166
https://github.com/grafana/tempo/blob/main/tempodb/backend/s3/s3.go#L38-L45