Skip to content

S3 KMS SSE Errors in Compactor #5711

@goodacres

Description

@goodacres

Describe the bug
When using SSE & S3 bucket IAM resource policies that enforce KMS SSE headers on write actions, block compaction fails with IAM error:

level=info ts=2025-10-03T08:34:52.296662765Z caller=retention.go:80 msg="marking block for deletion" blockID=9ed07faa-275d-495d-b211-61ed48b0f271 tenantID=single-tenant

level=error ts=2025-10-03T08:34:52.351880904Z caller=retention.go:83 msg="failed to mark block compacted during retention" blockID=9ed07faa-275d-495d-b211-61ed48b0f271 tenantID=single-tenant err="error copying obj meta to compacted obj meta: User: arn:aws:sts::123456789:assumed-role/role-name/session is not authorized to perform: s3:PutObject on resource: \"arn:aws:s3:::bucket-name/single-tenant/9ed07faa-275d-495d-b211-61ed48b0f271/meta.compacted.json\" with an explicit deny in a resource-based policy"

Other Tempo bucket write actions work successfully. When the bucket IAM resource policy is removed, the error clears up.

I believe it may be down to the minio CopyObject method not including SSE headers like it does for PutObject, but I am not super familiar with Go, so not completely sure.

To Reproduce
Steps to reproduce the behavior:

  1. Start Tempo (Version: 2.8.2)
  2. Configure storage config for S3 with KMS SSE:
storage:
    trace:
      backend: s3
      s3:
        bucket: bucket-name
        endpoint: s3.amazonaws.com
        region: region        
        access_key: ""
        secret_key: ""
        insecure: false
        sse:
          type: SSE-KMS
          kms_key_id: 1234-1234-1234-1234-1234
        tags: 
          creator: tempo
      wal:
        path: /tmp/tempo/wal
  1. Set IAM resource policy on S3 bucket:
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyUnencryptedUploads",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::bucket-name/*",
      "Condition": {
        "StringNotEquals": {
          "s3:x-amz-server-side-encryption": "aws:kms"
        }
      }
    },
    {
      "Sid": "DenyWrongKey",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::bucket-name/*",
      "Condition": {
        "StringNotEquals": {
          "s3:x-amz-server-side-encryption-aws-kms-key-id": "arn:aws:kms:region:123456789:key/1234-1234-1234-1234-1234"
        }
      }
    }
  ]
}

Expected behavior
Tempo sends CopyObject requests to S3 with required SSE headers

Environment:

  • Infrastructure: Kubernetes
  • Deployment tool: helm

Additional Context
Here are some links to the associated code:

SSE headers not passed for copyObject (sad scenario)

https://github.com/grafana/tempo/blob/main/tempodb/backend/s3/compactor.go#L15-L43

https://github.com/minio/minio-go/blob/master/core.go#L58-L61

https://github.com/minio/minio-go/blob/master/api-compose-object.go#L232-L319

SSE headers passed for putObject (happy scenario)

https://github.com/minio/minio-go/blob/master/api-put-object.go#L196-L198

https://github.com/grafana/tempo/blob/main/tempodb/backend/s3/s3.go#L158-L166

https://github.com/grafana/tempo/blob/main/tempodb/backend/s3/s3.go#L38-L45

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions