Introduce user scanner to Ruler/Alertmanager #6999

SungJin1212 · 2025-08-29T10:13:53Z

This PR introduces a user scanner to reduce the number of list calls to object storage.

Which issue(s) this PR fixes:
Fixes #6914

Checklist

Tests updated
Documentation added
CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

yeya24

I wonder if we should extract user scanner into a dedicated package? pkg/storage/tsdb might not be the best place.

SungJin1212 · 2025-08-30T03:31:59Z

@yeya24
Yeah, agree. What about /pkg/usersscanner?

yeya24 · 2025-08-30T04:34:17Z

Yeah, agree. What about /pkg/usersscanner?

I am thinking about /pkg/util/users which includes everything user related, not just user scanner.

SungJin1212 · 2025-08-30T05:10:27Z

I am thinking about /pkg/util/users which includes everything user related, not just user scanner.

ok, it looks good.
I moved user-related files to /pkg/util/users, and relocated some test files to prevent cycle imports.
I renamed some files:
users_scanner_config.go -> scanner_config.go
updater.go -> index_updater.go

danielblando · 2025-09-02T18:11:42Z

What is the difference for us between user and tenant? It is a bit weird to have /pkg/util/users/tenant/.

SungJin1212 · 2025-09-03T00:37:02Z

@danielblando
I think tenant should be integrated into users, but I've left it like this for now due to a cyclic import. I'll look into a way to integrate it.

SungJin1212 · 2025-09-03T11:27:49Z

@danielblando @yeya24
I've moved the tenant package to the users package. I also deprecated util.StringsContain and replaced it with slices.Contains to resolve a cyclic import.

yeya24 · 2025-09-14T06:33:04Z

pkg/util/errors/errors.go

 	}
 }
+
+func IsOneOfTheExpectedErrors(f ...objstore.IsOpFailureExpectedFunc) objstore.IsOpFailureExpectedFunc {


Why this function is not part of the bucket package but in a generate pkg/util/errors? This seems specific to object store failures so I am not sure if it should be in the generic error file

I moved IsOneOfTheExpectedErrors to the bucket package. It causes a cyclic import if it is in the tsdb package.

What do you think of using only the list strategy and only exposing cache_ttl as config?

yeya24 · 2025-09-14T06:41:30Z

pkg/alertmanager/alertstore/bucketclient/bucket_client.go

+func NewBucketAlertStore(bkt objstore.InstrumentedBucket, userScannerCfg users.UsersScannerConfig, cfgProvider bucket.TenantConfigProvider, logger log.Logger, reg prometheus.Registerer) (*BucketAlertStore, error) {
+	alertBucket := bucket.NewPrefixedBucketClient(bkt, alertsPrefix)
+
+	usersScanner, err := users.NewScanner(userScannerCfg, alertBucket, logger, extprom.WrapRegistererWith(prometheus.Labels{"component": "alertmanager"}, reg))


Do we support user index based user scanner? Who will work as the user index updater for AM and Ruler bucket?

Oh, the user index updater would also be necessary for Alertmanager and Ruler.

Yeah, for compactor, we use Ring to choose one compactor pod to periodically update the user index in the storage bucket. I think we need the same thing for Ruler/AM as they are using different S3 buckets

How about adding clean_up_interval for the user index updater? Currently, it uses clean_up_interval in the Compactor.

yeya24 · 2025-09-14T06:42:35Z

pkg/storage/tsdb/bucketindex/loader_test.go

-	cortex_testutil "github.com/cortexproject/cortex/pkg/storage/tsdb/testutil"
 	"github.com/cortexproject/cortex/pkg/util/services"
 	"github.com/cortexproject/cortex/pkg/util/test"
+	cotex_testutil "github.com/cortexproject/cortex/pkg/util/testutil"


This seems a typo.

SungJin1212 · 2025-09-16T01:34:13Z

@yeya24
I introduced a flag clean_up_interval for the user index updater and added updaters to Alertmanager and Ruler.

yeya24 · 2025-11-01T17:55:30Z

CHANGELOG.md

 * [CHANGE] StoreGateway/Alertmanager: Add default 5s connection timeout on client. #6603
 * [CHANGE] Ingester: Remove EnableNativeHistograms config flag and instead gate keep through new per-tenant limit at ingestion. #6718
 * [CHANGE] Validate a tenantID when to use a single tenant resolver. #6727
+* [FEATURE] Alertmanager/Ruler: Introduce a user scanner to reduce the number of list calls to object storage. #6999


Can you please move this changelog to master/unreleased section?
Also I think this is more an enhancement rather than a feature

Signed-off-by: SungJin1212 <[email protected]>

pull-request-size bot added the size/L label Aug 29, 2025

dosubot bot added component/alertmanager component/rules Bits & bobs todo with rules and alerts: the ruler, config service etc. labels Aug 29, 2025

yeya24 reviewed Aug 29, 2025

View reviewed changes

pull-request-size bot added size/XXL and removed size/L labels Aug 30, 2025

SungJin1212 force-pushed the Add-user-scanner-for-am-and-ruler branch 3 times, most recently from e8c616c to 04acdd5 Compare September 1, 2025 06:31

SungJin1212 marked this pull request as draft September 1, 2025 06:38

SungJin1212 force-pushed the Add-user-scanner-for-am-and-ruler branch 2 times, most recently from ee1808c to 15f2ccd Compare September 1, 2025 06:51

SungJin1212 force-pushed the Add-user-scanner-for-am-and-ruler branch from b0e778d to 22768c9 Compare September 3, 2025 11:17

SungJin1212 marked this pull request as ready for review September 4, 2025 01:25

SungJin1212 force-pushed the Add-user-scanner-for-am-and-ruler branch 2 times, most recently from fbfa1dd to 2953348 Compare September 10, 2025 01:59

SungJin1212 requested review from danielblando and yeya24 September 10, 2025 08:52

yeya24 reviewed Sep 14, 2025

View reviewed changes

SungJin1212 force-pushed the Add-user-scanner-for-am-and-ruler branch 3 times, most recently from f262ab1 to fd54015 Compare September 15, 2025 10:59

SungJin1212 requested a review from yeya24 September 17, 2025 03:55

SungJin1212 force-pushed the Add-user-scanner-for-am-and-ruler branch from fc2f24c to 271165b Compare September 30, 2025 05:13

SungJin1212 force-pushed the Add-user-scanner-for-am-and-ruler branch from 2be7635 to 0c582b3 Compare October 24, 2025 10:22

yeya24 reviewed Nov 1, 2025

View reviewed changes

SungJin1212 force-pushed the Add-user-scanner-for-am-and-ruler branch from 0c582b3 to 714d0bd Compare November 1, 2025 23:16

SungJin1212 added 15 commits November 4, 2025 11:01

Introduce user scanner to Ruler/Alertmanager

072f9b0

Signed-off-by: SungJin1212 <[email protected]>

fix tests

61b0d46

Signed-off-by: SungJin1212 <[email protected]>

fix lint

8abb08b

Signed-off-by: SungJin1212 <[email protected]>

Move user related files to pkg/util/users

8c0f4e9

Signed-off-by: SungJin1212 <[email protected]>

fix lint

06437cf

Signed-off-by: SungJin1212 <[email protected]>

move tenant pkg

b530422

Signed-off-by: SungJin1212 <[email protected]>

move tenant to users pkg

190b658

Signed-off-by: SungJin1212 <[email protected]>

fix lint

02d4755

Signed-off-by: SungJin1212 <[email protected]>

fix typo

8a1cfaa

Signed-off-by: SungJin1212 <[email protected]>

Move IsOneOfTheExpectedErrors to bucket package

a7429ad

Signed-off-by: SungJin1212 <[email protected]>

Add index updater to alertmanager/ruler

26e56e9

Signed-off-by: SungJin1212 <[email protected]>

fix lint

794ee64

Signed-off-by: SungJin1212 <[email protected]>

fix test

6ec62df

Signed-off-by: SungJin1212 <[email protected]>

update json config

bae3680

Signed-off-by: SungJin1212 <[email protected]>

move changelog

015b423

Signed-off-by: SungJin1212 <[email protected]>

SungJin1212 force-pushed the Add-user-scanner-for-am-and-ruler branch from 714d0bd to 015b423 Compare November 4, 2025 02:01

Introduce user scanner to Ruler/Alertmanager #6999

Are you sure you want to change the base?

Introduce user scanner to Ruler/Alertmanager #6999

Conversation

SungJin1212 commented Aug 29, 2025

Uh oh!

yeya24 left a comment

Choose a reason for hiding this comment

Uh oh!

SungJin1212 commented Aug 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yeya24 commented Aug 30, 2025

Uh oh!

SungJin1212 commented Aug 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

danielblando commented Sep 2, 2025

Uh oh!

SungJin1212 commented Sep 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SungJin1212 commented Sep 3, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SungJin1212 commented Sep 16, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

SungJin1212 commented Aug 30, 2025 •

edited

Loading

SungJin1212 commented Aug 30, 2025 •

edited

Loading

SungJin1212 commented Sep 3, 2025 •

edited

Loading