Is it safe to run multiple safe-settings container in parallel?

[martinm82]

Imagine you have thousands of repositories in your organization and would like to manage all the repos with safe-settings. Would it be possible to run multiple safe-settings containers in parallel (assuming with a load-balancer, e.g. traefik, in front of) and coping with the huge amount of check-run, check-suite, repository.edited, etc. events?

A change in the org settings file would not get any benefit out of it since the request would be called from one of the containers. But at least the handling of all the received events could be nicely spread among multiple containers.

Or would a different approach be better, something like adding a queuing system?

[martinm82]

Bumping this topic as we are struggling with performance. Our org has 9000 repositories and we have recently rolled out the first org setting (dependabot alerts). The cron job is running currently every hour but we already hit the rate limit of the GitHub App.

What I am observing now is that once the dependabot alerts are configured, any other request (e.g. setting topics) is just queued and waits until all 9k repos are traversed and the setting is applied.

In order to scale better I believe that the probot app should be separated into multiple ones, each of them being responsible for doing one thing, e.g., configuring branch protection rules, configuring security alerts, configuring general repository settings, etc.

The rate limit could be fixed by having multiple GitHub Apps, one for each category (branch protection, security alerts, etc.)

EDIT:
Actually we could achieve this partially with having multiple GitHub Apps, each with different events configured. One for branch protection events, one for repository events, team, etc.

[martinm82]

What I did so far is:

• two probot containers each with its separate GitHub Apps
• one is responsible for the cron sync and the GitHub App does not send any webhook events
• the other one is responsible to listen on events from the GitHub App (e.g., syncing settings in case branch protection rules got changed, repository edited/created, pull requests, check suite, etc.)

Advantage is that I have two rate limits for each container and I can guarantee that at least all events triggered from the GitHub App will be picked up by safe-settings.

In front of these two containers I am running traefik with some router rules configured to forward the requests to the appropriate probot instance.

[martinm82]

@decyjphr what do you think about such a setup? I am not sure but I believe I am hitting an issue where both apps are having a different state and the cron-based container overrides stuff due to some old cached state.

[decyjphr]

Hi @martinm82, Because the Cron-based trigger will cause the settings for all the repos to be applied, it would take a while -- even close to an hour -- to complete. If the config changes while this is running, it is possible that the Cron-based app will have the dirty config in memory and would apply the change. Based on the timing, the dirty config might be the last one being applied.
entire org.
We might have to change the design for the cron based model to make changes more atomically.

[martinm82]

Yeah, so I am running now in troubles a bit with this setup I created. I am just wondering how I could switch back to the usual setup and cope better with rate limits.
The problem is that events, e.g. branch protection rule changes might end up in the queue and be delayed due to the cron based run.

Any idea? I could of course run the cron job less often (currently every 2h).

[philthethrill99]

@martinm82 I'm planning on using safe-settings in an org with 17k repos at the moment and maybe more in the end. Do you still have issues with rate-limits and performance?

Thanks for your input

[decyjphr]

@philthethrill99
Yes, you will run into ratelimits and performance. 17K is a lot of repos for safe-settings .