Failover guidelines for submariner broker

[abguptms]

Our current setup for Submariner is like this:
Control plane K8s cluster - hosts broker
Data plane K8s clusters - host agents

Over time we have had multiple control plane issues which have made us start building solutions around failing over control pane components to a new clusters instead of always trying to fix existing control plane. We generally use AKS and work with identities for cluster role based access control. However, for submariner the tokens issued to agents are using a service account as a static form of authentication.

Question: Are there recommendation on how to perform broker failovers without disrupting the service discovery model?

[abguptms]

@tpantelis @vthapar tagging you explicitly to hopefully get some guidance here. We are hitting this requirement to migrate broker between clusters but not sure how to do it without downtime. It has become a critical blocker area for us to have 100% failover capabilities within region.