Allow association between notifiers and clusters #610
Description
Once notification thresholds and (and blacklist/whitelist) conditions are met, all notifiers will emit a notification.
However, this might not be the desired behaviour for all the cases. An example:
- Burrow is watching clusters C1 and C2
- Notifiers N1 and N2 are defined. N1 triggers a PagerDuty alert to the team that manages C1. N2 triggers a PagerDuty alert to the team that manages C2
- Consumer group CG1 is present on both N1 and N2. Once notification conditions are met on either cluster for CG1, N1 and N2 will be triggered, notifying one of the teams with a false positive.
If the notifiers configurations allow an association between cluster and notifier, we can have a default behaviour of triggering all Notifiers and the option of having C1 tied to N1 and C2 tied to N2.
We can reuse the tag cluster to allow this. I've tested modifications to the code and it's doable, since ConsumerGroupStatus carries the cluster name.