Add Readiness/Liveness EC2 API call probe to Controller Service

[AndrewSirenko]

What type of PR is this?

/kind feature

What is this PR about? / Why do we need it?

Lets customers realize that their EBS CSI Driver ebs-csi-controller pods aren't actually ready due to some Auth/networking error.

Should really help customer troubleshooting for new customers (and ensuring networking and auth issues aren't root cause of user problems)

Should work for other COs because this is done through CSI Probe RPC

Fixes #1551

How was this change tested?

Deploy driver without right IAM role, see not ready

Add IAM role, see ready

Remove role again, see driver eventually become not ready

Add role back and driver becomes ready again

Does this PR introduce a user-facing change?

Add Readiness probe via EC2 API call to Controller Service. 

Warning: Ensure that the IAM policy associated with your EBS CSI Driver has permission for ec2:DescribeAvailabilityZones. Clusters with missing IAM roles or networking issues may see ebs-csi-controller pod restarts.

[github-actions]

Code Coverage Diff

This PR does not change the code coverage

[AndrewSirenko]

/retest

[ConnorJC3]

Nitpick: Can we add "Fixes #1551" to the PR description?

Also this isn't a breaking change, that should be removed from the changelog entry.

[AndrewSirenko]

Also this isn't a breaking change, that should be removed from the changelog entry.

Can we switch to warning during changelog? I fear some customers with misconfigured drivers will be very concerned for a second after upgrading 😓

[torredil]

/lgtm

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from torredil. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ElijahQuinones]

/hold

holding for release

[ConnorJC3]

Mostly lgtm, one question: Have we tested what this looks like for a timeout scenario? Do we need to raise this value to avoid the dry run cutting out?

https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/charts/aws-ebs-csi-driver/templates/controller.yaml#L187

[AndrewSirenko]

Have we tested what this looks like for a timeout scenario? Do we need to raise this value to avoid the dry run cutting out?

Have tested for timeout scenario (purposefull broke my policy AND also tested breaking networking)

However good callout, I'll raise timeout to 10s for live and readiness, and then raise liveness failure threshold to 10 (so pod becomes not ready first, THEN starts restarting)

[AndrewSirenko]

/hold

For final manual test