Skip to content

Commit 01db425

Browse files
k8s-ci-robotk8s-ci-robot
authored
Merge pull request #2517 from AndrewSirenko/metadata-sources
Add `--metadata-sources` parameter to opt-out or choose fallback ordering of metadata sources
2 parents 95425ce + 37e51c0 commit 01db425

File tree

14 files changed

+307
-156
lines changed

14 files changed

+307
-156
lines changed

‎charts/aws-ebs-csi-driver/templates/_node-windows.tpl

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -76,6 +76,9 @@ spec:
7676
{{- with .Values.node.reservedVolumeAttachments }}
7777
- --reserved-volume-attachments={{ . }}
7878
{{- end }}
79+
{{- with .Values.node.metadataSources }}
80+
- --metadata-sources={{ . }}
81+
{{- end }}
7982
{{- with .Values.node.volumeAttachLimit }}
8083
- --volume-attach-limit={{ . }}
8184
{{- end }}

‎charts/aws-ebs-csi-driver/templates/_node.tpl

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -83,6 +83,9 @@ spec:
8383
{{- with .Values.node.volumeAttachLimit }}
8484
- --volume-attach-limit={{ . }}
8585
{{- end }}
86+
{{- with .Values.node.metadataSources }}
87+
- --metadata-sources={{ . }}
88+
{{- end }}
8689
{{- if .Values.node.legacyXFS }}
8790
- --legacy-xfs=true
8891
{{- end}}

‎charts/aws-ebs-csi-driver/values.schema.json

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -600,6 +600,11 @@
600600
"description": "Enable SELinux-only optimizations on the EBS CSI Driver node pods. Must only be set true if all linux nodes in the DaemonSet have SELinux enabled",
601601
"default": false
602602
},
603+
"metadataSources": {
604+
"description": "Comma separated list of metadata sources that override the default used by the EBS CSI Driver. Valid sources include 'imds' and 'kubernetes'",
605+
"type": ["string", "null"],
606+
"default": null
607+
},
603608
"reservedVolumeAttachments": {
604609
"type": ["integer", "null"],
605610
"description": "The number of attachment slots to reserve for system use (and not to be used for CSI volumes)\nWhen this parameter is not specified (or set to -1), the EBS CSI Driver will attempt to determine the number of reserved slots via heuristic",

‎charts/aws-ebs-csi-driver/values.yaml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -406,6 +406,7 @@ node:
406406
# Enable the linux daemonset creation
407407
enableLinux: true
408408
enableWindows: true
409+
metadataSources:
409410
# Warning: This option will be removed in a future release. It is a temporary workaround for users unable to immediately migrate off of older kernel versions.
410411
# Formats XFS volumes with bigtime=0,inobtcount=0,reflink=0, for mounting onto nodes with linux kernel version <= 5.4.
411412
# Note that XFS volumes formatted with this option will only have timestamp records until 2038.

‎cmd/main.go

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -135,8 +135,9 @@ func main() {
135135
}
136136

137137
cfg := metadata.MetadataServiceConfig{
138-
EC2MetadataClient: metadata.DefaultEC2MetadataClient,
139-
K8sAPIClient: metadata.DefaultKubernetesAPIClient(options.Kubeconfig),
138+
MetadataSources: options.MetadataSources,
139+
IMDSClient: metadata.DefaultIMDSClient,
140+
K8sAPIClient: metadata.DefaultKubernetesAPIClient(options.Kubeconfig),
140141
}
141142

142143
region := os.Getenv("AWS_REGION")

‎docs/install.md

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44

55
* Kubernetes Version >= 1.20
66

7-
* If you are using a self managed cluster, ensure the flag `--allow-privileged=true` for `kube-apiserver`.
7+
* If you are using a self-managed cluster, ensure the flag `--allow-privileged=true` for `kube-apiserver`.
88

99
* Important: If you intend to use the Volume Snapshot feature, the [Kubernetes Volume Snapshot CRDs](https://github.com/kubernetes-csi/external-snapshotter/tree/master/client/config/crd) must be installed **before** the EBS CSI driver. For installation instructions, see [CSI Snapshotter Usage](https://github.com/kubernetes-csi/external-snapshotter#usage).
1010

@@ -14,6 +14,8 @@ The EBS CSI Driver uses a metadata source in order to gather necessary informati
1414

1515
The controller `Deployment` can skip metadata if the region is provided via the `AWS_REGION` environment variable (Helm parameter `controller.region`). The node `DaemonSet` requires metadata and will not function without access to one of the sources.
1616

17+
You may override the default metadata behavior of attempting IMDS, then falling back to Kubernetes, through the `--metadata-sources` flag.
18+
1719
#### IMDS (EC2) Metadata
1820

1921
If the driver is able to access IMDS, it will utilize that as a preferred source of metadata. The EBS CSI Driver supports IMDSv1 and IMDSv2 (and will prefer IMDSv2 if both are available). However, by default, [IMDSv2 uses a hop limit of 1](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html#instance-metadata-v2-how-it-works). That will prevent the driver from accessing IMDSv2 if run inside a container with the default IMDSv2 configuration.
@@ -22,7 +24,7 @@ In order for the driver to access IMDS, it either must be run in host networking
2224

2325
#### Kubernetes Metadata
2426

25-
If the driver is unable to reach IMDS, it will fallback to using the Kubernetes API. For this metadata source to work, the driver pods must have access to the Kubernetes API server. Additionally, the Kubernetes node objects must include the following information:
27+
By default, if the driver is unable to reach IMDS, it will fall back to using the Kubernetes API. For this metadata source to work, the driver pods must have access to the Kubernetes API server. Additionally, the Kubernetes node objects must include the following information:
2628

2729
- Instance ID (in the `Node`'s `ProviderID`)
2830
- Instance Type (in the label `node.kubernetes.io/instance-type`)

‎docs/options.md

Lines changed: 19 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -2,21 +2,22 @@
22

33
There are a couple of driver options that can be passed as arguments when starting the driver container.
44

5-
| Option argument | value sample | default | Description |
6-
|---------------------------|-------------------------|--------------------------------------------------|---------------------|
7-
| endpoint | tcp://127.0.0.1:10000/ | unix:///var/lib/csi/sockets/pluginproxy/csi.sock | The socket on which the driver will listen for CSI RPCs|
8-
| http-endpoint | :8080 | | The TCP network address where the HTTP server for metrics will listen (example: `:8080`). The default is empty string, which means the server is disabled.|
9-
| metrics-cert-file | /metrics.crt | | The path to a certificate to use for serving the metrics server over HTTPS. If the certificate is signed by a certificate authority, this file should be the concatenation of the server's certificate, any intermediates, and the CA's certificate. If this is non-empty, `--http-endpoint` and `--metrics-key-file` MUST also be non-empty.|
10-
| metrics-key-file | /metrics.key | | The path to a key to use for serving the metrics server over HTTPS. If this is non-empty, `--http-endpoint` and `--metrics-cert-file` MUST also be non-empty.|
11-
| volume-attach-limit | 1,2,3 ... | -1 | Value for the maximum number of volumes attachable per node. If specified, the limit applies to all nodes. If not specified, the value is approximated from the instance type|
12-
| extra-tags | key1=value1,key2=value2 | | Tags attached to each dynamically provisioned resource|
13-
| k8s-tag-cluster-id | aws-cluster-id-1 | | ID of the Kubernetes cluster used for tagging provisioned EBS volumes|
14-
| aws-sdk-debug-log | true | false | If set to true, the driver will enable the aws sdk debug log level|
15-
| logging-format | json | text | Sets the log format. Permitted formats: text, json|
16-
| user-agent-extra | csi-ebs | helm | Extra string appended to user agent|
17-
| enable-otel-tracing | true | false | If set to true, the driver will enable opentelemetry tracing. Might need [additional env variables](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#general-sdk-configuration) to export the traces to the right collector|
18-
| batching | true | true | If set to true, the driver will enable batching of API calls. This is especially helpful for improving performance in workloads that are sensitive to EC2 rate limits at the cost of a small increase to worst-case latency|
19-
| modify-volume-request-handler-timeout | 10s | 2s | Timeout for the window in which volume modification calls must be received in order for them to coalesce into a single volume modification call to AWS. If changing this, be aware that the ebs-csi-controller's csi-resizer and volumemodifier containers both have timeouts on the calls they make, if this value exceeds those timeouts it will cause them to always fail and fall into a retry loop, so adjust those values accordingly.
20-
| warn-on-invalid-tag | true | false | To warn on invalid tags, instead of returning an error|
21-
| reserved-volume-attachments | 2 | -1 | Number of volume attachments reserved for system use. Not used when --volume-attach-limit is specified. When -1, the amount of reserved attachments is loaded from instance metadata that captured state at node boot and may include not only system disks but also CSI volumes.|
22-
| legacy-xfs | true | false | Warning: This option will be removed in a future release. It is a temporary workaround for users unable to immediately migrate off of older kernel versions. Formats XFS volumes with `bigtime=0,inobtcount=0,reflink=0`, so that they can be mounted onto nodes with linux kernel ≤ v5.4. Volumes formatted with this option may experience issues after 2038, and will be unable to use some XFS features (for example, reflinks).|
5+
| Option argument | value sample | default | Description |
6+
|---------------------------------------|-------------------------|--------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
7+
| endpoint | tcp://127.0.0.1:10000/ | unix:///var/lib/csi/sockets/pluginproxy/csi.sock | The socket on which the driver will listen for CSI RPCs |
8+
| http-endpoint | :8080 | | The TCP network address where the HTTP server for metrics will listen (example: `:8080`). The default is empty string, which means the server is disabled. |
9+
| metrics-cert-file | /metrics.crt | | The path to a certificate to use for serving the metrics server over HTTPS. If the certificate is signed by a certificate authority, this file should be the concatenation of the server's certificate, any intermediates, and the CA's certificate. If this is non-empty, `--http-endpoint` and `--metrics-key-file` MUST also be non-empty. |
10+
| metrics-key-file | /metrics.key | | The path to a key to use for serving the metrics server over HTTPS. If this is non-empty, `--http-endpoint` and `--metrics-cert-file` MUST also be non-empty. |
11+
| volume-attach-limit | 1,2,3 ... | -1 | Value for the maximum number of volumes attachable per node. If specified, the limit applies to all nodes. If not specified, the value is approximated from the instance type |
12+
| extra-tags | key1=value1,key2=value2 | | Tags attached to each dynamically provisioned resource |
13+
| k8s-tag-cluster-id | aws-cluster-id-1 | | ID of the Kubernetes cluster used for tagging provisioned EBS volumes |
14+
| aws-sdk-debug-log | true | false | If set to true, the driver will enable the aws sdk debug log level |
15+
| logging-format | json | text | Sets the log format. Permitted formats: text, json |
16+
| user-agent-extra | csi-ebs | helm | Extra string appended to user agent |
17+
| enable-otel-tracing | true | false | If set to true, the driver will enable opentelemetry tracing. Might need [additional env variables](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#general-sdk-configuration) to export the traces to the right collector |
18+
| batching | true | true | If set to true, the driver will enable batching of API calls. This is especially helpful for improving performance in workloads that are sensitive to EC2 rate limits at the cost of a small increase to worst-case latency |
19+
| modify-volume-request-handler-timeout | 10s | 2s | Timeout for the window in which volume modification calls must be received in order for them to coalesce into a single volume modification call to AWS. If changing this, be aware that the ebs-csi-controller's csi-resizer and volumemodifier containers both have timeouts on the calls they make, if this value exceeds those timeouts it will cause them to always fail and fall into a retry loop, so adjust those values accordingly.
20+
| warn-on-invalid-tag | true | false | To warn on invalid tags, instead of returning an error |
21+
| reserved-volume-attachments | 2 | -1 | Number of volume attachments reserved for system use. Not used when --volume-attach-limit is specified. When -1, the amount of reserved attachments is loaded from instance metadata that captured state at node boot and may include not only system disks but also CSI volumes. |
22+
| legacy-xfs | true | false | Warning: This option will be removed in a future release. It is a temporary workaround for users unable to immediately migrate off of older kernel versions. Formats XFS volumes with `bigtime=0,inobtcount=0,reflink=0`, so that they can be mounted onto nodes with linux kernel ≤ v5.4. Volumes formatted with this option may experience issues after 2038, and will be unable to use some XFS features (for example, reflinks). |
23+
| metadata-sources | imds | imds,kubernetes | Dictates which sources are used to retrieve instance metadata. The driver will attempt to rely on each source in order until one succeeds. Valid options include 'imds' and 'kubernetes'. |

0 commit comments

Comments
 (0)