Skip to content

Commit 67945ba

Browse files
razvanrazvan
committed
client spooling: make s3 filesystem consistent with FTE backend
1 parent 2bd9cd4 commit 67945ba

File tree

6 files changed

+126
-106
lines changed

6 files changed

+126
-106
lines changed

‎deploy/helm/trino-operator/crds/crds.yaml‎

Lines changed: 105 additions & 100 deletions
Original file line numberDiff line numberDiff line change
@@ -120,125 +120,130 @@ spec:
120120
- s3
121121
properties:
122122
s3:
123-
oneOf:
124-
- required:
125-
- inline
126-
- required:
127-
- reference
128123
properties:
129-
inline:
130-
description: S3 connection definition as a resource. Learn more on the [S3 concept documentation](https://docs.stackable.tech/home/nightly/concepts/s3).
124+
connection:
125+
oneOf:
126+
- required:
127+
- inline
128+
- required:
129+
- reference
131130
properties:
132-
accessStyle:
133-
default: VirtualHosted
134-
description: Which access style to use. Defaults to virtual hosted-style as most of the data products out there. Have a look at the [AWS documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html).
135-
enum:
136-
- Path
137-
- VirtualHosted
138-
type: string
139-
credentials:
140-
description: If the S3 uses authentication you have to specify you S3 credentials. In the most cases a [SecretClass](https://docs.stackable.tech/home/nightly/secret-operator/secretclass) providing `accessKey` and `secretKey` is sufficient.
141-
nullable: true
131+
inline:
132+
description: S3 connection definition as a resource. Learn more on the [S3 concept documentation](https://docs.stackable.tech/home/nightly/concepts/s3).
142133
properties:
143-
scope:
144-
description: '[Scope](https://docs.stackable.tech/home/nightly/secret-operator/scope) of the [SecretClass](https://docs.stackable.tech/home/nightly/secret-operator/secretclass).'
134+
accessStyle:
135+
default: VirtualHosted
136+
description: Which access style to use. Defaults to virtual hosted-style as most of the data products out there. Have a look at the [AWS documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html).
137+
enum:
138+
- Path
139+
- VirtualHosted
140+
type: string
141+
credentials:
142+
description: If the S3 uses authentication you have to specify you S3 credentials. In the most cases a [SecretClass](https://docs.stackable.tech/home/nightly/secret-operator/secretclass) providing `accessKey` and `secretKey` is sufficient.
145143
nullable: true
146144
properties:
147-
listenerVolumes:
148-
default: []
149-
description: The listener volume scope allows Node and Service scopes to be inferred from the applicable listeners. This must correspond to Volume names in the Pod that mount Listeners.
150-
items:
151-
type: string
152-
type: array
153-
node:
154-
default: false
155-
description: The node scope is resolved to the name of the Kubernetes Node object that the Pod is running on. This will typically be the DNS name of the node.
156-
type: boolean
157-
pod:
158-
default: false
159-
description: The pod scope is resolved to the name of the Kubernetes Pod. This allows the secret to differentiate between StatefulSet replicas.
160-
type: boolean
161-
services:
162-
default: []
163-
description: The service scope allows Pod objects to specify custom scopes. This should typically correspond to Service objects that the Pod participates in.
164-
items:
165-
type: string
166-
type: array
145+
scope:
146+
description: '[Scope](https://docs.stackable.tech/home/nightly/secret-operator/scope) of the [SecretClass](https://docs.stackable.tech/home/nightly/secret-operator/secretclass).'
147+
nullable: true
148+
properties:
149+
listenerVolumes:
150+
default: []
151+
description: The listener volume scope allows Node and Service scopes to be inferred from the applicable listeners. This must correspond to Volume names in the Pod that mount Listeners.
152+
items:
153+
type: string
154+
type: array
155+
node:
156+
default: false
157+
description: The node scope is resolved to the name of the Kubernetes Node object that the Pod is running on. This will typically be the DNS name of the node.
158+
type: boolean
159+
pod:
160+
default: false
161+
description: The pod scope is resolved to the name of the Kubernetes Pod. This allows the secret to differentiate between StatefulSet replicas.
162+
type: boolean
163+
services:
164+
default: []
165+
description: The service scope allows Pod objects to specify custom scopes. This should typically correspond to Service objects that the Pod participates in.
166+
items:
167+
type: string
168+
type: array
169+
type: object
170+
secretClass:
171+
description: '[SecretClass](https://docs.stackable.tech/home/nightly/secret-operator/secretclass) containing the LDAP bind credentials.'
172+
type: string
173+
required:
174+
- secretClass
167175
type: object
168-
secretClass:
169-
description: '[SecretClass](https://docs.stackable.tech/home/nightly/secret-operator/secretclass) containing the LDAP bind credentials.'
176+
host:
177+
description: 'Host of the S3 server without any protocol or port. For example: `west1.my-cloud.com`.'
170178
type: string
171-
required:
172-
- secretClass
173-
type: object
174-
host:
175-
description: 'Host of the S3 server without any protocol or port. For example: `west1.my-cloud.com`.'
176-
type: string
177-
port:
178-
description: Port the S3 server listens on. If not specified the product will determine the port to use.
179-
format: uint16
180-
minimum: 0.0
181-
nullable: true
182-
type: integer
183-
region:
184-
default:
185-
name: us-east-1
186-
description: |-
187-
Bucket region used for signing headers (sigv4).
179+
port:
180+
description: Port the S3 server listens on. If not specified the product will determine the port to use.
181+
format: uint16
182+
minimum: 0.0
183+
nullable: true
184+
type: integer
185+
region:
186+
default:
187+
name: us-east-1
188+
description: |-
189+
Bucket region used for signing headers (sigv4).
188190
189-
This defaults to `us-east-1` which is compatible with other implementations such as Minio.
191+
This defaults to `us-east-1` which is compatible with other implementations such as Minio.
190192
191-
WARNING: Some products use the Hadoop S3 implementation which falls back to us-east-2.
192-
properties:
193-
name:
194-
default: us-east-1
195-
type: string
196-
type: object
197-
tls:
198-
description: Use a TLS connection. If not specified no TLS will be used.
199-
nullable: true
200-
properties:
201-
verification:
202-
description: The verification method used to verify the certificates of the server and/or the client.
203-
oneOf:
204-
- required:
205-
- none
206-
- required:
207-
- server
193+
WARNING: Some products use the Hadoop S3 implementation which falls back to us-east-2.
208194
properties:
209-
none:
210-
description: Use TLS but don't verify certificates.
211-
type: object
212-
server:
213-
description: Use TLS and a CA certificate to verify the server.
195+
name:
196+
default: us-east-1
197+
type: string
198+
type: object
199+
tls:
200+
description: Use a TLS connection. If not specified no TLS will be used.
201+
nullable: true
202+
properties:
203+
verification:
204+
description: The verification method used to verify the certificates of the server and/or the client.
205+
oneOf:
206+
- required:
207+
- none
208+
- required:
209+
- server
214210
properties:
215-
caCert:
216-
description: CA cert to verify the server.
217-
oneOf:
218-
- required:
219-
- webPki
220-
- required:
221-
- secretClass
211+
none:
212+
description: Use TLS but don't verify certificates.
213+
type: object
214+
server:
215+
description: Use TLS and a CA certificate to verify the server.
222216
properties:
223-
secretClass:
224-
description: Name of the [SecretClass](https://docs.stackable.tech/home/nightly/secret-operator/secretclass) which will provide the CA certificate. Note that a SecretClass does not need to have a key but can also work with just a CA certificate, so if you got provided with a CA cert but don't have access to the key you can still use this method.
225-
type: string
226-
webPki:
227-
description: Use TLS and the CA certificates trusted by the common web browsers to verify the server. This can be useful when you e.g. use public AWS S3 or other public available services.
217+
caCert:
218+
description: CA cert to verify the server.
219+
oneOf:
220+
- required:
221+
- webPki
222+
- required:
223+
- secretClass
224+
properties:
225+
secretClass:
226+
description: Name of the [SecretClass](https://docs.stackable.tech/home/nightly/secret-operator/secretclass) which will provide the CA certificate. Note that a SecretClass does not need to have a key but can also work with just a CA certificate, so if you got provided with a CA cert but don't have access to the key you can still use this method.
227+
type: string
228+
webPki:
229+
description: Use TLS and the CA certificates trusted by the common web browsers to verify the server. This can be useful when you e.g. use public AWS S3 or other public available services.
230+
type: object
228231
type: object
232+
required:
233+
- caCert
229234
type: object
230-
required:
231-
- caCert
232235
type: object
236+
required:
237+
- verification
233238
type: object
234239
required:
235-
- verification
240+
- host
236241
type: object
237-
required:
238-
- host
242+
reference:
243+
type: string
239244
type: object
240-
reference:
241-
type: string
245+
required:
246+
- connection
242247
type: object
243248
type: object
244249
location:

‎docs/modules/trino/pages/usage-guide/client-spooling-protocol.adoc‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,8 @@ spec:
2626
location: "s3://spooling-bucket/trino/" # <1>
2727
filesystem:
2828
s3: # <2>
29-
reference: "minio"
29+
connection:
30+
reference: "minio"
3031
----
3132
<1> Specifies the location where spooled data will be stored. This example uses an S3 bucket.
3233
<2> Configures the filesystem type for spooling. Only S3 is supported currently via the custom resource definition.

‎rust/operator-binary/src/config/client_protocol.rs‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -73,7 +73,9 @@ impl ResolvedClientProtocolConfig {
7373
match &spooling_config.filesystem {
7474
SpoolingFileSystemConfig::S3(s3_config) => {
7575
let resolved_s3_config = config::s3::ResolvedS3Config::from_config(
76-
s3_config, client, namespace,
76+
&s3_config.connection,
77+
client,
78+
namespace,
7779
)
7880
.await
7981
.context(ResolveS3ConnectionSnafu)?;
@@ -132,7 +134,8 @@ mod tests {
132134
location: s3://my-bucket/spooling
133135
filesystem:
134136
s3:
135-
reference: test-s3-connection
137+
connection:
138+
reference: test-s3-connection
136139
"#};
137140

138141
let deserializer = serde_yaml::Deserializer::from_str(config_yaml);

0 commit comments

Comments
 (0)