feat: Retire CA certificates (#650)

* feat: Add retirement duration for CA certificates

* docs: Document the CA certificate retirement

* chore: Update changelog

* test: Add comment to `expect`

* chore: Fix rustdoc warnings

* chore: Address review comments

* chore: Explain the calculation of safe_max_cert_lifetime

Author: siegfriedweber

CHANGELOG.md

@@ -12,6 +12,8 @@ All notable changes to this project will be documented in this file.
   - `EOS_DISABLED` (`--eos-disabled`) to disable the EoS checker completely.
 - Support exporting the TrustStore CA certificate information to Secrets or ConfigMaps ([#597]).
 - New helm value for `priorityClassName` ([#641]).
+- CA certificates are retired one hour (configurable via
+  `autoTls.ca.caCertificateRetirementDuration`) before they expire ([#650]).
 
 ### Changed
 
@@ -30,6 +32,8 @@ All notable changes to this project will be documented in this file.
 - Bump csi-node-driver-registrar to `v2.15.0` ([#642]).
 - Bump csi-provisioner to `v5.3.0` ([#643]).
 - OLM deployer: patch the new Deployment object too and other changes to align with the new operator structure ([#648]).
+- BREAKING: Expired and retired CA certificates are no longer published in Volumes and TrustStores
+  ([#650]).
 
 [#597]: https://github.com/stackabletech/secret-operator/pull/597
 [#636]: https://github.com/stackabletech/secret-operator/pull/636
@@ -39,6 +43,7 @@ All notable changes to this project will be documented in this file.
 [#644]: https://github.com/stackabletech/secret-operator/pull/644
 [#645]: https://github.com/stackabletech/secret-operator/pull/645
 [#648]: https://github.com/stackabletech/secret-operator/pull/648
+[#650]: https://github.com/stackabletech/secret-operator/pull/650
 
 ## [25.7.0] - 2025-07-23
 

Cargo.nix

@@ -40,7 +40,7 @@ socket2 = { version = "0.6", features = ["all"] }
 strum = { version = "0.27", features = ["derive"] }
 sys-mount = { version = "3.0", default-features = false }
 tempfile = "3.12"
-time = { version = "0.3", features = ["parsing"] }
+time = { version = "0.3", features = ["macros", "parsing"] }
 tokio = { version = "1.40", features = ["full"] }
 tokio-stream = { version = "0.1", features = ["net"] }
 tonic = "0.14"

Cargo.toml

@@ -88,16 +88,18 @@ In case an operator sets a higher lifetime, a tracking issue must be created to
 Users can use xref:concepts:overrides.adoc#pod-overrides[podOverrides] to extend the certificate lifetime by adding volume annotations.
 Native support for customizing certificate lifetimes in Stacklet CRDs might be added in the future.
 
+[#ca-rotation]
 ==== Certificate Authority rotation
 
 Certificate authorities also have a limited lifetime, and need to be rotated before they expire to avoid cluster disruption.
 
-If configured to provision its own CA (`autoTls.ca.autoGenerate`), the Secret Operator will create CA certificates that are valid for 365 days (≃ 1 year, configurable via `autoTls.ca.caCertificateLifetime`), and initiate rotation once less than half of that time remains.
+If configured to provision its own CA (`autoTls.ca.autoGenerate`), the Secret Operator will create CA certificates that are valid for 365 days (≃ 1 year, configurable via `autoTls.ca.caCertificateLifetime`).
+The CA certificate is retired one hour before its expiration (configurable via `autoTls.ca.caCertificateRetirementDuration`), to avoid that an almost expired certificate must be deployed, which causes problems in some products, e.g. OpenSearch.
+Once less than half of the active lifetime (= lifetime - retirement duration) remains, the rotation is initiated.
 
 To avoid disruption and let the new CA propagate through the cluster, the Secret Operator will prefer using the oldest CA that will last for the entire lifetime of the issued certificate.
 
-NOTE: Expired CA certificates will currently not be deleted automatically.
-They should be cleaned up manually.
+NOTE: Expired and retired CA certificates will not be deployed.
 
 ==== Reference
 
@@ -112,6 +114,7 @@ spec:
           namespace: default
         autoGenerate: true
         caCertificateLifetime: 700d
+        caCertificateRetirementDuration: 1d
         keyGeneration:
           rsa:
             length: 4096
@@ -131,6 +134,7 @@ spec:
                       and `ca.key` respectively.
 `autoTls.ca.autoGenerate`:: Whether the certificate authority should be provisioned and managed by the Secret Operator.
 `autoTls.ca.caCertificateLifetime` :: The lifetime of the certificate authority's root certificate.
+`autoTls.ca.caCertificateRetirementDuration` :: Duration at the end of the CA certificate lifetime where no signed certificate will exist.
 `autoTls.ca.keyGeneration`:: Configures how keys should be generated.
 `autoTls.ca.keyGeneration.rsa`:: Declares that keys should be generated using the RSA algorithm.
 `autoTls.ca.keyGeneration.rsa.length`:: The amount of bits used for generating the RSA key pair. Currently, `2048`, `3072` and `4096` are supported. Defaults to `2048` bits.

docs/modules/secret-operator/pages/secretclass.adoc

@@ -20,5 +20,7 @@ include::example$truststore-tls.yaml[]
 This will create a ConfigMap (or `Secret` based on `targetKind`) named `truststore-pem` containing a `ca.crt` with the trust root certificates.
 It can then either be mounted into a Pod or retrieved and used from outside of Kubernetes.
 
+Expired or retired (see xref:secretclass.adoc#ca-rotation[Certificate Authority rotation]) certificates will not be published, because they should not be needed and some products, e.g. OpenSearch, have problems if they are present at startup.
+
 NOTE: Make sure to have a procedure for updating the retrieved certificates.
       The Secret Operator will automatically rotate the xref:secretclass.adoc#backend-autotls[autoTls] certificate authority as needed, but all trust roots will require some form of update occasionally.

docs/modules/secret-operator/pages/truststore.adoc

@@ -114,6 +114,14 @@ spec:
                               If `autoGenerate: true` then the Secret Operator will prepare a new CA certificate the old CA approaches expiration.
                               If `autoGenerate: false` then the Secret Operator will log a warning instead.
                             type: string
+                          caCertificateRetirementDuration:
+                            default: 1h
+                            description: |-
+                              Duration at the end of the CA certificate lifetime where no signed certificate will exist.
+
+                              Retired (or expired) CA certificates will not be published and will not be used for
+                              signing leaf certificates.
+                            type: string
                           keyGeneration:
                             default:
                               rsa:
@@ -165,6 +173,20 @@ spec:
                           In case consumers request a longer lifetime than allowed by this setting,
                           the lifetime will be the minimum of both, so this setting takes precedence.
                           The default value is 15 days.
+
+                          The maximum lifetime must be less than a quarter of the active CA certificate lifetime
+                          where the active CA certificate lifetime is  `ca.ca_certificate_lifetime -
+                          ca.ca_certificate_retirement_duration` to ensure that two subjects always have a common
+                          CA certificate in their trust stores – assuming that CAs are rotated at half of their
+                          active lifetimes.
+
+                          For instance, if a pod is created right before half of the active CA lifetime has
+                          passed, then it is signed by this CA but it does not know yet the new CA certificate
+                          which is created right afterwards. If another pod is created so that its certificate
+                          lifetime ends right after the first active CA lifetime then it is signed by the new CA.
+                          The `max_certificate_lifetime` must be chosen so that these two pods have no
+                          overlapping lifetimes, otherwise the first pod would see the second one signed by an
+                          unknown CA certificate. This can be achieved by the mentioned formula.
                         type: string
                     required:
                     - ca
@@ -519,6 +541,14 @@ spec:
                               If `autoGenerate: true` then the Secret Operator will prepare a new CA certificate the old CA approaches expiration.
                               If `autoGenerate: false` then the Secret Operator will log a warning instead.
                             type: string
+                          caCertificateRetirementDuration:
+                            default: 1h
+                            description: |-
+                              Duration at the end of the CA certificate lifetime where no signed certificate will exist.
+
+                              Retired (or expired) CA certificates will not be published and will not be used for
+                              signing leaf certificates.
+                            type: string
                           keyGeneration:
                             default:
                               rsa:
@@ -570,6 +600,20 @@ spec:
                           In case consumers request a longer lifetime than allowed by this setting,
                           the lifetime will be the minimum of both, so this setting takes precedence.
                           The default value is 15 days.
+
+                          The maximum lifetime must be less than a quarter of the active CA certificate lifetime
+                          where the active CA certificate lifetime is  `ca.ca_certificate_lifetime -
+                          ca.ca_certificate_retirement_duration` to ensure that two subjects always have a common
+                          CA certificate in their trust stores – assuming that CAs are rotated at half of their
+                          active lifetimes.
+
+                          For instance, if a pod is created right before half of the active CA lifetime has
+                          passed, then it is signed by this CA but it does not know yet the new CA certificate
+                          which is created right afterwards. If another pod is created so that its certificate
+                          lifetime ends right after the first active CA lifetime then it is signed by the new CA.
+                          The `max_certificate_lifetime` must be chosen so that these two pods have no
+                          overlapping lifetimes, otherwise the first pod would see the second one signed by an
+                          unknown CA certificate. This can be achieved by the mentioned formula.
                         type: string
                     required:
                     - ca