feat: Adding support for extraContainers and StatefuleSet deployment on v2.2.2

[Nicolas-Peiffer]

This cancels and replace a previous PR 87.

The purpose of these modifications is to add support for deploying the registry container as a StatefuleSet STS instead of a Deployment. This need several modifications on the templates, because the STS handles volume differently than the Deployment. A source of inspiration for this modification is the Grafana Helm Chart https://github.com/grafana/helm-charts/tree/main/charts/grafana.

And also adding support for extraContainers to be able to add sidecar containers to the registry container to support calling OCI syncing tools like dregsy or skopeo to populate the OCI registry.

# values.yaml for the `twuni/docker-registry.helm` helm chart
[...]
#extraContainers: []
# Additional containers to be added
extraContainers:
  ## Side car container for registry synchronization with dregsy and skopeo
  - name: dregsy
    image: docker.io/xelalex/dregsy
    imagePullPolicy: IfNotPresent
    command: ['dregsy', '-config=/home/dregsy/config/dregsy-config.yaml']
    resources:
      requests:
        cpu: 10m
        memory: 32Mi
    securityContext:
      allowPrivilegeEscalation: false
      privileged: false
      runAsNonRoot: true
      capabilities:
        drop:
        - all
    volumeMounts:
      - name: dregsy-config
        mountPath: /home/dregsy/config/
        readOnly: true
[...]

Signed-off-by: Nicolas-Peiffer [email protected]

Release Notes

• Adding support for deploying the container registry using a Kubernetes StatefuleSet manifest object with dedicated per-sts PVC. Compatible with podAntiAffinity.
• Adding support for extraContainers to support calling OCI syncing tools like dregsy or skopeo to populate the OCI registry.

[iDmple]

I tested this and it works well. When can this be merged?

[pracskai]

Would love to see this merged soon. Is there anything blocking the review or merge process?

[joshsizer]

From my understanding, deploying a statefulset for docker registry with more than one replica and more than one PVC would cause docker registry to function improperly because there's no notion of leader / follower or replication between members. Because of this restriction, a single replica stateful and a single replica deployment with a PVC deployed separately functionally achieves the same results. The ability to deploy a multi-replica statefulset would be misleading, whereas you can increase the deployment replicas, provided you are using an external storage provider.

@Nicolas-Peiffer @pracskai Would you be able to provide some additional context around the reason you need to deploy this as a statefulset?

[pracskai]

Our use case is to run the Docker registry in proxy (pull-through cache) mode to reduce traffic for image pulls. The goal is to run multiple replicas for high availability, but without relying on shared RWX storage, which can be operationally challenging or unavailable in some Kubernetes environments.

By using a StatefulSet with separate PVCs (RWO) per replica, each registry instance maintains its own local cache. This setup is acceptable because:

• The registry is read-through from the remote (e.g., Docker Hub), so there's no strict need for data replication between instances.

• We accept eventual inconsistency in cache contents across replicas. The worst case is a cache miss, which simply results in a pull from the remote source.

• We avoid the complexity and fragility of RWX volumes while still getting basic HA via independent pods with independent caches.

So, while a multi-replica statefulset may not suit a write-heavy or push-mode registry, it works well for proxy caching, especially when consistency isn’t critical and storage availability is limited to RWO PVCs.

[Nicolas-Peiffer]

Note: my use case shared approximately the same needs than @pracskai from its #89 (comment).

Note: When I say OCI registry in my message, I am talking about the distribution project which ships the distribution container image available here docker.io/registry or here ghcr.io/distribution/distribution. And of course by OCI I mean Open Container Initiative.

This OCI registry project is also know by its older original name "docker registry" which I do not use anymore because the name docker create confusion.

Hello 👋 , at the time in 2022 (and since this then, I moved to other projects), I had a small footprint kubernetes cluster use case (1 cluster of 2 nodes) with the need for redundancies (availability) for the services deployed in each k8s nodes.

Therefore, I needed to be able to deploy the exact same services on each of the 2 nodes, including the OCI registry service. And I was populating both OCI registries with tools likes dregsy and skopeo. I made sure the content of OCI registries was the same thanks to dregsy and skopeo.

Note: dregsy and skopeo were not perfect though and syncing the content of both OCI registries was a challenge, but with a little more work on the dregsy tool or some other tool, I could have achieve better results.

However in this 2 nodes situation for my use case, one could argue that it would be simpler to have 2 clusters of 1 node instead of having 1 cluster of 2 nodes. There are pros and cons:

• With the 1 cluster of 2 nodes approach, a StatefuleSet deployment makes sense if you want to have the same services on each node. And if I add more than 2 nodes, all my node will have my services. Having a dedicated PVC volume on each node allow for a better redundancies (availability) in case one node fails.
• With the 2 clusters of 1 node approach, I follow you @joshsizer this woulds have been okay to use a Deployment and affinity policies instead of a StatefuleSet .

I know we can configure replicaSet, affinity, nodeAffinity , nodeSelector, etc... to deploy on several different k8s nodes. This would fit a situation where we have 10 k8s nodes, but we want to deploy the OCI regitries on 3 different nodes.

But this was not my use case. So why should I bother configuring an affinity policy if I can use a StatefuleSet that would implement by design the right affinity policy I needed?

I hope this helps 😸

[Nicolas-Peiffer]

By the way: I added support for extraContainers because I ran dregsy as a side car container that was periodically syncing the content of the OCI registry instances in my cluster.

I added a couple of explanation in the PR main first comment.

Maybe I should have done 2 different PR since the StatefuleSet feature is different than the extraContainers feature. Feel free to separate my PR into 2. I will not do it myself.

[joshsizer]

@Nicolas-Peiffer @pracskai Thanks so much for the context, the pull-through cache and manual replication strategies are some use cases I hadn't considered. I would love to get this PR through. I need to spend some more time reviewing and I think I will have some feedback.

@Nicolas-Peiffer I'm okay with keeping the extraContainers in this PR. If I have any suggestions or feedback, would you be able to make those changes? If I were to push any commits to this branch, I would no longer be able to approve this PR due to the repo rules.

Again, thanks for the context and for such quick responses as well, it really helps get things moving.

[Nicolas-Peiffer]

I can do the changes if this means changing 1 or 2 lines of Helm template.
I can render the k8s manifests with helm template --debug if you provide a values.yaml sample, but I would prefer not to do the tests myself on a real kubernetes cluster (even if this is probably easy to do on something like Kubernetes KinD...).
I can tell you 3 years ago it worked 😄 on my fork.

For extraContainers, there was a duplicate PR #140. We should close it once the merge(s) are completed.

[joshsizer]

I tried simplifying the logic here, let me know what you think. One thing that still needs work is the garbage collection cronjob - if that's enabled and stateful set is enabled the cronjob doesn't mount the PVC

[Nicolas-Peiffer]

Hey, I had a look at your suggestions. Unfortunately, I was not able to test them, but it looks okay. I commented under each of your remark.