Add caching to gitlab-ci example

[ochorocho]

Caching the downloaded docker images on a project base would make sense to speed up builds.
Currently, all images are download on each and every job run.

[rfay]

Our experience with trying to cache images in GitHub hasn't been very successful, largely because just saving them or extracting them takes more time than downloading them in the first place.

[ayalon]

@rfay @ochorocho
I was thinking a lot about how we could cache the pipeline stuff.
For the moment, this is the biggest draw back of this approach.

I was thinking of a possible solution.

Is DDEV somehow capable of building and pushing a docker image to a registry?

If this would be possible, we could somehow check if the DDEV configuration has changed (maybe a salt.txt file or similar).
In the pipeline we start and build a DDEV container first and push the resulting docker image to the registry like that:
docker push $CI_REGISTRY_IMAGE:$DDEV_SALT

If the DDEV configuration did not change, we can just pull the image from the registry and run the ddev tests.

Or are there other ideas how to avoid the constant download of all docker layers within the container?

[rfay]

DDEV's docker images are all in the hub.docker.com registry. That's the problem here, we don't have a way to efficiently store the images locally. Downloading them doesn't take long, but unpacking/extracting does take a long time. So caching locally (which isn't hard) still takes a long time because the extraction into the local (fresh) docker instance does take time even if the download is fast.

[ayalon]

But isn't DDEV building a new docker image with docker compose?

Locally I can profit from the fact, that the layer are cached. I understand that downloading is as fast as geting them from the cache.

But I thought We could put the resulting image from docker compose which uses the DDEV docker images as a base could be pushed to a registry and reused.

Or is there another mechanism possibly speed up the process? Instead of running Docker-in-Docker, we build the DDEV final docker image, then push it to gitlab registry and in the next step we run the tests inside this prebuild DDEV image.

Maybe this is also the wrong approach. Do you have another idea? We are using kaniko to build and push docker images and I was imaging that this could be a way to save some resources in the pipeline.

[rfay]

DDEV adds a new layer to the image at start, but the key problem with testing is the (download and) extraction of the base image, especially ddev/ddev-webserver. The addition of the extra layers (for username, etc.) takes nearly no time. You can watch it yourself on a ddev start. The dots show the build time.

IMO the general problem is figuring out how to have the actual docker server persist state, so that images are ready there when they're needed.

[ayalon]

Ok I understand. You are absolutely right. Unfortunatly I lack the knowledge how to do that. I'm not even sure that it is technically possible.

But I agree that if the base image would be available, the process would be very fast.

[ochorocho]

If the images are stored in ddev itself and not in the DIND-Service image we could certainly extend the image to contain the images.
Just dunno if these images vary depending on the service version in use.

I'd like to keep this image small if possible.

ddev/ddev#4479

[rfay]

You point to

• ddev debug download-images shouldn't require a project ddev#4479

for some reason... but that was fixed ages ago so ddev debug download-images does not require a project.

[ochorocho]

👍 my intention was to use ddev debug download-images to download the images and ship this image with the images pre-downloaded.

[rfay]

Will be great to see how that comes out!

[ayalon]

Great idea! That make things faster for sure!

[AkibaAT]

I've had pretty decent success improving the pipeline duration by building the required images only once, and pushing them to the registry for reuse, as well as using the GitLab dependency proxy to pull in external images.

I've also tried caching the images as local tar files, but there was no major additional time saving there compared to the dependency proxy.

[ayalon]

Could you share how you have done this? I'm highly interested.

[AkibaAT]

I've created a gist with a simplified recap: https://gist.github.com/AkibaAT/06ff24091055388a62813ff60368176b

The basic process is:

• ddev step creates a checksum of all files involved in creating the built ddev images, per default that's the web and the db services.
  It checks in the GitLab registry if the image is already built, and if so, skips the build, and if not, builds the images and then pushes them to the registry.
  It also create 2 override YAML files for docker compose. One to change the image reference and remove the build instructions, and the other to make the web service startable independently (for simple tasks like npm & composer install that don't need the full DDEV).
• Because the ddev gitlab ci image is based on the docker dind image, I start dockerd directly within it, instead of using an extra dind service.
• Since we store the override YAML as artifact of the .ddev directory, ddev automatically includes and uses it.
  ddev currently always tries to docker compose build, but since we removed the build config via the override, that step finishes immediately.
• To make use of the GitLab dependency proxy, gitlab-preload-images script is called, which prefixes all images found in the generated compose-full yaml and downloads them via the proxy.
  Since ddev expects them without a prefix, we also retag them to remove the proxy prefix.
• The second override YAML is stored outside the .ddev directory as we don't want ddev to include it automatically. It can instead be used to start the web service via docker compose directly.

[AkibaAT]

👍 my intention was to use ddev debug download-images to download the images and ship this image with the images pre-downloaded.

I've realized this idea via a GitHub Workflow now in my fork: https://github.com/AkibaAT/ddev-gitlab-ci/pkgs/container/ddev-gitlab-ci

It doesn't run download-images, because that also pulls in mariadb. Instead, I just pulled in all the images that are always required:

• busybox:latest
• ddev/ddev-ssh-agent:${DDEV_VERSION}
• ddev/ddev-traefik-router:${DDEV_VERSION}
• ddev/ddev-utilities:latest
• ddev/ddev-webserver:${DDEV_VERSION}

I've also adapted the image to start dockerd itself in the entrypoint, instead of having to start an extra dind service. Just run the image, and you're good to go to use ddev.

The GitHub Workflow itself is adapted as well. Instead of using QEMU, I switched the workflow to use the native ARM64 runner that's become available for public use a few months ago.