Utilizing docker build cache

[orthodoX]

Why?

Every single time our deploy task is ran in CI we are performing a docker build from scratch and wait 3+ minutes for every deployment task. Just as we do for development, we can use Docker's build cache to drastically speed up this process by reusing unchanged layers from previously built images. The test I ran on staging completed a deploy in 13s!

More context

For multistage Dockerfile definitions that we do for production, this requires registry cache strategy to be used, where cache artifacts are saved independently from the Docker image used to deploy the apps (which is a win win for us). A requirement for registry strategy is docker buildx driver instead of the default docker build.

The caching strategy implemented in this PR looks like this:

• For the main branch, import main cache only and export main cache only
• For other branches: import main cache + branch cache, export branch cache only - this multi import allows us to still reuse most of the cache (if not all) for freshly created branches which in the first CI run still do not have branch specific cache of their own

When exporting cache the mode=max param is passed, which yields the maximum cache hit ratio for layers. The size of these cache artifacts are ~700MB per file. We'll generate 1 file for main branch and 1 for every other branch. We can experiment with other modes, like mode=min to see the file size and cache hit ratio, if we think that size is a concern.

After a successful build when new cache artifact get pushed, the previous artifact for that branch automatically becomes stale and untagged in the ECR. And since these stale/untagged files are of no use to us, we'll prune them at the end of every build. Here's how these stale artifacts appear in ECR, just as a - 👇🏻

Conclusion 🚀

By paying some price in ECR storage we can get almost instantaneous deploys, well worth trade imo. And with better defined ECR cleaning policies, we can free up space regularly by removing branches and cache artifacts every N amount of days that we agree on.

[epaune-alt]

LGTM, but bash scripts and docker are not my strong points. Aleks and Adrian could be more helpful here. And maybe Alex E?