Skip to content

Conversation

@otargowski
Copy link
Contributor

@otargowski otargowski commented Oct 10, 2025

This PR decreases both the docker image size and its build time by:

  • Not including the downloaded sandboxes twice in the final image by using an additional build stage.
  • Purging texlive documentation, since Debian doesn't always put it in separate packages.
  • Using no-install-recommends with apt install (which alone saves ~1GB), as up to now e.g. mupdf (a full-fledged pdf viewer) was included in the image. The only needed package installed this way seems to be tex-gyre.
  • Switching to python:3.11-slim as the base image. This required explicitly installing make, g++ and libc6-dev.
  • Replacing pip with uv pip, which is almost a drop-in replacement and provides a 155 -> 8s speedup for the requirements.txt installation step on my modern cpu. On a i7 2640M from 2011 it takes 15s. The only issue was the lack of --user support, which was handled by using .local/ as a venv.
  • Parallelizing upload_sandboxes_to_filetracker, which is cpu-bound on the client size while compressing the already compressed sandboxes.
  • Starting only filetracker for uploading the sandboxes to it, because mailnotifyd and rankingsd consume noticeable cpu while probably trying to access the non-existent DB.

The resulting development image size decrease is 7.08 -> 3.3 3.14 GB.

The image could be shrunk further by removing e.g. old gcc sandboxes from the Manifest file, which might simply work since existing installations should already have all old sandboxes on filetracker. Alternatively, a separate Manifest file could be maintained.
Regardless, that is out of scope for this PR.

@otargowski otargowski marked this pull request as draft October 11, 2025 15:26
@otargowski otargowski force-pushed the docker-optimizations branch from a8af619 to 2cebd35 Compare October 11, 2025 20:25
@otargowski otargowski marked this pull request as ready for review October 11, 2025 20:25
@Iteron-dev
Copy link
Member

I think that if we’re using uv in the Dockerfile, we should also use it in the workflows (e.g. in pytest.yml and nightly.yml

Besides that, it would be good to update the various README files in the repo that mention pip, to also mention the option of using uv.

@otargowski otargowski force-pushed the docker-optimizations branch from 2cebd35 to fe11922 Compare October 18, 2025 12:36
@otargowski otargowski marked this pull request as draft October 18, 2025 20:52
Switch to the *-slim docker image as the base, don't include sandboxes
twice in the final image and pass --no-install-recommends to apt.
@otargowski otargowski marked this pull request as ready for review November 10, 2025 20:16
@otargowski
Copy link
Contributor Author

I updated the PR to accomodate s3dedup. The uv migration will happen in another PR.
Tests fail on master too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants