AZP: Switch GPU tests to Ubuntu 24.04 DOCA 3.1 (GPUNetIO) image #10861
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Why: previous RHEL9 image lacked doca-gpunetio and headers Signed-off-by: Alexey Rivkin <[email protected]>
yosefe
approved these changes
Sep 8, 2025
dpressle
previously approved these changes
Sep 9, 2025
yosefe
requested changes
Sep 9, 2025
There was a problem hiding this comment.
This container cannot build UCX device API:
- doca verbs library is missing
- it has cuda13 while the doca gpunetio requires cuda12
configure:30577: gcc -o conftest -I/opt/mellanox/doca/include -L/opt/mellanox/doca/lib -Wl,-rpath-link,/opt/mellanox/doca/lib -L/opt/mellanox/doca/lib/x86_64-linux-gnu -Wl,-rpath-link,/opt/mellanox/doca/lib/x86_64-linux-gnu -Wl,-rpath-link, conftest.c -ldoca_gpunetio -ldoca_gpunetio -lrt -lrt >&5
/usr/bin/ld: warning: libdoca_verbs.so.2, needed by /opt/mellanox/doca/lib/x86_64-linux-gnu/libdoca_gpunetio.so, not found (try using -rpath or -rpath-link)
/usr/bin/ld: warning: libcuda.so.1, needed by /opt/mellanox/doca/lib/x86_64-linux-gnu/libdoca_gpunetio.so, not found (try using -rpath or -rpath-link)
/usr/bin/ld: /opt/mellanox/doca/lib/x86_64-linux-gnu/libdoca_gpunetio.so: undefined reference to `cuDeviceGet'
/usr/bin/ld: /opt/mellanox/doca/lib/x86_64-linux-gnu/libdoca_gpunetio.so: undefined reference to `doca_verbs_cq_get_dbr_addr@EXPERIMENTAL'
Merged
Alexey-Rivkin
added a commit
to Alexey-Rivkin/ucx
that referenced
this pull request
Sep 9, 2025
Why: GPU CI builds now require GPUNetIO; silently skipping it can pass CI and fail at runtime. Refs: openucx#10861 openucx#10865 Signed-off-by: Alexey Rivkin <[email protected]>
Why: GPU CI builds now require GPUNetIO; silently skipping it can pass CI and fail at runtime. Refs: openucx#10861 openucx#10865 Signed-off-by: Alexey Rivkin <[email protected]>
Alexey-Rivkin
force-pushed
the
doca3.1_gpunetio
branch
from
e0254d5 to
b5b58be
Compare
- az-helpers.sh: try_load_cuda_env() sets have_cuda=yes when nvcc is on PATH. Falls back to loading the CUDA module only if nvcc is absent. Why: CUDA-enabled builder images already include the correct toolchain. Preferring local CUDA avoids module mismatches and supports DOCA GPUNetIO builds. Signed-off-by: Alexey Rivkin <[email protected]>
Why: GPU CI failed Refs: openucx#10861 openucx#10865 Signed-off-by: Alexey Rivkin <[email protected]>
|
move the changes to #10865 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What?
Switch GPU tests to use Ubuntu 24.04 DOCA 3.1 image that includes GPUNetIO runtime and headers.
Follow-up to #10849
Why?
The previous RHEL 9 image lacked doca-gpunetio and its headers, blocking GPUNetIO-enabled tests.