You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Fix aarch64/arm64 Linux wheel build failure due to missing libatomic linking
Problem
The tokenizers library fails to build on aarch64 Linux systems during CI with:
/opt/rh/gcc-toolset-13/root/usr/libexec/gcc/aarch64-redhat-linux/13/ld: cannot find -latomic: No such file or directory
Root Cause
- On aarch64, certain atomic operations require explicit linking with libatomic
- The sentencepiece dependency detects libatomic (Found atomic: /usr/lib64/libatomic.so.1) but incorrectly adds just the string "atomic" to link flags instead of the actual library path
- This causes the linker to fail when building the Python extension
Solution
Added proper libatomic detection and linking for aarch64/arm64 systems in two places:
1. For the tokenizers static library
2. For the pytorch_tokenizers_cpp Python extension module
The fix uses CMake's find_library to locate libatomic and explicitly links it when building on aarch64/arm64 architectures.
0 commit comments