Updating to latest faiss?

[jondot]

I've forked @SkyFan2002 's static build, and in the effort to simplify, I've also updated to latest Faiss:
https://github.com/jondot/faiss-rs
Compiling on Mac M1 and running vector search and indexing works well.

As now we have static build in this repo, I'm wondering if its possible to point to a recent Faiss version (and not a specific C_api flavor), or there's something hidden from my knowledge that forbids that?

[Enet4]

Updating the inner Faiss to v1.7.4 seems like a good plan for v0.13.0. If you would like to send in a PR, the steps would be:

1. Update the scripts in faiss-sys to point to v1.7.4
2. Run gen_bindings.sh (may need tweaking depending on the version of bindgen used)
3. Include changes to bindings.rs and bindings_gpu.rs
4. Adjust code at faiss if necessary

[jondot]

on the fork up top i just updated the git submodule without running gen_bindings or any source updates.
having said that, I selfishly built for my Mac M1 only, you're saying that if I attempt building for other platforms I might need these kind of tweaks?

[Enet4]

I'm not sure if I understood you correctly. Do you want to use a custom version of the Faiss library? That makes things a bit trickier. You will either need to depend on dynamic linking (assuming the C API is left working and compatible), or bring those changes upstream to Faiss so that they are released officially in a future version.

The changes I mentioned above are necessary to target a new version of Faiss, regardless of platform. Since Faiss does not follow semver, the crate is intended to always target a specific version of the library, or a version with a compatible C API.

[jondot]

What I'm saying is that I went ahead and updated my personal fork to this https://github.com/facebookresearch/faiss/commits/v1.7.4

And without running any binding gen steps, building statically, everything worked.

So i'm wondering -- did everything work only because I built for Mac? or if I tried building for other platforms, then I would need to do gen_bindings etc.

Hope that's clearer?

[Enet4]

I mean, it might have worked because of the range of API compatibility between those versions, but that may have been purely coincidental. In addition, new parts of the API are not exposed by faiss-sys unless the bindings are regenerated. So a complete Faiss upgrade should still go through the full process.

[jondot]

Got it, thanks!

[web3creator]

I've forked @SkyFan2002 's static build, and in the effort to simplify, I've also updated to latest Faiss: https://github.com/jondot/faiss-rs Compiling on Mac M1 and running vector search and indexing works well.

As now we have static build in this repo, I'm wondering if its possible to point to a recent Faiss version (and not a specific C_api flavor), or there's something hidden from my knowledge that forbids that?

@jondot hello，I hava a bug on mac m1

  = note: ld: library not found for -lgomp

do you know what's going on?

I hava installed libopm
(base) simonyi@simondeMacBook-Pro-2 faiss-rs %  brew info libomp
==> libomp: stable 16.0.6 (bottled) [keg-only]
LLVM's OpenMP runtime library
https://openmp.llvm.org/
/opt/homebrew/Cellar/libomp/16.0.6 (7 files, 1.7MB)
  Poured from bottle using the formulae.brew.sh API on 2023-09-05 at 13:57:14
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/libomp.rb
License: MIT
==> Dependencies
Build: cmake ✔, lit ✔
==> Caveats
libomp is keg-only, which means it was not symlinked into /opt/homebrew,
because it can override GCC headers and result in broken builds.

For compilers to find libomp you may need to set:
  export LDFLAGS="-L/opt/homebrew/opt/libomp/lib"
  export CPPFLAGS="-I/opt/homebrew/opt/libomp/include"
==> Analytics
install: 19,756 (30 days), 80,619 (90 days), 231,651 (365 days)
install-on-request: 3,859 (30 days), 16,318 (90 days), 46,744 (365 days)
build-error: 2 (30 days)

[jondot]

yes, on mac, we need omp and not gomp, and after omp is specified in build.rs, then you need to have libomp on your computer.

[jondot]

@Enet4 I haven't yet moved back to your repo, so these are the relevant lines:
https://github.com/jondot/faiss-rs/blob/master/faiss-sys/build.rs#L39-L43

[aalekhpatel07]

Wondering if there's interest in updating to the latest faiss (v1.12.0) and expose the binary indexes from the C API as well?

I've put together some work with passing CI (on my private mirror) to that end if you'd like to review.

It would be nice if I didn't have to maintain a mirror just for the binary index support, and I thought this could be beneficial to other users too so I've decided on contributing.

[Enet4]

@aalekhpatel07 Thank you for reaching out. Yes, there is interest in keeping faiss-rs updated. I just need to sort out which pending pull requests should be included before the next release. Would you be willing to recreate the necessary PRs onto this repository?

[aalekhpatel07]

For sure! I'd be happy to contribute. Let me know if there's a specific order of PRs/work you have in mind otherwise I can begin by getting bindings ready for v.1.12.0 and basic plumbing for binary indexes.

[Enet4]

Your assistance is much appreciated. I have just now merged #91, so any new PRs should be based on top of this. I am unsure about the other PRs because they are rather platform specific and makes me wonder whether we can offer more control over the building process through Cargo features or environment variables.

[aalekhpatel07]

Awesome! Thanks for getting started with this. I rebased my changes onto the current master and #99 seems ready for some feedback.