Summary:
Remove the dependency on `raft::compiled` and modify GPU implementations to use cuVS backend in place of RAFT.
A deeper insight into the dependency:
FAISS gets the ANN algorithm implementations such as IVF-Flat and IVF-PQ from cuVS. RAFT is meant to be a lightweight C++ header-only template library that cuVS relies on for the more fundamental / low-level utilities. Some examples of these are RAFT's device mdarray and mdspan objects; the RAFT resource object (`raft::resource`) that takes care of the stream ordering of device functions; linear algebra functions such as mapping, reduction, BLAS routines etc. A lot of the cuVS functions take the RAFT mdspan objects as arguments (for example `raft::device_matrix_view`). Therefore FAISS relies on both cuVS and RAFT. FAISS gets RAFT headers through cuVS and uses them to create the function arguments that can be consumed by cuVS. Note that we are not explicitly linking FAISS against `raft::raft` or `raft::compiled`. Only the required headers are included and compiled rather than compiling the whole RAFT shared library. This is the reason we still see mentions of `raft` in FAISS.
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3549
Reviewed By: ramilbakhshyiev
Differential Revision: D62041013
Pulled By: asadoughi
fbshipit-source-id: 7230dcc06cf47baf95873adc1dec2adca4a8f82a
Summary:
Sometimes between Sept 25 to Oct 2, downloading and linking against `openblas=*=*openmp*` package to run tests have caused a 4-7x slow down. Link it with the regular openblas package which is not compiled with `USE_OPENMP=1`. We will set the openblas omp threads via the environment variable `OPENBLAS_NUM_THREADS` according to https://github.com/OpenMathLib/OpenBLAS/wiki/Faq#multi-threaded
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3918
Test Plan: SVE CI should finish within 40 minutes
Reviewed By: ramilbakhshyiev
Differential Revision: D64059860
Pulled By: mengdilin
fbshipit-source-id: 3ba2bda5fce5122f051421f459692f15ad5360a4
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3747
This change converts the ROCm build to run inside containers and updates it to run on AMD GPU based runners. Still working with the AMD team to resolve test failures before enabled those.
Differential Revision: D61049115
fbshipit-source-id: 28274e0bde795f99b3d78711beaf9b3ed3c5e66c
Summary:
The TestPartitioning.TestPartitioningBigRange test case fails on gcc version 13.2. We can avoid this by requiring gcc version 11.2 where the test case works.
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3655
Test Plan: Check github workflows and test branches
Reviewed By: ramilbakhshyiev
Differential Revision: D59988036
Pulled By: gtwang01
fbshipit-source-id: ae6d7f7888c9d7a2e59f557e05dbd4f318983668
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3717
they have been running in shadow mode for quite some time now.
I spot-checked the builds for the past 10 jobs and they all look good. Since `continue-on-error` will always mark a job as "green" even if it fails, I need a way to holistically verify these builds actually work reliably. Turning the builds to blocking to accomplish that.
Reviewed By: ramilbakhshyiev
Differential Revision: D60692521
fbshipit-source-id: 172a6362c672b0376c76559f12852110936756df
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3703
Now that SVE PR has been merged, we can turn on SVE opt mode in CI
Reviewed By: ramilbakhshyiev
Differential Revision: D60457456
fbshipit-source-id: 053b1f8ac805afba9035095c5df811da05675a81
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3701
Gating ARM SVE behind the plain vanilla linux x64 build like all the other builds.
Reviewed By: mengdilin
Differential Revision: D60425535
fbshipit-source-id: f2e082fbaa6ea1e314ffe0e2e7260c8634cab989
Summary:
Add instructions to download arm64 specific conda dependencies and cmake command and run it on CI. This should prepare us to turn on CI with SVE optimization
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3653
Reviewed By: ramilbakhshyiev
Differential Revision: D60043435
Pulled By: mengdilin
fbshipit-source-id: d81bb1c1022681c3da8f98bbf080d5e1d65d6b80
Summary:
With the refactoring diff landed in [D59650573](https://www.internalfb.com/diff/D59650573), we take in the instance type string for aws and register it as the label for the runner. There was a typo in the previous registration (`m7i.xlarge` instead of `m7i.large`). The refactoring diff fixed the typo from the runner registration side but we need to fix it on the CI side as well so CI can find the runners
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3649
Test Plan: CI's AVX512 run on Github Action should succeed instead of stuck in pending
Reviewed By: ramilbakhshyiev
Differential Revision: D59916841
Pulled By: mengdilin
fbshipit-source-id: 5403e1a449ed765f0bf1a00e4a0d81d6c97463b9
Summary:
Start beta-testing AVX512 Build on self-hosted github runner with label faiss-aws-m7i.xlarge. This build is non-blocking on the PRs right now (via the parameter `continue-on-error`)
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3602
Reviewed By: ramilbakhshyiev
Differential Revision: D59290555
Pulled By: mengdilin
fbshipit-source-id: 6a82538bd44fecf4913885ac3b1999eb0fa52047
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3476
The long tail will still be the RAFT build but we can save on cost if the build has errors by incurring ~10m penalty added by waiting for the basic cmake build to complete. Both GPU and RAFT builds will start together so this will take less time overall to complete.
Reviewed By: algoriddle
Differential Revision: D57707298
fbshipit-source-id: 3589842e9bda9ebca9b25e089e6177fe96b6a0f5
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3466
Flattening Conda include directories breaks CUDA 11.4.4 build on Ubuntu 20 / v5 kernel. This change updates the logic to only flatten includes on Ubuntu 22 / v6 kernel (aka as running on GitHub Actions runners).
Reviewed By: algoriddle
Differential Revision: D57602154
fbshipit-source-id: 00c14ca7c64644b8b86483ac6b4d40c6d8f12372
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3428
GitHub Actions currently does not support runners with AVX-512 but committed to add this support in early 2025. We will be running these on CircleCI until then. This placeholder build configuration will allow us to enable it with a 1-liner when the hosts are available.
Reviewed By: algoriddle
Differential Revision: D57261783
fbshipit-source-id: 1fb985a0c3dbb11851af63c95bde6494d25d0ac2
Summary:
This commit is the first in a series in an attempt to incrementally enable all jobs currenlty performed by CircleCI. It includes the main configuration files provided by GitHub team + 1 build.
Original PR: https://github.com/facebookresearch/faiss/pull/3325
Reviewed By: junjieqi
Differential Revision: D56671582
fbshipit-source-id: c8a21cd69aabaf86134eb86753e90b1facf51bc3
