Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3516
This diff seeks to simplify the steps that install conda packages and environment configuration into a single step at the start of the cmake build action.
Reviewed By: mnorris11
Differential Revision: D58560454
fbshipit-source-id: ee2c6b36865809f31eb335cfb3c2fffdccaa318d
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3512
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3510
GitHub hosted runners some with the build-essentials package pre-installed, self-hosted runners on AWS do not have this package. This made it all steps other than the `all targets` one fall back to the system executables which unintentially worked on GitHub hosted runners but not on the self-hosted ones. This diff fixes it by pulling the line that adds conda bin to path early in the cmake build action.
Reviewed By: asadoughi
Differential Revision: D58513853
fbshipit-source-id: 23e95459e0031c96bd142515db07d1b700d713cf
Summary:
Adding build folder to gitignore, so that they don't show up in the commit tree while building from source
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3492
Reviewed By: junjieqi
Differential Revision: D58171359
Pulled By: asadoughi
fbshipit-source-id: b0efed348769328a3bdbcc13098dcb84cadb6c4f
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3494
This tasks focus on the refine index construction tutorial with different index refinement on fp16/sq8 quantization. The python version was added a while ago.
Reviewed By: junjieqi
Differential Revision: D58161983
fbshipit-source-id: 1c598fe612b5dee3952c5f7398e6802e117f141d
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3487
Sometimes it is not useful to serialize the storage index along with a HNSW index. This diff adds a flag that supports skipping the storage of the index.
Searchign and adding to the index is not possible until a storage index is added back in.
Reviewed By: junjieqi
Differential Revision: D57911060
fbshipit-source-id: 5a4ceee4a8f53f6f746df59af3942b813a99c14f
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3423
Adding small fixes to run experiments from fbcode.
1. Added buck target
2. Full import path of faiss bench_fw modules
3. new dataset path to run tests locally as we can't use an existing directory ./data in fbcode.
Reviewed By: algoriddle, junjieqi
Differential Revision: D57235092
fbshipit-source-id: f78a23199e619b640a19ca37f8b52ff0abdd8298
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3422
Found vec_io failing when running some benchmarking.
There is no such field named big_endian in sys. So, reverting it to original field byteorder
Reviewed By: algoriddle
Differential Revision: D56718607
fbshipit-source-id: 553f1d2d6bc967581142a92282e534f3f164e8f9
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3480
This tutorial summarize the methods to construct different indexs for PQFastScan refinement. It shows how the choice can impact on accuracy.
Reviewed By: junjieqi
Differential Revision: D57799598
fbshipit-source-id: a75c52c60a5217366f3361676da8f03f0c4a9feb
Summary:
Small Raft related modification to StandardGpuResources:
if the stream for a particular device is modified by a user, delete the Raft handle for that device. On any subsequent call to `getRaftHandle(device)`, a new raft handle with the updated stream will be created.
Closes https://github.com/facebookresearch/faiss/issues/3424
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3435
Reviewed By: ramilbakhshyiev
Differential Revision: D57640976
Pulled By: junjieqi
fbshipit-source-id: 41e2898a39250b7e52e920b71e819fc21ca9fc85
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3479
`-Wextra-semi` or `-Wextra-semi-stmt`
If the code compiles, this is safe to land.
Reviewed By: palmje
Differential Revision: D57632759
fbshipit-source-id: 48bc23e87b3f518182085124c4c8e68ddbb3ca8f
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3476
The long tail will still be the RAFT build but we can save on cost if the build has errors by incurring ~10m penalty added by waiting for the basic cmake build to complete. Both GPU and RAFT builds will start together so this will take less time overall to complete.
Reviewed By: algoriddle
Differential Revision: D57707298
fbshipit-source-id: 3589842e9bda9ebca9b25e089e6177fe96b6a0f5
Summary:
mdouze Please let me know if any additional unit tests are needed
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3444
Reviewed By: algoriddle
Differential Revision: D57665641
Pulled By: mdouze
fbshipit-source-id: 9bec91306a1c31ea4f1f1d726c9d60ac6415fdfc
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3474
This commit focus on the cpp version of PQfastscan tutorial with index refinement by defining the k factor.
Reviewed By: junjieqi
Differential Revision: D57680905
fbshipit-source-id: 980c2990172f24ec9a4f870685e706195883408f
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3473
Previous diff (D57602154) fixed the CircleCI version and the PR build version of GHA but not the nightly one.
Reviewed By: junjieqi
Differential Revision: D57680576
fbshipit-source-id: 39f49c20df824c915f536b1ed3ffc35db2907988
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3470
Hour and minute values are swapped, the goal is to run is at 1:10am UTC.
Reviewed By: algoriddle
Differential Revision: D57654059
fbshipit-source-id: 23bcb42e5c95f731cd4713ad4691d0f475ed8ad2
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3468
This commit includes the tutorial for PQFastScan in the cpp environment.
Reviewed By: junjieqi
Differential Revision: D57631441
fbshipit-source-id: f5e17eee2a584ebfc9ff63868d741d0da6b3b413
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3465
This commit include python version of tutorial for FastScan. It includes all the parameters enabled within PQFastScan.
Reviewed By: junjieqi
Differential Revision: D57594044
fbshipit-source-id: cb12679b6fc241a654b9545c5bc7bd0517aa1813
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3467
1. Cron is scheduled to execute at 1:10am UTC. This is per GitHub's recommendation to avoid hotspots. The docs mention that when GH backend gets overloaded, they can drop scheduled jobs which we want to avoid so we scheduled off hour, off midnight.
2. The plan is to let these nightlies run once and, if successful, then disable them in GitHub UI to perform validation. Also disable if things break and need to be fixed of course.
Reviewed By: algoriddle
Differential Revision: D57602833
fbshipit-source-id: 4f4d9abbaa5ed3d1edb024ea4dd3f87aa78dd9b5
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3466
Flattening Conda include directories breaks CUDA 11.4.4 build on Ubuntu 20 / v5 kernel. This change updates the logic to only flatten includes on Ubuntu 22 / v6 kernel (aka as running on GitHub Actions runners).
Reviewed By: algoriddle
Differential Revision: D57602154
fbshipit-source-id: 00c14ca7c64644b8b86483ac6b4d40c6d8f12372
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3459
When building with CUDA 11.4.4, CMake does not properly include files under Conda environment. This workaround flattens the include sub-directories in to the include root. It will unblock us for now while we are looking for a fix through CMakeLists files or figure out if it's a CMake bug and it gets fixed.
Reviewed By: junjieqi
Differential Revision: D57545169
fbshipit-source-id: 9cbdd0866e00e899cc889930a59448da55d873c2
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3455
Code quality control by reducing the number of prints
Reviewed By: junjieqi
Differential Revision: D57502194
fbshipit-source-id: a6cd65ed4cc49590ce73d2978d41b640b5259c17
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3452
Delete all remaining print within the Tests to improve the readability and effectiveness of the codebase.
Reviewed By: junjieqi
Differential Revision: D57466393
fbshipit-source-id: 6ebd66ae2e769894d810d4ba7a5f69fc865b797d
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3451
This is a first step to clean up the faiss codebase following T187322081
Reviewed By: junjieqi
Differential Revision: D57448335
fbshipit-source-id: c9760d01479d3352b786bbcf2015251e7a7168d6
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3414https://github.com/facebookresearch/faiss/issues/3355
A couple open questions:
- Given L2 was squared, I figured I would leave this one as squared as well?
- Also, wasn't sure if we wanted to return nan when present == 0 or -1?
Reviewed By: mdouze
Differential Revision: D57017608
fbshipit-source-id: ba14458b92c8b055f3bf2a871565175935c8333a
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3440
This change is required to unblock the migration to GitHub Actions. `cuda-toolkit` was only specified in the `libfaiss` package and it was not available in `faiss-gpu` or `faiss-gpu-raft`. This currently works on CircleCI because the runner image has CUDA toolkit of the needed version installed on the system and the build logic falls back to that but breaks on GitHub Actions because their runner images do not come with CUDA toolkit pre-installed.
Reviewed By: junjieqi
Differential Revision: D57371597
fbshipit-source-id: 8bededd53e2528f033fac797b296d74b47f9403e
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3439
LLVM-15 has a warning `-Wunused-but-set-variable` which we treat as an error because it's so often diagnostic of a code issue. Unused variables can compromise readability or, worse, performance.
This diff either (a) removes an unused variable and, possibly, it's associated code, or (b) qualifies the variable with `[[maybe_unused]]`, mostly in cases where the variable _is_ used, but, eg, in an `assert` statement that isn't present in production code.
- If you approve of this diff, please use the "Accept & Ship" button :-)
Reviewed By: palmje, junjieqi
Differential Revision: D57344013
fbshipit-source-id: adf410139d2e6ca69a26ccdbff8511c9b7620489
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3432
Addresses the issue in https://github.com/facebookresearch/faiss/issues/3173
for `IndexNNDescent`, I see that there is already interrupt implemented for it's [search](https://fburl.com/code/iwn3tqic) API, so I looked into it's `add` API.
For a given dataset nb = 10 mil, iter = 10, K = 32, d = 32 on a CPU only machine reveals that bulk of the cost comes from [nndescent](https://fburl.com/code/5rdb1p5o). For every iteration of `nndescent` takes around ~12 seconds, ~70-80% of the time is spent on `join` method (~10 seconds per iteration) and ~20-30% spent on `update` (~2 second per iteration). Adding the interrupt on the `join` should suffice on quickly terminating the program when users hit ctrl+C (happy to move the interrupt elsewhere if we think otherwise)
Reviewed By: junjieqi, mdouze
Differential Revision: D57300514
fbshipit-source-id: d343e0a292c35027ffdb8cbd0131e945b9881d63
Summary:
This PR removes unneeded ARM NEON SIMD instructions for ScalarQuantizer.
The removed instructions are completely redundant, and I believe that it is a funky way of converting two `float32x4_t` variables (which hold 4 float values in a single SIMD register) into a single `float32x4x2_t` variable (two SIMD registers packed together).
Clang compiler is capable of eliminating these instructions. The only GCC that can eliminate these unneeded instructions is GCC 14, which was released very recently (Apr-May 2024).
mdouze
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3430
Reviewed By: mlomeli1
Differential Revision: D57369849
Pulled By: mdouze
fbshipit-source-id: 09d7cf16e113df3eb9ddbfa54d074b58b452ba7f
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3428
GitHub Actions currently does not support runners with AVX-512 but committed to add this support in early 2025. We will be running these on CircleCI until then. This placeholder build configuration will allow us to enable it with a 1-liner when the hosts are available.
Reviewed By: algoriddle
Differential Revision: D57261783
fbshipit-source-id: 1fb985a0c3dbb11851af63c95bde6494d25d0ac2
