1202 Commits

Author SHA1 Message Date
xinhuitian
31a29d0378 HNSW search use efSearch from params if provided (#3233)
Summary:
I found that HNSW search method uses its own efSearch to create candidates, which should be got from params if provided I think?

Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3233

Reviewed By: algoriddle

Differential Revision: D53342879

Pulled By: mdouze

fbshipit-source-id: 4f47f19d3673d1bb8d7e8e82745f9e5c1b9afb42
2024-02-02 02:31:57 -08:00
Gergely Szilvasy
c1822a81b2 cuda dependencies (#3230)
Summary: Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3230

Reviewed By: mlomeli1

Differential Revision: D53271742

Pulled By: algoriddle

fbshipit-source-id: b0f09e26ab0b4cfcbdb10528b2f57e15c10f4265
2024-01-31 14:25:13 -08:00
Gergely Szilvasy
1d0e8d489f index optimizer (#3154)
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3154

Using the benchmark to find Pareto optimal indices, in this case on BigANN as an example.

Separately optimize the coarse quantizer and the vector codec and use Pareto optimal configurations to construct IVF indices, which are then retested at various scales. See `optimize()` in `optimize.py` as the main function driving the process.

The results can be interpreted with `bench_fw_notebook.ipynb`, which allows:
* filtering by maximum code size
* maximum time
* minimum accuracy
* space or time Pareto optimal options
* and visualize the results and output them as a table.

This version is intentionally limited to IVF(Flat|HNSW),PQ|SQ indices...

Reviewed By: mdouze

Differential Revision: D51781670

fbshipit-source-id: 2c0f800d374ea845255934f519cc28095c00a51f
2024-01-30 10:58:13 -08:00
Richard Barnes
75ae0bfb7f Remove unused variables in faiss/IndexScalarQuantizer.cpp
Summary:
LLVM-15 has a warning `-Wunused-but-set-variable` which we treat as an error because it's so often diagnostic of a code issue. Unused variables can compromise readability or, worse, performance.

This diff either (a) removes an unused variable and, possibly, it's associated code, or (b) qualifies the variable with `[[maybe_unused]]`, mostly in cases where the variable _is_ used, but, eg, in an `assert` statement that isn't present in production code.

 - If you approve of this diff, please use the "Accept & Ship" button :-)

Reviewed By: algoriddle

Differential Revision: D52981077

fbshipit-source-id: ad164f9fbe267c6f594f9298132578f0a3805a2c
2024-01-30 09:48:48 -08:00
Maria Lomeli
420d25f51c Index pretransform support in search_preassigned (#3225)
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3225

This diff fixes issue [#3113](https://github.com/facebookresearch/faiss/issues/3113), e.g. introduces support for index pretransform in `search_preassigned`.

Reviewed By: mdouze

Differential Revision: D53188584

fbshipit-source-id: 8189c0a59f957a2606391f22cf3fdc8874110a6e
2024-01-30 09:20:07 -08:00
Gergely Szilvasy
31bddeb0cf add faiss_gpu lib (#3222)
Summary:
1. Split the GPU code into a separate static library to reduce build times.
2. faiss-gpu now depends on pytorch-cuda to stop clobbering of CUDA libraries when Faiss and Pytorch installed in the same environment
3. Add CUDA 12 (we support both 11.8 and 12.1)
4. Add Python 3.12

Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3222

Reviewed By: mlomeli1

Differential Revision: D53215033

Pulled By: algoriddle

fbshipit-source-id: d48c5707af20b7e5ff72c3aa27b0e677259f22f8
2024-01-30 01:44:50 -08:00
Gergely Szilvasy
2817344f6f fix ACCESS VIOLATION error when searching using IDSelectorArray
Summary:
Fixes #3156

Metamate says: "This diff fixes an ACCESS VIOLATION error that occurs when searching using IDSelectorArray. The code changes include adding a new parameter to the knn_inner_products_by_idx and knn_L2sqr_by_idx functions in the distances.cpp file, as well as modifying the test_search_params.py file to test the bounds of the IDSelectorArray."

Reviewed By: mdouze

Differential Revision: D53185461

fbshipit-source-id: c7ec4783f77455684c078bba3aace160078f6c27
2024-01-29 23:08:45 -08:00
Richard Barnes
67c6a191b7 Remove extra semi colon from faiss/impl/ProductAdditiveQuantizer.h
Summary:
`-Wextra-semi` or `-Wextra-semi-stmt`

If the code compiles, this is safe to land.

Reviewed By: algoriddle

Differential Revision: D52969139

fbshipit-source-id: 563843d43c64fb988ff0d08ccdb0408cd0eba454
2024-01-27 10:28:14 -08:00
Richard Barnes
898ce35a8a Remove unused function from faiss/utils/distances_simd.cpp
Summary: `-Wunused-function` has identified an unused function. This diff removes it. In many cases these functions have existed for years in an unused state.

Reviewed By: algoriddle

Differential Revision: D53049745

fbshipit-source-id: aa6d9fa5b3ac93c777dd9a5ca239312d1a188a1c
2024-01-25 08:03:31 -08:00
Kota Yamaguchi
1dcb5d314d Add missing header files (#3218)
Summary:
The current CMakeList.txt misses a few header files. This PR adds these missing files.

Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3218

Reviewed By: mlomeli1

Differential Revision: D53081725

Pulled By: algoriddle

fbshipit-source-id: 94531b39350f6d5b9f8c1bedbb1286f265e99a09
2024-01-25 07:39:57 -08:00
Gergely Szilvasy
51b6083187 faiss on rocksdb demo (#3216)
Summary: Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3216

Reviewed By: mdouze

Differential Revision: D53051090

Pulled By: algoriddle

fbshipit-source-id: 13a027db36207af9be11a2f181116994b2aff2cb
2024-01-25 07:39:53 -08:00
Gergely Szilvasy
c4b91a54d1 Replace pickle serialization to address security vulnerability
Summary: This diff replaces the use of pickle serialization with json to address a security vulnerability. Adding a warning message that this code is for demonstration purposes only.

Reviewed By: mdouze

Differential Revision: D52777650

fbshipit-source-id: d9d6a00fd341b29ac854adcbf675d2cd303d2f29
2024-01-25 07:24:04 -08:00
Richard Barnes
a30fd74333 Remove unused function from faiss/clone_index.cpp
Summary: `-Wunused-function` has identified an unused function. This diff removes it. In many cases these functions have existed for years in an unused state.

Reviewed By: palmje

Differential Revision: D53049728

fbshipit-source-id: b3d6b98e4ef035f257e9120369b0e70b3200bec0
2024-01-24 13:19:29 -08:00
Richard Barnes
fc4dbd513c Remove extra semi colon from faiss/IndexIVFSpectralHash.cpp
Summary:
`-Wextra-semi` or `-Wextra-semi-stmt`

If the code compiles, this is safe to land.

Reviewed By: algoriddle

Differential Revision: D52968923

fbshipit-source-id: 5754c0b17b46bc90d0f05d66bbfacb24fee5caae
2024-01-24 12:40:15 -08:00
Richard Barnes
6b2c79c0e3 Remove extra semi colon from faiss/index_factory.cpp
Summary:
`-Wextra-semi` or `-Wextra-semi-stmt`

If the code compiles, this is safe to land.

Reviewed By: algoriddle

Differential Revision: D52969102

fbshipit-source-id: f6e4992783a2a1ab0598bf96c664da950e5bba5d
2024-01-24 12:39:15 -08:00
Richard Barnes
a7b76a7d52 Fix shadowed variable in faiss/IndexAdditiveQuantizerFastScan.cpp
Summary:
Our upcoming compiler upgrade will require us not to have shadowed variables. Such variables have a _high_ bug rate and reduce readability, so we would like to avoid them even if the compiler was not forcing us to do so.

This codemod attempts to fix an instance of a shadowed variable. Please review with care: if it's failed the result will be a silent bug.

**What's a shadowed variable?**

Shadowed variables are variables in an inner scope with the same name as another variable in an outer scope. Having the same name for both variables might be semantically correct, but it can make the code confusing to read! It can also hide subtle bugs.

This diff fixes such an issue by renaming the variable.

 - If you approve of this diff, please use the "Accept & Ship" button :-)

Reviewed By: algoriddle

Differential Revision: D52959038

fbshipit-source-id: 2b688a08710ff60aea0fea9d8be34f3c1f524f85
2024-01-24 12:28:21 -08:00
Richard Barnes
ae25b1bef1 Fix shadowed variable in faiss/impl/residual_quantizer_encode_steps.cpp
Summary:
Our upcoming compiler upgrade will require us not to have shadowed variables. Such variables have a _high_ bug rate and reduce readability, so we would like to avoid them even if the compiler was not forcing us to do so.

This codemod attempts to fix an instance of a shadowed variable. Please review with care: if it's failed the result will be a silent bug.

**What's a shadowed variable?**

Shadowed variables are variables in an inner scope with the same name as another variable in an outer scope. Having the same name for both variables might be semantically correct, but it can make the code confusing to read! It can also hide subtle bugs.

This diff fixes such an issue by renaming the variable.

 - If you approve of this diff, please use the "Accept & Ship" button :-)

Reviewed By: algoriddle

Differential Revision: D52959180

fbshipit-source-id: 90f735f65f1306c817d80c99c8a2108aee7c599a
2024-01-24 11:28:16 -08:00
Richard Barnes
7c4fb6dfe7 Fix shadowed variable in faiss/impl/ResultHandler.h
Summary:
Our upcoming compiler upgrade will require us not to have shadowed variables. Such variables have a _high_ bug rate and reduce readability, so we would like to avoid them even if the compiler was not forcing us to do so.

This codemod attempts to fix an instance of a shadowed variable. Please review with care: if it's failed the result will be a silent bug.

**What's a shadowed variable?**

Shadowed variables are variables in an inner scope with the same name as another variable in an outer scope. Having the same name for both variables might be semantically correct, but it can make the code confusing to read! It can also hide subtle bugs.

This diff fixes such an issue by renaming the variable.

 - If you approve of this diff, please use the "Accept & Ship" button :-)

Reviewed By: algoriddle

Differential Revision: D52958916

fbshipit-source-id: 408f0cc2c4687ef812b62631d708f2633754df69
2024-01-24 11:27:22 -08:00
Richard Barnes
12637a2b12 Fix shadowed variable in faiss/utils/utils.cpp
Summary:
Our upcoming compiler upgrade will require us not to have shadowed variables. Such variables have a _high_ bug rate and reduce readability, so we would like to avoid them even if the compiler was not forcing us to do so.

This codemod attempts to fix an instance of a shadowed variable. Please review with care: if it's failed the result will be a silent bug.

**What's a shadowed variable?**

Shadowed variables are variables in an inner scope with the same name as another variable in an outer scope. Having the same name for both variables might be semantically correct, but it can make the code confusing to read! It can also hide subtle bugs.

This diff fixes such an issue by renaming the variable.

 - If you approve of this diff, please use the "Accept & Ship" button :-)

Reviewed By: algoriddle

Differential Revision: D52959071

fbshipit-source-id: c71b331f9a1ee214cfef8143fdd41c336284d8a2
2024-01-24 11:26:51 -08:00
Kota Yamaguchi
63edc32435 Fix AVX512 build targets in CMakeLists.txt (#3214)
Summary:
When `FAISS_OPT_LEVEL=avx512`, the current default target omits the `avx2` build, which conflicts with the install targets that specify both `avx2` and `avx512` builds. This PR fixes the default build targets for AVX512.

Install targets:

e19de27d72/faiss/CMakeLists.txt (L332)

With this PR, cmake can build and install without any error.

```bash
cmake -B build -DFAISS_OPT_LEVEL=avx512 .
cmake --build build --config Release -j
cmake --install build
```

Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3214

Reviewed By: mlomeli1

Differential Revision: D52996716

Pulled By: algoriddle

fbshipit-source-id: b8a46eee6cc15c2043a1a74c5e15d7a606e94acc
2024-01-24 09:36:53 -08:00
Richard Barnes
99c0e2bd59 Remove unused variables in faiss/utils/sorting.cpp
Summary:
LLVM-15 has a warning `-Wunused-but-set-variable` which we treat as an error because it's so often diagnostic of a code issue. Unused variables can compromise readability or, worse, performance.

This diff either (a) removes an unused variable and, possibly, it's associated code, or (b) qualifies the variable with `[[maybe_unused]]`, mostly in cases where the variable _is_ used, but, eg, in an `assert` statement that isn't present in production code.

 - If you approve of this diff, please use the "Accept & Ship" button :-)

Reviewed By: bunnypak, dmm-fb

Differential Revision: D53011673

fbshipit-source-id: 011204944fdc48627a34e1358833fc525dcc2c28
2024-01-24 09:00:55 -08:00
Richard Barnes
e8494f62d9 Remove unused variables in faiss/IndexBinaryHash.cpp
Summary:
LLVM-15 has a warning `-Wunused-but-set-variable` which we treat as an error because it's so often diagnostic of a code issue. Unused variables can compromise readability or, worse, performance.

This diff either (a) removes an unused variable and, possibly, it's associated code, or (b) qualifies the variable with `[[maybe_unused]]`, mostly in cases where the variable _is_ used, but, eg, in an `assert` statement that isn't present in production code.

 - If you approve of this diff, please use the "Accept & Ship" button :-)

Reviewed By: bunnypak, dmm-fb

Differential Revision: D53011679

fbshipit-source-id: 90cdfb645d66e155b3de853dfab7586a8c720465
2024-01-24 08:50:57 -08:00
Richard Barnes
bbb6ec7e1d Remove unused variables in faiss/IndexShardsIVF.cpp
Summary:
LLVM-15 has a warning `-Wunused-but-set-variable` which we treat as an error because it's so often diagnostic of a code issue. Unused variables can compromise readability or, worse, performance.

This diff either (a) removes an unused variable and, possibly, it's associated code, or (b) qualifies the variable with `[[maybe_unused]]`, mostly in cases where the variable _is_ used, but, eg, in an `assert` statement that isn't present in production code.

 - If you approve of this diff, please use the "Accept & Ship" button :-)

Reviewed By: bunnypak, dmm-fb

Differential Revision: D53011658

fbshipit-source-id: a8ba9102751def7a584167c3f50e4d02e9e6442c
2024-01-23 13:06:41 -08:00
Richard Barnes
0716bde250 Fix shadowed variable in faiss/IndexNSG.cpp
Summary:
Our upcoming compiler upgrade will require us not to have shadowed variables. Such variables have a _high_ bug rate and reduce readability, so we would like to avoid them even if the compiler was not forcing us to do so.

This codemod attempts to fix an instance of a shadowed variable. Please review with care: if it's failed the result will be a silent bug.

**What's a shadowed variable?**

Shadowed variables are variables in an inner scope with the same name as another variable in an outer scope. Having the same name for both variables might be semantically correct, but it can make the code confusing to read! It can also hide subtle bugs.

This diff fixes such an issue by renaming the variable.

 - If you approve of this diff, please use the "Accept & Ship" button :-)

Reviewed By: dmm-fb

Differential Revision: D52959118

fbshipit-source-id: f3b44eb294ae534ebebfcb7d2da4cd70e259eaa3
2024-01-23 08:40:54 -08:00
Richard Barnes
bffedda903 Fix shadowed variable in faiss/IndexIDMap.cpp
Summary:
Our upcoming compiler upgrade will require us not to have shadowed variables. Such variables have a _high_ bug rate and reduce readability, so we would like to avoid them even if the compiler was not forcing us to do so.

This codemod attempts to fix an instance of a shadowed variable. Please review with care: if it's failed the result will be a silent bug.

**What's a shadowed variable?**

Shadowed variables are variables in an inner scope with the same name as another variable in an outer scope. Having the same name for both variables might be semantically correct, but it can make the code confusing to read! It can also hide subtle bugs.

This diff fixes such an issue by renaming the variable.

 - If you approve of this diff, please use the "Accept & Ship" button :-)

Reviewed By: dmm-fb

Differential Revision: D52959009

fbshipit-source-id: 4842fdae7c20105746660304b3c04d20066c70ae
2024-01-23 08:38:30 -08:00
Richard Barnes
a70a8a5e1d Remove unused variables in faiss/impl/HNSW.cpp
Summary:
LLVM-15 has a warning `-Wunused-but-set-variable` which we treat as an error because it's so often diagnostic of a code issue. Unused variables can compromise readability or, worse, performance.

This diff either (a) removes an unused variable and, possibly, it's associated code, or (b) qualifies the variable with `[[maybe_unused]]`, mostly in cases where the variable _is_ used, but, eg, in an `assert` statement that isn't present in production code.

 - If you approve of this diff, please use the "Accept & Ship" button :-)

Reviewed By: algoriddle

Differential Revision: D52981042

fbshipit-source-id: 6748dc957a26a6b5dec3922505c18a1996e3a002
2024-01-23 06:26:35 -08:00
Richard Barnes
b274cb4f4f Remove unused variables in faiss/IndexIVFFastScan.cpp
Summary:
LLVM-15 has a warning `-Wunused-but-set-variable` which we treat as an error because it's so often diagnostic of a code issue. Unused variables can compromise readability or, worse, performance.

This diff either (a) removes an unused variable and, possibly, it's associated code, or (b) qualifies the variable with `[[maybe_unused]]`, mostly in cases where the variable _is_ used, but, eg, in an `assert` statement that isn't present in production code.

 - If you approve of this diff, please use the "Accept & Ship" button :-)

Reviewed By: algoriddle

Differential Revision: D52981034

fbshipit-source-id: 0315bb594a2dfaa059132b5f4e41855a4e15034b
2024-01-23 06:15:33 -08:00
Richard Barnes
e55a0ac5b5 Remove unused variables in faiss/impl/index_write.cpp
Summary:
LLVM-15 has a warning `-Wunused-but-set-variable` which we treat as an error because it's so often diagnostic of a code issue. Unused variables can compromise readability or, worse, performance.

This diff either (a) removes an unused variable and, possibly, it's associated code, or (b) qualifies the variable with `[[maybe_unused]]`, mostly in cases where the variable _is_ used, but, eg, in an `assert` statement that isn't present in production code.

 - If you approve of this diff, please use the "Accept & Ship" button :-)

Reviewed By: algoriddle

Differential Revision: D52981052

fbshipit-source-id: 203e7ead56c69ffd77ee4a61eed1d0a502a14866
2024-01-23 04:40:52 -08:00
Richard Barnes
a651069a37 Remove unused variables in faiss/impl/index_read.cpp
Summary:
LLVM-15 has a warning `-Wunused-but-set-variable` which we treat as an error because it's so often diagnostic of a code issue. Unused variables can compromise readability or, worse, performance.

This diff either (a) removes an unused variable and, possibly, it's associated code, or (b) qualifies the variable with `[[maybe_unused]]`, mostly in cases where the variable _is_ used, but, eg, in an `assert` statement that isn't present in production code.

 - If you approve of this diff, please use the "Accept & Ship" button :-)

Reviewed By: algoriddle

Differential Revision: D52981028

fbshipit-source-id: dbdf5c56e1d594e534183483c236ed2f08f98857
2024-01-23 04:40:47 -08:00
Richard Barnes
e19de27d72 Remove extra semi colon from faiss/IndexIVFFastScan.cpp
Summary:
`-Wextra-semi` or `-Wextra-semi-stmt`

If the code compiles, this is safe to land.

Reviewed By: algoriddle

Differential Revision: D52969081

fbshipit-source-id: d728a4887ac1415ae88dae643055380a183c6990
2024-01-22 20:32:04 -08:00
Richard Barnes
b10eb35569 Remove extra semi colon from faiss/Clustering.cpp
Summary:
`-Wextra-semi` or `-Wextra-semi-stmt`

If the code compiles, this is safe to land.

Reviewed By: algoriddle

Differential Revision: D52969138

fbshipit-source-id: bdde568b9bc6f4cc650e131a471a41f6bc2f3f95
2024-01-22 20:15:40 -08:00
Richard Barnes
683eadf1a0 Remove extra semi colon from faiss/utils/distances_simd.cpp
Summary:
`-Wextra-semi` or `-Wextra-semi-stmt`

If the code compiles, this is safe to land.

Reviewed By: algoriddle

Differential Revision: D52965948

fbshipit-source-id: e24e38ecbb7e0c957b024a1ae7b2ec94d2fa0adf
2024-01-22 19:43:11 -08:00
Richard Barnes
522452f7d3 Remove extra semi colon from faiss/python/python_callbacks.cpp
Summary:
`-Wextra-semi` or `-Wextra-semi-stmt`

If the code compiles, this is safe to land.

Reviewed By: algoriddle

Differential Revision: D52969076

fbshipit-source-id: 97692db6dd7a2d6ffef12ebcdaadd99abf089ca1
2024-01-22 19:06:17 -08:00
Richard Barnes
3e666aea8b Remove extra semi colon from faiss/invlists/InvertedLists.cpp
Summary:
`-Wextra-semi` or `-Wextra-semi-stmt`

If the code compiles, this is safe to land.

Reviewed By: algoriddle

Differential Revision: D52969085

fbshipit-source-id: f532f9000abd580e53159f660f6703dacb02796f
2024-01-22 19:01:12 -08:00
Gergely Szilvasy
7e01b47a1c fix raft build
Summary:
Raft conda packages are built with conda-forge, which ships a newer version of swig, which broke the build. I'm pinning the version as a zero-effort attempt to fix this.

```
[ 80%] Building CXX object CMakeFiles/swigfaiss.dir/CMakeFiles/swigfaiss.dir/swigfaissPYTHON_wrap.cxx.o
/home/circleci/miniconda/conda-bld/faiss-pkg_1705881814815/work/_build_python_3.9/CMakeFiles/swigfaiss.dir/swigfaissPYTHON_wrap.cxx: In function 'PyObject* swig_ptr(PyObject*)':
/home/circleci/miniconda/conda-bld/faiss-pkg_1705881814815/work/_build_python_3.9/CMakeFiles/swigfaiss.dir/swigfaissPYTHON_wrap.cxx:6573:41: error: 'SWIGTYPE_p_unsigned_long_long' was not declared in this scope; did you mean 'SWIGTYPE_p_unsigned_long'?
 6573 |         return SWIG_NewPointerObj(data, SWIGTYPE_p_unsigned_long_long, 0);
      |                                         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/circleci/miniconda/conda-bld/faiss-pkg_1705881814815/work/_build_python_3.9/CMakeFiles/swigfaiss.dir/swigfaissPYTHON_wrap.cxx:1136:94: note: in definition of macro 'SWIG_NewPointerObj'
 1136 | #define SWIG_NewPointerObj(ptr, type, flags)            SWIG_Python_NewPointerObj(NULL, ptr, type, flags)
      |                                                                                              ^~~~
/home/circleci/miniconda/conda-bld/faiss-pkg_1705881814815/work/_build_python_3.9/CMakeFiles/swigfaiss.dir/swigfaissPYTHON_wrap.cxx:6580:41: error: 'SWIGTYPE_p_long_long' was not declared in this scope; did you mean 'SWIGTYPE_p_MapLong2Long'?
 6580 |         return SWIG_NewPointerObj(data, SWIGTYPE_p_long_long, 0);
      |                                         ^~~~~~~~~~~~~~~~~~~~
/home/circleci/miniconda/conda-bld/faiss-pkg_1705881814815/work/_build_python_3.9/CMakeFiles/swigfaiss.dir/swigfaissPYTHON_wrap.cxx:1136:94: note: in definition of macro 'SWIG_NewPointerObj'
 1136 | #define SWIG_NewPointerObj(ptr, type, flags)            SWIG_Python_NewPointerObj(NULL, ptr, type, flags)
      |                                                                                              ^~~~
make[3]: *** [CMakeFiles/swigfaiss.dir/build.make:76: CMakeFiles/swigfaiss.dir/CMakeFiles/swigfaiss.dir/swigfaissPYTHON_wrap.cxx.o] Error 1
```

Reviewed By: mlomeli1

Differential Revision: D52962236

fbshipit-source-id: e1a6b4c6ba03c9a0a320b5dbb54a8d618b8f8e74
2024-01-22 09:51:35 -08:00
Richard Barnes
7f3e0a349f Mismatch tags take 2 (#3211)
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3211

X-link: https://github.com/facebookincubator/velox/pull/8461

Reviewed By: luciang

Differential Revision: D52931746

fbshipit-source-id: 0aa40ff310ec48ace13927faaa47e0ebfa8b921e
2024-01-21 12:14:46 -08:00
Richard Barnes
aff812ed4c Fix shadowed variable in faiss/clone_index.cpp
Summary:
Our upcoming compiler upgrade will require us not to have shadowed variables. Such variables have a _high_ bug rate and reduce readability, so we would like to avoid them even if the compiler was not forcing us to do so.

This codemod attempts to fix an instance of a shadowed variable. Please review with care: if it's failed the result will be a silent bug.

**What's a shadowed variable?**

Shadowed variables are variables in an inner scope with the same name as another variable in an outer scope. Having the same name for both variables might be semantically correct, but it can make the code confusing to read! It can also hide subtle bugs.

This diff fixes such an issue by renaming the variable.

 - If you approve of this diff, please use the "Accept & Ship" button :-)

Reviewed By: dmm-fb

Differential Revision: D52582793

fbshipit-source-id: 2941630671d99de39c19cfc5d24b2e27e495c171
2024-01-19 03:54:36 -08:00
Kota Yamaguchi
c540e762ca Use packaging instead of deprecated distutils for version handling (#3191)
Summary:
This PR replaces deprecated `distutils.version.LooseVersion` with [`packaging.version.Version`](https://packaging.pypa.io/en/stable/index.html). This change is needed to support CPython 3.12+.

Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3191

Reviewed By: algoriddle

Differential Revision: D52872218

Pulled By: mlomeli1

fbshipit-source-id: 336b2a618fde0a65d70b53c7d8a27aef3b62ece1
2024-01-18 11:15:03 -08:00
Gergely Szilvasy
091f344a34 add gpu to avx512 (#3210)
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3210

Adds support for GPUs to the AVX512 binaries

Reviewed By: mlomeli1

Differential Revision: D52874274

fbshipit-source-id: 6b10311f03007c47fe0d7690193591a393942bad
2024-01-18 11:11:58 -08:00
Richard Barnes
5e3eae4fcc Remove unused variables in faiss/utils/hamming.cpp
Summary:
LLVM-15 has a warning `-Wunused-but-set-variable` which we treat as an error because it's so often diagnostic of a code issue. Unused variables can compromise readability or, worse, performance.

This diff either (a) removes an unused variable and, possibly, it's associated code, or (b) qualifies the variable with `[[maybe_unused]]`, mostly in cases where the variable _is_ used, but, eg, in an `assert` statement that isn't present in production code.

 - If you approve of this diff, please use the "Accept & Ship" button :-)

Reviewed By: danzimm

Differential Revision: D52847958

fbshipit-source-id: b3f98d8e0e4fd0e13191f0c0d907edd7c0e78db0
2024-01-17 16:48:14 -08:00
Matthijs Douze
739ce139e9 doc update to mention the paper
Summary: reference the new Faiss paper in the docs. Also added Gergely as a co-author of Faiss. Some minor updates.

Reviewed By: mlomeli1

Differential Revision: D52829321

fbshipit-source-id: 0f1845beace6cd88f809ba50b87a8d446475d30b
2024-01-17 01:47:46 -08:00
Richard Barnes
9a63a3c28d Fix shadowed variable in faiss/IndexFastScan.cpp
Summary:
Our upcoming compiler upgrade will require us not to have shadowed variables. Such variables have a _high_ bug rate and reduce readability, so we would like to avoid them even if the compiler was not forcing us to do so.

This codemod attempts to fix an instance of a shadowed variable. Please review with care: if it's failed the result will be a silent bug.

**What's a shadowed variable?**

Shadowed variables are variables in an inner scope with the same name as another variable in an outer scope. Having the same name for both variables might be semantically correct, but it can make the code confusing to read! It can also hide subtle bugs.

This diff fixes such an issue by renaming the variable.

 - If you approve of this diff, please use the "Accept & Ship" button :-)

Reviewed By: meyering

Differential Revision: D52582914

fbshipit-source-id: edb3983635de4343c4a0b20096e81852cfa058c6
2024-01-16 07:45:48 -08:00
Richard Barnes
7dd06dd188 Fix shadowed variable in faiss/impl/NNDescent.cpp
Summary:
Our upcoming compiler upgrade will require us not to have shadowed variables. Such variables have a _high_ bug rate and reduce readability, so we would like to avoid them even if the compiler was not forcing us to do so.

This codemod attempts to fix an instance of a shadowed variable. Please review with care: if it's failed the result will be a silent bug.

**What's a shadowed variable?**

Shadowed variables are variables in an inner scope with the same name as another variable in an outer scope. Having the same name for both variables might be semantically correct, but it can make the code confusing to read! It can also hide subtle bugs.

This diff fixes such an issue by renaming the variable.

 - If you approve of this diff, please use the "Accept & Ship" button :-)

Reviewed By: meyering

Differential Revision: D52582837

fbshipit-source-id: 275d20f940102030989f9bcfb4884891457b149c
2024-01-16 07:18:20 -08:00
Maria Lomeli
0fc8456e1d Offline IVF powered by faiss big batch search (#3202)
Summary:
This PR introduces the offline IVF (OIVF) framework which contains some tooling to run search using IVFPQ indexes (plus OPQ pretransforms) for large batches of queries using [big_batch_search](https://github.com/mlomeli1/faiss/blob/main/contrib/big_batch_search.py) and GPU faiss. See the [README](36226f5fe8/demos/offline_ivf/README.md) for details about using this framework.

This PR includes the following unit tests, which can be run with the unittest library as so:
````
~/faiss/demos/offline_ivf$ python3 -m unittest tests/test_iterate_input.py -k test_iterate_back
````
In test_offline_ivf:
````
test_consistency_check
test_train_index
test_index_shard_equal_file_sizes
test_index_shard_unequal_file_sizes
test_search
test_evaluate_without_margin
test_evaluate_without_margin_OPQ
````
In test_iterate_input:
````
test_iterate_input_file_larger_than_batch
test_get_vs_iterate
test_iterate_back

````

Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3202

Reviewed By: algoriddle

Differential Revision: D52734222

Pulled By: mlomeli1

fbshipit-source-id: 61fd0084277c1b14bdae1189db8ae43340611e16
2024-01-16 05:05:15 -08:00
Richard Barnes
4150fb1827 Fix shadowed variable in faiss/impl/PolysemousTraining.cpp
Summary:
Our upcoming compiler upgrade will require us not to have shadowed variables. Such variables have a _high_ bug rate and reduce readability, so we would like to avoid them even if the compiler was not forcing us to do so.

This codemod attempts to fix an instance of a shadowed variable. Please review with care: if it's failed the result will be a silent bug.

**What's a shadowed variable?**

Shadowed variables are variables in an inner scope with the same name as another variable in an outer scope. Having the same name for both variables might be semantically correct, but it can make the code confusing to read! It can also hide subtle bugs.

This diff fixes such an issue by renaming the variable.

 - If you approve of this diff, please use the "Accept & Ship" button :-)

Reviewed By: meyering

Differential Revision: D52582870

fbshipit-source-id: 4c81218e109ff97052036645ab33bc9b17802e91
2024-01-15 20:53:34 -08:00
Richard Barnes
3973017742 Fix shadowed variable in faiss/IndexBinaryIVF.cpp
Summary:
Our upcoming compiler upgrade will require us not to have shadowed variables. Such variables have a _high_ bug rate and reduce readability, so we would like to avoid them even if the compiler was not forcing us to do so.

This codemod attempts to fix an instance of a shadowed variable. Please review with care: if it's failed the result will be a silent bug.

**What's a shadowed variable?**

Shadowed variables are variables in an inner scope with the same name as another variable in an outer scope. Having the same name for both variables might be semantically correct, but it can make the code confusing to read! It can also hide subtle bugs.

This diff fixes such an issue by renaming the variable.

 - If you approve of this diff, please use the "Accept & Ship" button :-)

Reviewed By: palmje

Differential Revision: D52582927

fbshipit-source-id: be03fb1354898d0637d287f285ea99d096c39008
2024-01-15 19:29:08 -08:00
Richard Barnes
46320e0e7a Fix shadowed variable in faiss/utils/hamming_distance/avx2-inl.h
Summary:
Our upcoming compiler upgrade will require us not to have shadowed variables. Such variables have a _high_ bug rate and reduce readability, so we would like to avoid them even if the compiler was not forcing us to do so.

This codemod attempts to fix an instance of a shadowed variable. Please review with care: if it's failed the result will be a silent bug.

**What's a shadowed variable?**

Shadowed variables are variables in an inner scope with the same name as another variable in an outer scope. Having the same name for both variables might be semantically correct, but it can make the code confusing to read! It can also hide subtle bugs.

This diff fixes such an issue by renaming the variable.

 - If you approve of this diff, please use the "Accept & Ship" button :-)

Reviewed By: meyering

Differential Revision: D52582813

fbshipit-source-id: 02a742df17614c546e10446bc530835e876942f0
2024-01-14 15:44:03 -08:00
Richard Barnes
1be1d3233a Fix shadowed variable in faiss/impl/io.cpp
Summary:
Our upcoming compiler upgrade will require us not to have shadowed variables. Such variables have a _high_ bug rate and reduce readability, so we would like to avoid them even if the compiler was not forcing us to do so.

This codemod attempts to fix an instance of a shadowed variable. Please review with care: if it's failed the result will be a silent bug.

**What's a shadowed variable?**

Shadowed variables are variables in an inner scope with the same name as another variable in an outer scope. Having the same name for both variables might be semantically correct, but it can make the code confusing to read! It can also hide subtle bugs.

This diff fixes such an issue by renaming the variable.

 - If you approve of this diff, please use the "Accept & Ship" button :-)

Reviewed By: meyering

Differential Revision: D52582928

fbshipit-source-id: 57bd901e87cbb8ddfd423c8ae6baefe1048c206f
2024-01-14 15:42:08 -08:00
Richard Barnes
7442a54d01 Fix shadowed variable in faiss/impl/index_write.cpp
Summary:
Our upcoming compiler upgrade will require us not to have shadowed variables. Such variables have a _high_ bug rate and reduce readability, so we would like to avoid them even if the compiler was not forcing us to do so.

This codemod attempts to fix an instance of a shadowed variable. Please review with care: if it's failed the result will be a silent bug.

**What's a shadowed variable?**

Shadowed variables are variables in an inner scope with the same name as another variable in an outer scope. Having the same name for both variables might be semantically correct, but it can make the code confusing to read! It can also hide subtle bugs.

This diff fixes such an issue by renaming the variable.

 - If you approve of this diff, please use the "Accept & Ship" button :-)

Reviewed By: meyering

Differential Revision: D52582916

fbshipit-source-id: 367ea85b569063efa50c82702c3ba5efa2fb92f9
2024-01-14 15:39:59 -08:00
Richard Barnes
cfcefc0595 Remove unused variables in faiss/utils/sorting.cpp
Summary:
LLVM-15 has a warning `-Wunused-but-set-variable` which we treat as an error because it's so often diagnostic of a code issue. Unused variables can compromise readability or, worse, performance.

This diff either (a) removes an unused variable and, possibly, it's associated code, or (b) qualifies the variable with `[[maybe_unused]]`, mostly in cases where the variable _is_ used, but, eg, in an `assert` statement that isn't present in production code.

 - If you approve of this diff, please use the "Accept & Ship" button :-)

Reviewed By: meyering

Differential Revision: D52734585

fbshipit-source-id: be5da3411d2965f6cd2516376598576049c03f09
2024-01-14 15:23:28 -08:00