1065 Commits

Author SHA1 Message Date
Kota Yamaguchi
c540e762ca Use packaging instead of deprecated distutils for version handling (#3191)
Summary:
This PR replaces deprecated `distutils.version.LooseVersion` with [`packaging.version.Version`](https://packaging.pypa.io/en/stable/index.html). This change is needed to support CPython 3.12+.

Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3191

Reviewed By: algoriddle

Differential Revision: D52872218

Pulled By: mlomeli1

fbshipit-source-id: 336b2a618fde0a65d70b53c7d8a27aef3b62ece1
2024-01-18 11:15:03 -08:00
Gergely Szilvasy
091f344a34 add gpu to avx512 (#3210)
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3210

Adds support for GPUs to the AVX512 binaries

Reviewed By: mlomeli1

Differential Revision: D52874274

fbshipit-source-id: 6b10311f03007c47fe0d7690193591a393942bad
2024-01-18 11:11:58 -08:00
Richard Barnes
5e3eae4fcc Remove unused variables in faiss/utils/hamming.cpp
Summary:
LLVM-15 has a warning `-Wunused-but-set-variable` which we treat as an error because it's so often diagnostic of a code issue. Unused variables can compromise readability or, worse, performance.

This diff either (a) removes an unused variable and, possibly, it's associated code, or (b) qualifies the variable with `[[maybe_unused]]`, mostly in cases where the variable _is_ used, but, eg, in an `assert` statement that isn't present in production code.

 - If you approve of this diff, please use the "Accept & Ship" button :-)

Reviewed By: danzimm

Differential Revision: D52847958

fbshipit-source-id: b3f98d8e0e4fd0e13191f0c0d907edd7c0e78db0
2024-01-17 16:48:14 -08:00
Matthijs Douze
739ce139e9 doc update to mention the paper
Summary: reference the new Faiss paper in the docs. Also added Gergely as a co-author of Faiss. Some minor updates.

Reviewed By: mlomeli1

Differential Revision: D52829321

fbshipit-source-id: 0f1845beace6cd88f809ba50b87a8d446475d30b
2024-01-17 01:47:46 -08:00
Richard Barnes
9a63a3c28d Fix shadowed variable in faiss/IndexFastScan.cpp
Summary:
Our upcoming compiler upgrade will require us not to have shadowed variables. Such variables have a _high_ bug rate and reduce readability, so we would like to avoid them even if the compiler was not forcing us to do so.

This codemod attempts to fix an instance of a shadowed variable. Please review with care: if it's failed the result will be a silent bug.

**What's a shadowed variable?**

Shadowed variables are variables in an inner scope with the same name as another variable in an outer scope. Having the same name for both variables might be semantically correct, but it can make the code confusing to read! It can also hide subtle bugs.

This diff fixes such an issue by renaming the variable.

 - If you approve of this diff, please use the "Accept & Ship" button :-)

Reviewed By: meyering

Differential Revision: D52582914

fbshipit-source-id: edb3983635de4343c4a0b20096e81852cfa058c6
2024-01-16 07:45:48 -08:00
Richard Barnes
7dd06dd188 Fix shadowed variable in faiss/impl/NNDescent.cpp
Summary:
Our upcoming compiler upgrade will require us not to have shadowed variables. Such variables have a _high_ bug rate and reduce readability, so we would like to avoid them even if the compiler was not forcing us to do so.

This codemod attempts to fix an instance of a shadowed variable. Please review with care: if it's failed the result will be a silent bug.

**What's a shadowed variable?**

Shadowed variables are variables in an inner scope with the same name as another variable in an outer scope. Having the same name for both variables might be semantically correct, but it can make the code confusing to read! It can also hide subtle bugs.

This diff fixes such an issue by renaming the variable.

 - If you approve of this diff, please use the "Accept & Ship" button :-)

Reviewed By: meyering

Differential Revision: D52582837

fbshipit-source-id: 275d20f940102030989f9bcfb4884891457b149c
2024-01-16 07:18:20 -08:00
Maria Lomeli
0fc8456e1d Offline IVF powered by faiss big batch search (#3202)
Summary:
This PR introduces the offline IVF (OIVF) framework which contains some tooling to run search using IVFPQ indexes (plus OPQ pretransforms) for large batches of queries using [big_batch_search](https://github.com/mlomeli1/faiss/blob/main/contrib/big_batch_search.py) and GPU faiss. See the [README](36226f5fe8/demos/offline_ivf/README.md) for details about using this framework.

This PR includes the following unit tests, which can be run with the unittest library as so:
````
~/faiss/demos/offline_ivf$ python3 -m unittest tests/test_iterate_input.py -k test_iterate_back
````
In test_offline_ivf:
````
test_consistency_check
test_train_index
test_index_shard_equal_file_sizes
test_index_shard_unequal_file_sizes
test_search
test_evaluate_without_margin
test_evaluate_without_margin_OPQ
````
In test_iterate_input:
````
test_iterate_input_file_larger_than_batch
test_get_vs_iterate
test_iterate_back

````

Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3202

Reviewed By: algoriddle

Differential Revision: D52734222

Pulled By: mlomeli1

fbshipit-source-id: 61fd0084277c1b14bdae1189db8ae43340611e16
2024-01-16 05:05:15 -08:00
Richard Barnes
4150fb1827 Fix shadowed variable in faiss/impl/PolysemousTraining.cpp
Summary:
Our upcoming compiler upgrade will require us not to have shadowed variables. Such variables have a _high_ bug rate and reduce readability, so we would like to avoid them even if the compiler was not forcing us to do so.

This codemod attempts to fix an instance of a shadowed variable. Please review with care: if it's failed the result will be a silent bug.

**What's a shadowed variable?**

Shadowed variables are variables in an inner scope with the same name as another variable in an outer scope. Having the same name for both variables might be semantically correct, but it can make the code confusing to read! It can also hide subtle bugs.

This diff fixes such an issue by renaming the variable.

 - If you approve of this diff, please use the "Accept & Ship" button :-)

Reviewed By: meyering

Differential Revision: D52582870

fbshipit-source-id: 4c81218e109ff97052036645ab33bc9b17802e91
2024-01-15 20:53:34 -08:00
Richard Barnes
3973017742 Fix shadowed variable in faiss/IndexBinaryIVF.cpp
Summary:
Our upcoming compiler upgrade will require us not to have shadowed variables. Such variables have a _high_ bug rate and reduce readability, so we would like to avoid them even if the compiler was not forcing us to do so.

This codemod attempts to fix an instance of a shadowed variable. Please review with care: if it's failed the result will be a silent bug.

**What's a shadowed variable?**

Shadowed variables are variables in an inner scope with the same name as another variable in an outer scope. Having the same name for both variables might be semantically correct, but it can make the code confusing to read! It can also hide subtle bugs.

This diff fixes such an issue by renaming the variable.

 - If you approve of this diff, please use the "Accept & Ship" button :-)

Reviewed By: palmje

Differential Revision: D52582927

fbshipit-source-id: be03fb1354898d0637d287f285ea99d096c39008
2024-01-15 19:29:08 -08:00
Richard Barnes
46320e0e7a Fix shadowed variable in faiss/utils/hamming_distance/avx2-inl.h
Summary:
Our upcoming compiler upgrade will require us not to have shadowed variables. Such variables have a _high_ bug rate and reduce readability, so we would like to avoid them even if the compiler was not forcing us to do so.

This codemod attempts to fix an instance of a shadowed variable. Please review with care: if it's failed the result will be a silent bug.

**What's a shadowed variable?**

Shadowed variables are variables in an inner scope with the same name as another variable in an outer scope. Having the same name for both variables might be semantically correct, but it can make the code confusing to read! It can also hide subtle bugs.

This diff fixes such an issue by renaming the variable.

 - If you approve of this diff, please use the "Accept & Ship" button :-)

Reviewed By: meyering

Differential Revision: D52582813

fbshipit-source-id: 02a742df17614c546e10446bc530835e876942f0
2024-01-14 15:44:03 -08:00
Richard Barnes
1be1d3233a Fix shadowed variable in faiss/impl/io.cpp
Summary:
Our upcoming compiler upgrade will require us not to have shadowed variables. Such variables have a _high_ bug rate and reduce readability, so we would like to avoid them even if the compiler was not forcing us to do so.

This codemod attempts to fix an instance of a shadowed variable. Please review with care: if it's failed the result will be a silent bug.

**What's a shadowed variable?**

Shadowed variables are variables in an inner scope with the same name as another variable in an outer scope. Having the same name for both variables might be semantically correct, but it can make the code confusing to read! It can also hide subtle bugs.

This diff fixes such an issue by renaming the variable.

 - If you approve of this diff, please use the "Accept & Ship" button :-)

Reviewed By: meyering

Differential Revision: D52582928

fbshipit-source-id: 57bd901e87cbb8ddfd423c8ae6baefe1048c206f
2024-01-14 15:42:08 -08:00
Richard Barnes
7442a54d01 Fix shadowed variable in faiss/impl/index_write.cpp
Summary:
Our upcoming compiler upgrade will require us not to have shadowed variables. Such variables have a _high_ bug rate and reduce readability, so we would like to avoid them even if the compiler was not forcing us to do so.

This codemod attempts to fix an instance of a shadowed variable. Please review with care: if it's failed the result will be a silent bug.

**What's a shadowed variable?**

Shadowed variables are variables in an inner scope with the same name as another variable in an outer scope. Having the same name for both variables might be semantically correct, but it can make the code confusing to read! It can also hide subtle bugs.

This diff fixes such an issue by renaming the variable.

 - If you approve of this diff, please use the "Accept & Ship" button :-)

Reviewed By: meyering

Differential Revision: D52582916

fbshipit-source-id: 367ea85b569063efa50c82702c3ba5efa2fb92f9
2024-01-14 15:39:59 -08:00
Richard Barnes
cfcefc0595 Remove unused variables in faiss/utils/sorting.cpp
Summary:
LLVM-15 has a warning `-Wunused-but-set-variable` which we treat as an error because it's so often diagnostic of a code issue. Unused variables can compromise readability or, worse, performance.

This diff either (a) removes an unused variable and, possibly, it's associated code, or (b) qualifies the variable with `[[maybe_unused]]`, mostly in cases where the variable _is_ used, but, eg, in an `assert` statement that isn't present in production code.

 - If you approve of this diff, please use the "Accept & Ship" button :-)

Reviewed By: meyering

Differential Revision: D52734585

fbshipit-source-id: be5da3411d2965f6cd2516376598576049c03f09
2024-01-14 15:23:28 -08:00
Richard Barnes
f884ba240e Remove unused variables in faiss/IndexHNSW.cpp
Summary:
LLVM-15 has a warning `-Wunused-but-set-variable` which we treat as an error because it's so often diagnostic of a code issue. Unused variables can compromise readability or, worse, performance.

This diff either (a) removes an unused variable and, possibly, it's associated code, or (b) qualifies the variable with `[[maybe_unused]]`, mostly in cases where the variable _is_ used, but, eg, in an `assert` statement that isn't present in production code.

 - If you approve of this diff, please use the "Accept & Ship" button :-)

Reviewed By: meyering

Differential Revision: D52734573

fbshipit-source-id: edf315a4e2e7ddfaf8b879b7f4a3fc3a6c2e607f
2024-01-14 15:22:31 -08:00
Gergely Szilvasy
b7efd3add7 bump libraft version to fix nightly
Summary:
Diff summary courtesy of Metamate:

This diff bumps the version of libraft to fix nightly issues in the faiss-gpu-raft package. The code changes include updating the libraft version in the meta.yaml file to 24.02. This change is necessary to ensure that the package is compatible with the latest version of libraft.

Reviewed By: mlomeli1

Differential Revision: D52736355

fbshipit-source-id: 7ac2427210c06eb2d1c3ab4a3d37d48494fb039e
2024-01-13 10:05:35 -08:00
Naveen Tatikonda
c3aa526ec7 Add SIMD NEON Optimization for QT_FP16 in Scalar Quantizer (#3166)
Summary:
### Description
We have SIMD support for x86 architecture using AVX2 optimization. But, we don't have a similar optimization for ARM architecture in Scalar Quantizer for the Quantization Type `QT_FP16`. This PR adds SIMD support for ARM using NEON optimization.

### Issues resolved
https://github.com/facebookresearch/faiss/issues/3014

Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3166

Reviewed By: algoriddle, pemazare

Differential Revision: D52510486

Pulled By: mdouze

fbshipit-source-id: 2bb360475a0d9e0816c8e05b44d7da7f2e6b28c5
2024-01-11 13:53:27 -08:00
Matthijs Douze
32f0e8cf92 Generalize ResultHanlder, support range search for HNSW and Fast Scan (#3190)
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3190

This diff adds more result handlers in order to expose them externally.
This enables range search for HSNW and Fast Scan, and nprobe parameter support for FastScan.

Reviewed By: pemazare

Differential Revision: D52547384

fbshipit-source-id: 271da5ffea6411df3d8e50641abade18bd7b774b
2024-01-11 11:46:30 -08:00
Gergely Szilvasy
0013c702f4 avx512 CI + conda packages (#3197)
Summary: Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3197

Reviewed By: mlomeli1

Differential Revision: D52689379

Pulled By: algoriddle

fbshipit-source-id: 54e27c6d310d6da14777ae10ae62f46e7076cacf
2024-01-11 08:26:33 -08:00
Richard Barnes
0710cbd48a Fix shadowed variable in faiss/impl/ResultHandler.h
Summary:
Our upcoming compiler upgrade will require us not to have shadowed variables. Such variables have a _high_ bug rate and reduce readability, so we would like to avoid them even if the compiler was not forcing us to do so.

This codemod attempts to fix an instance of a shadowed variable. Please review with care: if it's failed the result will be a silent bug.

**What's a shadowed variable?**

Shadowed variables are variables in an inner scope with the same name as another variable in an outer scope. Having the same name for both variables might be semantically correct, but it can make the code confusing to read! It can also hide subtle bugs.

This diff fixes such an issue by renaming the variable.

 - If you approve of this diff, please use the "Accept & Ship" button :-)

Reviewed By: algoriddle

Differential Revision: D52582910

fbshipit-source-id: 92ee06e819f15c411a3fee75a6c9e17fba96a7f3
2024-01-09 08:30:59 -08:00
Richard Barnes
42b62163ca Fix shadowed variable in faiss/impl/ScalarQuantizer.cpp
Summary:
Our upcoming compiler upgrade will require us not to have shadowed variables. Such variables have a _high_ bug rate and reduce readability, so we would like to avoid them even if the compiler was not forcing us to do so.

This codemod attempts to fix an instance of a shadowed variable. Please review with care: if it's failed the result will be a silent bug.

**What's a shadowed variable?**

Shadowed variables are variables in an inner scope with the same name as another variable in an outer scope. Having the same name for both variables might be semantically correct, but it can make the code confusing to read! It can also hide subtle bugs.

This diff fixes such an issue by renaming the variable.

 - If you approve of this diff, please use the "Accept & Ship" button :-)

Reviewed By: algoriddle

Differential Revision: D52582823

fbshipit-source-id: 575c286c0445e2388038aac10327d6c1bd63d6df
2024-01-09 08:27:25 -08:00
Richard Barnes
c399b110a0 Fix shadowed variable in faiss/utils/hamming.cpp
Summary:
Our upcoming compiler upgrade will require us not to have shadowed variables. Such variables have a _high_ bug rate and reduce readability, so we would like to avoid them even if the compiler was not forcing us to do so.

This codemod attempts to fix an instance of a shadowed variable. Please review with care: if it's failed the result will be a silent bug.

**What's a shadowed variable?**

Shadowed variables are variables in an inner scope with the same name as another variable in an outer scope. Having the same name for both variables might be semantically correct, but it can make the code confusing to read! It can also hide subtle bugs.

This diff fixes such an issue by renaming the variable.

 - If you approve of this diff, please use the "Accept & Ship" button :-)

Reviewed By: algoriddle

Differential Revision: D52582797

fbshipit-source-id: 4ade95834fee68a4ac86b4d03a00df0760941f4e
2024-01-09 08:26:20 -08:00
Richard Barnes
952941ba3f Fix shadowed variable in faiss/MatrixStats.cpp
Summary:
Our upcoming compiler upgrade will require us not to have shadowed variables. Such variables have a _high_ bug rate and reduce readability, so we would like to avoid them even if the compiler was not forcing us to do so.

This codemod attempts to fix an instance of a shadowed variable. Please review with care: if it's failed the result will be a silent bug.

**What's a shadowed variable?**

Shadowed variables are variables in an inner scope with the same name as another variable in an outer scope. Having the same name for both variables might be semantically correct, but it can make the code confusing to read! It can also hide subtle bugs.

This diff fixes such an issue by renaming the variable.

 - If you approve of this diff, please use the "Accept & Ship" button :-)

Reviewed By: algoriddle

Differential Revision: D52582890

fbshipit-source-id: ef7e44805fb212c6415030b98f65c00b2fd4a9b7
2024-01-09 08:17:54 -08:00
Richard Barnes
db09984c7d Fix shadowed variable in faiss/impl/simd_result_handlers.h
Summary:
Our upcoming compiler upgrade will require us not to have shadowed variables. Such variables have a _high_ bug rate and reduce readability, so we would like to avoid them even if the compiler was not forcing us to do so.

This codemod attempts to fix an instance of a shadowed variable. Please review with care: if it's failed the result will be a silent bug.

**What's a shadowed variable?**

Shadowed variables are variables in an inner scope with the same name as another variable in an outer scope. Having the same name for both variables might be semantically correct, but it can make the code confusing to read! It can also hide subtle bugs.

This diff fixes such an issue by renaming the variable.

 - If you approve of this diff, please use the "Accept & Ship" button :-)

Reviewed By: dmm-fb

Differential Revision: D52582819

fbshipit-source-id: 34c4d5eca31bcc2df5872c9eb3a286335c8ae131
2024-01-07 20:53:05 -08:00
Gergely Szilvasy
beef6107fc faiss paper benchmarks (#3189)
Summary:
- IVF benchmarks: `bench_fw_ivf.py bench_fw_ivf.py bigann /checkpoint/gsz/bench_fw/ivf`
- Codec benchmarks: `bench_fw_codecs.py contriever /checkpoint/gsz/bench_fw/codecs` and `bench_fw_codecs.py deep1b /checkpoint/gsz/bench_fw/codecs`
- A range codec evaluation: `bench_fw_range.py ssnpp /checkpoint/gsz/bench_fw/range`
- Visualize with `bench_fw_notebook.ipynb`
- Support for running on a cluster

Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3189

Reviewed By: mdouze

Differential Revision: D52544642

Pulled By: algoriddle

fbshipit-source-id: 21dcdfd076aef6d36467c908e6be78ef851b0e98
2024-01-05 09:27:04 -08:00
Richard Barnes
b7681bef40 Remove unused exception parameter from files inc facer/engine/utils/StatsD.cpp
Summary:
`-Wunused-exception-parameter` has identified an unused exception parameter. This diff removes it.

This:
```
try {
    ...
} catch (exception& e) {
    // no use of e
}
```
should instead be written as
```
} catch (exception&) {
```

If the code compiles, this is safe to land.

Reviewed By: dmm-fb

Differential Revision: D51977749

fbshipit-source-id: aab6f8bcb8a0d9273b7894b8e85615d4a31cac86
2024-01-03 14:53:08 -08:00
Richard Barnes
449f02c449 Add missing fallthrough to fbcode/faiss/utils/hamming_distance/neon-inl.h
Differential Revision: D52436745

fbshipit-source-id: 7e4c55a8255b2f25b3394146a24d1af698b3b2e9
2023-12-27 14:34:55 -08:00
Eddy Li
0727fff941 Adding [[fallthrough]] annotation to case statements in faiss/utils/hamming_distance/neon-inl.h
Summary:
Error: https://www.internalfb.com/sandcastle/workflow/1661828262501424314
Due to a recent change (D52393687) to increase compiler warnings arcoss faiss/PACKAGE aggregator build is failing. The annotation [[fallthrough]]; needs to be used in every [nested]case statement where fallthrough can occur.

WHAT I DIDNT DO
I did NOT EXPORTED TO OSS FAISS
mdouze please do this if it is required. I do not know your policy. Prior to landing i saw the message but was unsure. As well as not having github account.

Reviewed By: r-barnes

Differential Revision: D52420730

fbshipit-source-id: 92ec8382226ea35bdacb668b3c329bd8b41a67fc
2023-12-26 15:14:13 -08:00
Jason Sylka
9d74f0994f Adding [[fallthrough]] annotation to case statements in faiss/utils/hamming_distance/generic-inl.h
Summary:
## Context
Due to a recent change (D52393687) to increase compiler warnings arcoss faiss/PACKAGE aggregator [build](https://www.internalfb.com/chronos/job/gp/9007207455707765) is failing. The annotation [[fallthrough]]; needs to be used in every [nested]case statement where fallthrough can occur.

## COMPILER ERROR
```
[2023-12-22T22:06:48.478-08:00] Action failed: fbcode//faiss:faiss (ovr_config//platform/linux:x86_64-fbcode-platform010-clang15-jemalloc-master#a916d57931e3a679) (cxx_compile impl/LocalSearchQuantizer.cpp)
[CONTEXT] [2023-12-22T22:06:48.478-08:00] Remote command returned non-zero exit code 1
[CONTEXT] [2023-12-22T22:06:48.478-08:00] Remote action, reproduce with: `frecli cas download-action d02e5a31e82ab2a6a2ce8b84dff4d4bce3997ca49783013f860d16dd566edaaf:145`
buck-out/v2/gen/fbcode/a916d57931e3a679/faiss/__faiss__/buck-headers/faiss/utils/hamming_distance/generic-inl.h:278:21: error: unannotated fall-through between switch labels [-Werror,-Wimplicit-fallthrough]
[CONTEXT]                     case 7:
[CONTEXT]                     ^
buck-out/v2/gen/fbcode/a916d57931e3a679/faiss/__faiss__/buck-headers/faiss/utils/hamming_distance/generic-inl.h:281:21: error: unannotated fall-through between switch labels [-Werror,-Wimplicit-fallthrough]
[CONTEXT]                     case 6:
[CONTEXT]                     ^
buck-out/v2/gen/fbcode/a916d57931e3a679/faiss/__faiss__/buck-headers/faiss/utils/hamming_distance/generic-inl.h:284:21: error: unannotated fall-through between switch labels [-Werror,-Wimplicit-fallthrough]
[CONTEXT]                     case 5:
```

## WHAT I DID
 +1 annotation **[[fallthrough]];** at every case statement that required it.

## WHAT I DIDNT DO
 - I did NOT **EXPORTED TO OSS FAISS**
   - mdouze please do this if it is required. I do not know your policy. Prior to landing i saw the message but was unsure. As well as not having github account.

Differential Revision: D52396801

fbshipit-source-id: a563068af00135f7fd24de6633007a1957fc6e31
2023-12-23 07:39:10 -08:00
Gergely Szilvasy
77c28f8ba4 Back out "Offline IVF powered by faiss big batch search"
Summary: Revert so that the test data can be replaced with synthetic

Reviewed By: mdouze

Differential Revision: D52388604

fbshipit-source-id: c0037635a4e66c54d42400294d13d9a80610b845
2023-12-22 03:43:05 -08:00
Richard Barnes
65b08ef2b7 Remove unused function from faiss/utils/quantize_lut.cpp
Summary: `-Wunused-function` has identified an unused function. This diff removes it. In many cases these functions have existed for years in an unused state.

Reviewed By: bunnypak

Differential Revision: D52355736

fbshipit-source-id: e315cf420053472218bbf2fdd647ab073c6f6cd6
2023-12-21 15:31:48 -08:00
Richard Barnes
562148786e Remove unused function from faiss/impl/ProductQuantizer.cpp
Summary: `-Wunused-function` has identified an unused function. This diff removes it. In many cases these functions have existed for years in an unused state.

Reviewed By: bunnypak

Differential Revision: D52355753

fbshipit-source-id: 54c8b47224c1dafc4e2464fe4e528c78ecabf0b8
2023-12-21 15:18:16 -08:00
Maria
9a8b34e295 Offline IVF powered by faiss big batch search (#3175)
Summary:
This PR introduces the offline IVF (OIVF) framework  which contains some tooling to run search using IVFPQ indexes (plus OPQ pretransforms) for large batches of queries using [big_batch_search](https://github.com/mlomeli1/faiss/blob/main/contrib/big_batch_search.py) and GPU faiss. See the [README](https://github.com/mlomeli1/faiss/blob/oivf/demos/offline_ivf/README.md) for details about using this framework.

This PR includes the following unit tests, which can be run with the unittest library as so:
````
~/faiss/demos/offline_ivf$ python3 -m unittest tests/test_iterate_input.py -k test_iterate_back
````

In test_offline_ivf:
````
test_consistency_check
test_train_index
test_index_shard_equal_file_sizes
test_index_shard_unequal_file_sizes
 test_search
test_evaluate_without_margin
test_evaluate_without_margin_OPQ
test_evaluate_with_margin
test_split_batch_size_bigger_than_file_sizes
test_split_batch_size_smaller_than_file_sizes
test_split_files_with_corrupted_input_file
````

In test_iterate_input:
````
test_iterate_input_file_larger_than_batch
test_get_vs_iterate
test_iterate_back
````

Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3175

Reviewed By: algoriddle

Differential Revision: D52218447

Pulled By: mlomeli1

fbshipit-source-id: 78b12457c79b02eb2c9ae993560f2e295798e7e5
2023-12-18 15:09:31 -08:00
Maria Lomeli
be1242775a Upstream changes to big batch search (#3170)
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3170

Logging info, adding to heap and wait_in and out times.

Reviewed By: algoriddle

Differential Revision: D52034667

fbshipit-source-id: 8ab864c5c43d534d094c6e81bb810c74e20c9ac2
2023-12-12 09:51:05 -08:00
Xiaozhong Pan
79f558f1d9 set keep_max for InvertedListScanner child classes (#3169)
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3169

InvertedListScanner.keep_max is used in iterate_codes method, this field is not set to true when metric is IP, causing problems when using InvertedListsIterator interface.

Reviewed By: algoriddle

Differential Revision: D52006595

fbshipit-source-id: 6adbe377c0b9f198c9fd8704869133407ef845e5
2023-12-11 14:29:34 -08:00
Bader-eddine Ouaich
9f8f2384c9 Fix potential buffer overflow (#3146)
Summary:
Size of pointer `sub_x` used instead of size of its data. This is likely to lead to a buffer overflow if the user is not lucky enough to be in a x32 bit machine where `sizeof(float*) == sizeof(float)`.
You probably intend to write `sizeof(*sub_x)` or `sizeof(float)` ?

Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3146

Reviewed By: mlomeli1

Differential Revision: D51618892

Pulled By: algoriddle

fbshipit-source-id: 82acd98009515e963c547d5c62946f93ef5561b5
2023-12-11 06:57:27 -08:00
Jaepil Jeong
91070353b1 Fix compile error (#3162)
Summary:
`WorkerThread.h` uses `std::function` but it does not have `#include <functional>` and this causes compile errors in some cases. This simple PR fixes this issue.

Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3162

Reviewed By: mlomeli1

Differential Revision: D51987803

Pulled By: algoriddle

fbshipit-source-id: 21db00a7c9836e83401bb2c82e86a49c80d52f88
2023-12-11 05:42:10 -08:00
Alexandr Guzhva
d40b39aeb7 Nit: fix variable types (#3147)
Summary:
The inconsistency in variable types would generate waaaaaaay too many warnings `comparison of integer expressions of different signedness` during the compilation if all the warnings are made reported

Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3147

Reviewed By: mlomeli1

Differential Revision: D51988074

Pulled By: algoriddle

fbshipit-source-id: 31a57bbe3a14105a968d3f06c0807c2fe60f1a62
2023-12-11 05:33:54 -08:00
Richard Barnes
aebc1d5fda Remove extra semi colon from faiss/impl/IDSelector.h
Summary:
`-Wextra-semi` or `-Wextra-semi-stmt`

If the code compiles, this is safe to land.

Reviewed By: palmje

Differential Revision: D51995085

fbshipit-source-id: c0a4f134f9361143f337521759793d1cd86e145c
2023-12-08 18:19:46 -08:00
Richard Barnes
3dab47cf23 Remove extra semi colon from facer/engine/libsvm/svm.cpp
Summary:
`-Wextra-semi` or `-Wextra-semi-stmt`

If the code compiles, this is safe to land.

Reviewed By: palmje

Differential Revision: D51995074

fbshipit-source-id: 4863250ff9f4b87241f6c01cf3b39b6b87172852
2023-12-08 17:16:44 -08:00
Richard Barnes
d2cb016ff8 Remove unused exception parameter from files inc faiss/tests/test_approx_topk.cpp
Summary:
`-Wunused-exception-parameter` has identified an unused exception parameter. This diff removes it.

This:
```
try {
    ...
} catch (exception& e) {
    // no use of e
}
```
should instead be written as
```
} catch (exception&) {
```

If the code compiles, this is safe to land.

Reviewed By: algoriddle

Differential Revision: D51977717

fbshipit-source-id: 47a308ff3570935829531663252409b4e0cfdec4
2023-12-08 05:37:19 -08:00
Matthijs Douze
5b6c4b40c8 Back out "printf -> fmt::print in files inc faiss/IndexBinaryHNSW.cpp" (#3164)
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3164

The original diff breaks the open-source Faiss compiles.

Original commit changeset: 230540b26ec8

Original Phabricator Diff: D51486397

Reviewed By: algoriddle

Differential Revision: D51938993

fbshipit-source-id: a57433c4267493d2fe2249e8f4191612c0f1da59
2023-12-07 06:40:34 -08:00
Richard Barnes
131adc5743 printf -> fmt::print in files inc faiss/IndexBinaryHNSW.cpp
Summary:
One of our [C++ 2024 Modern Code Better Engineering Goals](https://fb.workplace.com/groups/fbcode.fyi/posts/6837863056249437) is to reduce or eliminate the use of printf in our codebase, replacing it with safer and more performant alternatives.

This diff replaces `printf` with its preferred alternative: `fmt::print`.

This diff was auto-generated by fbcode/scripts/rbarnes/printf_upgrader.py, please reach out to r-barnes if you see problems.

 - If you approve of this diff, please use the "Accept & Ship" button :-)

(11 file modified.)

Reviewed By: bcardosolopes

Differential Revision: D51486397

fbshipit-source-id: 230540b26ec8717d8e8aa4ae14d6a34718a964ff
2023-12-05 10:00:38 -08:00
Alexandr Guzhva
eefa39105e Introduce avx512 optimization mode and FAISS_OPT_LEVEL env variable (#3150)
Summary:
Enables avx512 optimized code (AVX512 subsets F, CD, VL, DQ and BW, which are available for Intel Skylake+ and all AMD Zen4).
Also, introduces `FAISS_OPT_LEVEL` environment variable. Set it to `AVX2`, `AVX512` or empty to pick the appropriate x86_64 instruction set.

Compiled via the following
```
cmake -B build -DCMAKE_BUILD_TYPE=Release -DFAISS_ENABLE_GPU=OFF -DFAISS_OPT_LEVEL=avx512 -DBUILD_TESTING=ON .
make -C build -j 8 faiss_test
make -C build -j 8 swigfaiss
make -C build -j 8 swigfaiss_avx2
make -C build -j 8 swigfaiss_avx512
cd build/faiss/python
python3 setup.py build
python3 setup.py install --force
```

Now, running the following script `1.py`
```
import logging
logging.basicConfig(level=logging.DEBUG)

import faiss
```
produces the following:
```
root@6179abeef23c:~/faiss# LOGLEVEL=DEBUG FAISS_OPT_LEVEL= python3 1.py
DEBUG:faiss.loader:Using  as an instruction set.
INFO:faiss.loader:Loading faiss.
INFO:faiss.loader:Successfully loaded faiss.
root@6179abeef23c:~/faiss# LOGLEVEL=DEBUG FAISS_OPT_LEVEL=AVX2 python3 1.py
DEBUG:faiss.loader:Using AVX2 as an instruction set.
INFO:faiss.loader:Loading faiss with AVX2 support.
INFO:faiss.loader:Successfully loaded faiss with AVX2 support.
root@6179abeef23c:~/faiss# LOGLEVEL=DEBUG FAISS_OPT_LEVEL=AVX512 python3 1.py
DEBUG:faiss.loader:Using AVX512 as an instruction set.
INFO:faiss.loader:Loading faiss with AVX512 support.
INFO:faiss.loader:Successfully loaded faiss with AVX512 support.
root@6179abeef23c:~/faiss# LOGLEVEL=DEBUG python3 1.py
DEBUG:faiss.loader:Environment variable FAISS_OPT_LEVEL is not set, so let's pick the instruction set according to the current CPU
INFO:faiss.loader:Loading faiss with AVX512 support.
INFO:faiss.loader:Successfully loaded faiss with AVX512 support.
```

Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3150

Reviewed By: algoriddle

Differential Revision: D51701077

Pulled By: mdouze

fbshipit-source-id: 4db05a287e763ff1ce1f676df7f7402532bf1e9e
2023-12-05 01:39:52 -08:00
Gergely Szilvasy
4c83965d2b benchmark view results (#3144)
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3144

Visualize results of running the benchmark with Pareto optima filtering:
1. per index or across indices
2. for space, time or space & time
3. knn or range search, the latter @ specific precision

Reviewed By: mdouze

Differential Revision: D51552775

fbshipit-source-id: d4f29e3d46ef044e71b54439b3972548c86af5a7
2023-12-04 05:53:17 -08:00
Gergely Szilvasy
9519a19f42 benchmark refactor
Summary:
1. Support for index construction parameters outside of the factory string (arbitrary depth of quantizers).
2. Refactor that provides an index wrapper which is a prereq for the optimizer, which will generate indices from pre-optimized components (particularly quantizers)

Reviewed By: mdouze

Differential Revision: D51427452

fbshipit-source-id: 014d05dd798d856360f2546963e7cad64c2fcaeb
2023-12-04 05:53:17 -08:00
Alexandr Guzhva
a5b03cb9f6 Fix build on Alpine Linux (#3148)
Summary:
Fixes https://github.com/facebookresearch/faiss/issues/3142

Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3148

Reviewed By: algoriddle

Differential Revision: D51741575

Pulled By: mdouze

fbshipit-source-id: b4ea2302802ef70b53f8afe5a7ab4ee02bf6659e
2023-12-01 02:52:52 -08:00
Yuri Vanin
4bf8f939d6 Add NegativeDistanceComputer::distances_batch_4 override (#3149)
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3149

Enables vectorized distance calculation in [NegativeDistanceComputer](b109d086a2/faiss/IndexHNSW.cpp (L74)), whenever supported by the [NegativeDistanceComputer::basedis](b109d086a2/faiss/IndexHNSW.cpp (L76)).

Otherwise the default sequential calculation of [DistanceComputer::distances_batch_4](b109d086a2/faiss/impl/DistanceComputer.h (L36-L54)) is always chosen.

Reviewed By: algoriddle

Differential Revision: D51596177

fbshipit-source-id: fee510c0a229991ecb7d81a51bc53a20880685be
2023-11-29 16:38:08 -08:00
Gergely Szilvasy
90654d6011 benchmark core faiss prereqs
Summary:
1. Support `search_preassigned` in IVFFastScan
2. `try_extract_index_ivf` to search recursively and support `IndexRefine`
3. `get_InvertedListScanner` to fail where not available
4. Workaround an OpenMP issue with `IndexIVFSpectralHash`

Reviewed By: mdouze

Differential Revision: D51427241

fbshipit-source-id: 365e3f11d24e80f101f986fc358c28dcc00805fa
2023-11-28 11:50:03 -08:00
Alexandr Guzhva
04bb0a810c improve ScalarQuantizer performance, ESPECIALLY on old GCC (#3141)
Summary:
Introduces `FAISS_ALWAYS_INLINE` pragma directive and improves `ScalarQuantizer` performance with it.

Most of performance-critical methods for `ScalarQuantizer` are marked with this new directive, because a compiler (especially, an old one) may be unable to inline it properly. In some of my GCC experiments, such an inlining yields +50% queries per second in a search.

Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3141

Reviewed By: algoriddle

Differential Revision: D51615609

Pulled By: mdouze

fbshipit-source-id: 9c755c3e1a289b5d498306c1b9d6fcc21b0bec28
2023-11-28 10:34:38 -08:00
Matthijs Douze
43f8220818 fix scopedeleter diff
Summary: It seems that for some build modes, swig chokes on static_assert, so protect this with #idndef SWIG. Let's see what the tests say....

Reviewed By: algoriddle

Differential Revision: D50971042

fbshipit-source-id: 83e2ccb464c0bd024cbf3a494357147d75a76ca2
2023-11-28 09:42:57 -08:00