1 Commits

Author SHA1 Message Date
Maria
9a8b34e295 Offline IVF powered by faiss big batch search (#3175)
Summary:
This PR introduces the offline IVF (OIVF) framework  which contains some tooling to run search using IVFPQ indexes (plus OPQ pretransforms) for large batches of queries using [big_batch_search](https://github.com/mlomeli1/faiss/blob/main/contrib/big_batch_search.py) and GPU faiss. See the [README](https://github.com/mlomeli1/faiss/blob/oivf/demos/offline_ivf/README.md) for details about using this framework.

This PR includes the following unit tests, which can be run with the unittest library as so:
````
~/faiss/demos/offline_ivf$ python3 -m unittest tests/test_iterate_input.py -k test_iterate_back
````

In test_offline_ivf:
````
test_consistency_check
test_train_index
test_index_shard_equal_file_sizes
test_index_shard_unequal_file_sizes
 test_search
test_evaluate_without_margin
test_evaluate_without_margin_OPQ
test_evaluate_with_margin
test_split_batch_size_bigger_than_file_sizes
test_split_batch_size_smaller_than_file_sizes
test_split_files_with_corrupted_input_file
````

In test_iterate_input:
````
test_iterate_input_file_larger_than_batch
test_get_vs_iterate
test_iterate_back
````

Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3175

Reviewed By: algoriddle

Differential Revision: D52218447

Pulled By: mlomeli1

fbshipit-source-id: 78b12457c79b02eb2c9ae993560f2e295798e7e5
2023-12-18 15:09:31 -08:00