mirrors
/
faiss
mirror of https://github.com/facebookresearch/faiss.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
faiss/contrib
History
Matthijs Douze c5975cda72 PQ4 fast scan benchmarks (#1555) 
Summary:
Code + scripts for Faiss benchmarks around the  Fast scan codes.

Pull Request resolved: https://github.com/facebookresearch/faiss/pull/1555

Test Plan: buck test //faiss/tests/:test_refine

Reviewed By: wickedfoo

Differential Revision: D25546505

Pulled By: mdouze

fbshipit-source-id: 902486b7f47e36221a2671d124df8c114f25db58
 		2020-12-16 01:18:58 -08:00
..
README.md PQ4 fast scan benchmarks (#1555) 2020-12-16 01:18:58 -08:00
client_server.py Add missing copyright headers. (#1460) 2020-10-13 11:15:59 -07:00
datasets.py PQ4 fast scan benchmarks (#1555) 2020-12-16 01:18:58 -08:00
exhaustive_search.py PQ4 fast scan benchmarks (#1555) 2020-12-16 01:18:58 -08:00
factory_tools.py PQ4 fast scan benchmarks (#1555) 2020-12-16 01:18:58 -08:00
inspect_tools.py Implementation of PQ4 search with SIMD instructions (#1542) 2020-12-03 10:06:38 -08:00
ondisk.py Add missing copyright headers. (#1460) 2020-10-13 11:15:59 -07:00
rpc.py Add missing copyright headers. (#1460) 2020-10-13 11:15:59 -07:00
torch_utils.py PyTorch tensor / Faiss index interoperability (#1484) 2020-10-23 22:24:22 -07:00
vecs_io.py PQ4 fast scan benchmarks (#1555) 2020-12-16 01:18:58 -08:00

README.md

The contrib modules

The contrib directory contains helper modules for Faiss for various tasks.

Code structure

The contrib directory gets compiled in the module faiss.contrib. Note that although some of the modules may depend on additional modules (eg. GPU Faiss, pytorch, hdf5), they are not necessarily compiled in to avoid adding dependencies. It is the user's responsibility to provide them.

In contrib, we are progressively dropping python2 support.

List of contrib modules

rpc.py

A very simple Remote Procedure Call library, where function parameters and results are pickled, for use with client_server.py

client_server.py

The server handles requests to a Faiss index. The client calls the remote index. This is mainly to shard datasets over several machines, see Distributd index

ondisk.py

Encloses the main logic to merge indexes into an on-disk index. See On-disk storage

exhaustive_search.py

Computes the ground-truth search results for a dataset that possibly does not fit in RAM. Uses GPU if available. Tested in tests/test_contrib.TestComputeGT

torch_utils.py

Interoperability functions for pytorch and Faiss: Importing this will allow pytorch Tensors (CPU or GPU) to be used as arguments to Faiss indexes and other functions. Torch GPU tensors can only be used with Faiss GPU indexes. If this is imported with a package that supports Faiss GPU, the necessary stream synchronization with the current pytorch stream will be automatically performed.

Numpy ndarrays can continue to be used in the Faiss python interface after importing this file. All arguments must be uniformly either numpy ndarrays or Torch tensors; no mixing is allowed.

Tested in tests/test_contrib_torch.py (CPU) and gpu/test/test_contrib_torch_gpu.py (GPU).

inspect_tools.py

Functions to inspect C++ objects wrapped by SWIG. Most often this just means reading fields and converting them to the proper python array.

datasets.py

(may require h5py)

Defintion of how to access data for some standard datsets.

factory_tools.py

Functions related to factory strings.