Matthijs Douze 8eab15eca3 LUT based search for additive quantizers (#1908) ...

Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/1908

To search the best combination of codebooks, the method that was implemented so far is via a beam search.

It is possible to make this faster for a query vector q by precomputing look-up tables in the form of

LUT_m = <q, cent_m>

where cent_m is the set of centroids for quantizer m=0..M-1.

The LUT can then be used as

inner_prod = sum_m LUT_m[c_m]

and

L2_distance = norm_q + norm_db - 2 * inner_prod

This diff implements this computation by:

- adding the LUT precomputation

- storing an exhaustive table of all centroid norms (when using L2)

This is only practical for small additive quantizers, eg. when a residual vector quantizer is used as coarse quantizer (ResidualCoarseQuantizer).

This diff is based on AdditiveQuantizer diff because it applies equally to other quantizers (eg. the LSQ).

Reviewed By: sc268

Differential Revision: D28467746

fbshipit-source-id: 82611fe1e4908c290204d4de866338c622ae4148

2021-05-25 01:54:53 -07:00

..

CMakeLists.txt

Add separate targets for libfaiss/libfaiss_avx2. (#1772)

2021-03-26 14:28:16 -07:00

common_faiss_tests.py

Add manifold check for size 0 (#1867)

2021-05-09 22:30:31 -07:00

test_autotune.py

PQ4 fast scan benchmarks (#1555)

2020-12-16 01:18:58 -08:00

test_binary_factory.py

…

test_binary_flat.cpp

Enable clang-format + autofix.

2021-02-25 04:46:10 -08:00

test_binary_hashindex.py

Add manifold check for size 0 (#1867)

2021-05-09 22:30:31 -07:00

test_binary_io.py

…

test_build_blocks.py

Add manifold check for size 0 (#1867)

2021-05-09 22:30:31 -07:00

test_clustering.py

Add manifold check for size 0 (#1867)

2021-05-09 22:30:31 -07:00

test_contrib.py

Add manifold check for size 0 (#1867)

2021-05-09 22:30:31 -07:00

test_dealloc_invlists.cpp

Enable clang-format + autofix.

2021-02-25 04:46:10 -08:00

test_documentation.py

Add docstrings for most of the replaced index methods

2021-03-05 17:19:17 -08:00

test_extra_distances.py

Add manifold check for size 0 (#1867)

2021-05-09 22:30:31 -07:00

test_factory.py

Support NSG in the index factory (#1758)

2021-03-23 07:28:20 -07:00

test_fast_scan_ivf.py

Re-enable fast scan on Windows tests (#1663)

2021-02-03 07:48:52 -08:00

test_fast_scan.py

Add tests for avx2 building (#1905)

2021-05-20 22:16:06 -07:00

test_index_accuracy.py

Add manifold check for size 0 (#1867)

2021-05-09 22:30:31 -07:00

test_index_binary_from_float.py

Fix inefficient float / binary conversion

2021-03-01 21:43:31 -08:00

test_index_binary.py

Add manifold check for size 0 (#1867)

2021-05-09 22:30:31 -07:00

test_index_composite.py

Add manifold check for size 0 (#1867)

2021-05-09 22:30:31 -07:00

test_index.py

Range search bug

2021-05-17 23:10:20 -07:00

test_io.py

Add manifold check for size 0 (#1867)

2021-05-09 22:30:31 -07:00

test_ivflib.py

Handle the situation where nprobe > nlist in IndexBinaryIVF (#1695)

2021-02-23 12:20:37 -08:00

test_ivfpq_codec.cpp

Enable clang-format + autofix.

2021-02-25 04:46:10 -08:00

test_ivfpq_indexing.cpp

Enable clang-format + autofix.

2021-02-25 04:46:10 -08:00

test_lowlevel_ivf.cpp

Enable clang-format + autofix.

2021-02-25 04:46:10 -08:00

test_lsq.py

Add LocalSearchQuantizer (#1906)

2021-05-21 01:33:55 -07:00

test_merge.cpp

Enable clang-format + autofix.

2021-02-25 04:46:10 -08:00

test_meta_index.py

Add manifold check for size 0 (#1867)

2021-05-09 22:30:31 -07:00

test_omp_threads_py.py

…

test_omp_threads.cpp

Enable clang-format + autofix.

2021-02-25 04:46:10 -08:00

test_ondisk_ivf.cpp

Enable clang-format + autofix.

2021-02-25 04:46:10 -08:00

test_oom_exception.py

Portable SWIG Vectors (#1742)

2021-04-02 07:11:47 -07:00

test_pairs_decoding.cpp

Enable clang-format + autofix.

2021-02-25 04:46:10 -08:00

test_params_override.cpp

Enable clang-format + autofix.

2021-02-25 04:46:10 -08:00

test_partition.py

Fix unused variables in python

2021-02-24 11:52:18 -08:00

test_pq_encoding.cpp

Enable clang-format + autofix.

2021-02-25 04:46:10 -08:00

test_product_quantizer.py

Use BLAS to compute sdc table (#1809)

2021-04-16 00:17:51 -07:00

test_referenced_objects.py

…

test_refine.py

PQ4 fast scan benchmarks (#1555)

2020-12-16 01:18:58 -08:00

test_residual_quantizer.py

LUT based search for additive quantizers (#1908)

2021-05-25 01:54:53 -07:00

test_sliding_ivf.cpp

Enable clang-format + autofix.

2021-02-25 04:46:10 -08:00

test_standalone_codec.py

Add manifold check for size 0 (#1867)

2021-05-09 22:30:31 -07:00

test_threaded_index.cpp

Enable clang-format + autofix.

2021-02-25 04:46:10 -08:00

test_transfer_invlists.cpp

Enable clang-format + autofix.

2021-02-25 04:46:10 -08:00

torch_test_contrib.py

…