Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2123
One of the encodings used by LCC is based on a RCQ coarse quantizer and a "payload" of ITQ. The codes are compared with Hamming distances.
The index type `IndexIVFSpectralHash` can be re-purposed to perfrorm this type of index.
This diff contains a small demo demo_rcq_itq script in python to show how:
* the RCQ + ITQ are trained
* the RCQ + ITQ index add and search work (with a very inefficient python implementation)
* they can be transferred to an `IndexIVFSpectralHash`
* the python implementation and `IndexIVFSpectralHash` give the same results
The advantage of using to an `IndexIVFSpectralHash` is that in C++ it offers an `InvertedListScanner` object that can be used to compute query to code distances with its `distance_to_code` method. This is generic and will generalize to other types of encodings and coarse quantizers.
What is missing is an index_factory to make instanciation easier.
Reviewed By: sc268
Differential Revision: D32642900
fbshipit-source-id: 284f3029d239b7946bbca44a748def4e058489bd
Summary:
This is required for the renaming of the default branch from `master` to `main`, in accordance with the new Facebook OSS guidelines.
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2029
Reviewed By: mdouze
Differential Revision: D30672862
Pulled By: beauby
fbshipit-source-id: 0b6458a4ff02a12aae14cf94057e85fdcbcbff96
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/1906
This PR implemented LSQ/LSQ++, a vector quantization technique described in the following two papers:
1. Revisiting additive quantization
2. LSQ++: Lower running time and higher recall in multi-codebook quantization
Here is a benchmark running on SIFT1M for 64 bits encoding:
```
===== lsq:
mean square error = 17335.390208
training time: 312.729779958725 s
encoding time: 244.6277096271515 s
===== pq:
mean square error = 23743.004672
training time: 1.1610801219940186 s
encoding time: 2.636141061782837 s
===== rq:
mean square error = 20999.737344
training time: 31.813055515289307 s
encoding time: 307.51959800720215 s
```
Changes:
1. Add LocalSearchQuantizer object
2. Fix an out of memory bug in ResidualQuantizer
3. Add a benchmark for evaluating quantizers
4. Add tests for LocalSearchQuantizer
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/1862
Test Plan:
```
buck test //faiss/tests/:test_lsq
buck run mode/opt //faiss/benchs/:bench_quantizer -- lsq pq rq
```
Reviewed By: beauby
Differential Revision: D28376369
Pulled By: mdouze
fbshipit-source-id: 2a394d38bf75b9de0a1c2cd6faddf7dd362a6fa8
Summary: The synthetic dataset can now have IP groundtruth
Reviewed By: wickedfoo
Differential Revision: D24219860
fbshipit-source-id: 42e094479311135e932821ac0a97ed0fb237bf78
Summary:
This diff adds an object for a few useful dataset in faiss.contrib.
This includes synthetic datasets and the classic ones.
It is intended to work on:
- the FAIR cluster
- gluster
- manifold
Reviewed By: wickedfoo
Differential Revision: D23378763
fbshipit-source-id: 2437a7be9e712fd5ad1bccbe523cc1c936f7ab35
Matthijs Douze
Lucas Hosseini
Chengqi Deng
Lucas Hosseini