Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2625
This diff introduces a new abstraction for the code layouts that are not simply flat one after another.
The packed codes are assumed to be packed together in fixed-size blocks. Hence, code `#i` is stored at offset `i % nvec` of block `floor(i / nvec)`. Each block has size `block_size`.
The `CodePacker` object takes care of the translation between packed and flat codes. The packing / unpacking functions are virtual functions now, but they could as well be inlined for performance.
The `CodePacker` object makes it possible to do manipulations onarrays of codes (including inverted lists) in a uniform way, for example merging / adding / updating / removing / converting to&from CPU.
In this diff, the only non-trivial CodePacker implemnted is for the FastScan code. The new functionality supported is merging IVFFastScan indexes.
Reviewed By: alexanderguzhva
Differential Revision: D42072972
fbshipit-source-id: d1f8bdbcf7ab0f454b5d9c37ba2720fd191833d0
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2618
The Faiss tests run in dev mode are very slow
The PQ polysemous training is particularly sensitive to this with the default settings.
This diff adds a "np" suffix to two PQ factory strings to disable polysemous training. The tests that are detected as flaky because they occasionally time out.
Reviewed By: alexanderguzhva
Differential Revision: D41955699
fbshipit-source-id: b1e0382a0142a3ed28b498c5ea6f5499de2c1b3f
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2533
Implements merge_from for IndexIDMap[2] and IndexPreTransform. In the process, split off IndexIDMap to their own .h/.cpp files.
Reviewed By: alexanderguzhva
Differential Revision: D40420373
fbshipit-source-id: 1570a460706dd3fbc1447f9fcc0e2721eab869bb
Summary:
* Modify pq4_get_paked_element to make it not depend on an auxiliary table
* Create pq4_set_packed_element which sets a single element in codes in packed format
(These methods would be used in merge and remove for IndexFastScan
get method is also used in FastScan indices for reconstruction)
* Add remove feature for IndexFastScan
* Add merge feature for indexFast Scan
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2497
Test Plan:
cd build && make -j
make test
cd faiss/python && python setup.py build && cd ../../..
PYTHONPATH="$(ls -d ./build/faiss/python/build/lib*/)" pytest tests/test_*.py
Reviewed By: mdouze
Differential Revision: D39927403
Pulled By: mdouze
fbshipit-source-id: 45271b98419203dfb1cea4f4e7eaf0662523a5b5
Summary:
support merge for all IndexFlatCodes children
make merge_from and check_compatible_for_merge methods of Index and IndexIVF and IndexFlatCodes(the only supported types) inherit them from Index.
This is part 1 of 2 as merge_into still not updated
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2488
Test Plan:
cd build
make -j
make test
cd faiss/python && python setup.py build
cd ../../..
PYTHONPATH="$(ls -d ./build/faiss/python/build/lib*/)" pytest tests/test_*.py
# fbcode
buck test //faiss/tests/:test_index_merge
buck test //faiss/tests/:test_io
Reviewed By: mdouze
Differential Revision: D39726378
Pulled By: AbdelrahmanElmeniawy
fbshipit-source-id: 6739477fddcad3c7a990f3aae9be07c1b2b74fef
Alexandr Guzhva
Matthijs Douze
Abdelrahman Elmeniawy