Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3455
Code quality control by reducing the number of prints
Reviewed By: junjieqi
Differential Revision: D57502194
fbshipit-source-id: a6cd65ed4cc49590ce73d2978d41b640b5259c17
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3354
**Change was previously reverted because of build failure as change D55577576 removed the definition of FAISS_THROW_IF_MSG**
**Context**
[Issue 3128](https://github.com/facebookresearch/faiss/issues/3128) is an enhancement request to support remove_ids for IVFPQFastScan.
Existing mechanism use direct map and iterate over items in selector and use scopecodes and scopeIds to replace item to be removed. Given that codes are packed, it is hard to return single code how it is packed in CodePackerPQ4. Thus, we need a custom implementation to removed_ids.
**In this diff**,
1. We have added custom implementation of remove_ids from BlockInvertedLists which unpack code as it iterate and repack in new position. DirectMap use this remove_id function in BlockInvertedLists for type NoMap in DirectMap.
2. Also, we are throwing exception for other map type in DirectMap i.e. HashTable
Reviewed By: ramilbakhshyiev
Differential Revision: D55858959
fbshipit-source-id: c8a0631495380b7dead36720e4507f4d1900d39f
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3349
**Context**
[Issue 3128](https://github.com/facebookresearch/faiss/issues/3128) is an enhancement request to support remove_ids for IVFPQFastScan.
Existing mechanism use direct map and iterate over items in selector and use scopecodes and scopeIds to replace item to be removed. Given that codes are packed, it is hard to return single code how it is packed in CodePackerPQ4. Thus, we need a custom implementation to removed_ids.
**In this diff**,
1. We have added custom implementation of remove_ids from BlockInvertedLists which unpack code as it iterate and repack in new position. DirectMap use this remove_id function in BlockInvertedLists for type NoMap in DirectMap.
2. Also, we are throwing exception for other map type in DirectMap i.e. HashTable
Reviewed By: mdouze
Differential Revision: D55723390
fbshipit-source-id: 0017b556bd790765251e778ac48ed37ff3a29a45
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2625
This diff introduces a new abstraction for the code layouts that are not simply flat one after another.
The packed codes are assumed to be packed together in fixed-size blocks. Hence, code `#i` is stored at offset `i % nvec` of block `floor(i / nvec)`. Each block has size `block_size`.
The `CodePacker` object takes care of the translation between packed and flat codes. The packing / unpacking functions are virtual functions now, but they could as well be inlined for performance.
The `CodePacker` object makes it possible to do manipulations onarrays of codes (including inverted lists) in a uniform way, for example merging / adding / updating / removing / converting to&from CPU.
In this diff, the only non-trivial CodePacker implemnted is for the FastScan code. The new functionality supported is merging IVFFastScan indexes.
Reviewed By: alexanderguzhva
Differential Revision: D42072972
fbshipit-source-id: d1f8bdbcf7ab0f454b5d9c37ba2720fd191833d0
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2618
The Faiss tests run in dev mode are very slow
The PQ polysemous training is particularly sensitive to this with the default settings.
This diff adds a "np" suffix to two PQ factory strings to disable polysemous training. The tests that are detected as flaky because they occasionally time out.
Reviewed By: alexanderguzhva
Differential Revision: D41955699
fbshipit-source-id: b1e0382a0142a3ed28b498c5ea6f5499de2c1b3f
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2533
Implements merge_from for IndexIDMap[2] and IndexPreTransform. In the process, split off IndexIDMap to their own .h/.cpp files.
Reviewed By: alexanderguzhva
Differential Revision: D40420373
fbshipit-source-id: 1570a460706dd3fbc1447f9fcc0e2721eab869bb
Summary:
* Modify pq4_get_paked_element to make it not depend on an auxiliary table
* Create pq4_set_packed_element which sets a single element in codes in packed format
(These methods would be used in merge and remove for IndexFastScan
get method is also used in FastScan indices for reconstruction)
* Add remove feature for IndexFastScan
* Add merge feature for indexFast Scan
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2497
Test Plan:
cd build && make -j
make test
cd faiss/python && python setup.py build && cd ../../..
PYTHONPATH="$(ls -d ./build/faiss/python/build/lib*/)" pytest tests/test_*.py
Reviewed By: mdouze
Differential Revision: D39927403
Pulled By: mdouze
fbshipit-source-id: 45271b98419203dfb1cea4f4e7eaf0662523a5b5
Summary:
support merge for all IndexFlatCodes children
make merge_from and check_compatible_for_merge methods of Index and IndexIVF and IndexFlatCodes(the only supported types) inherit them from Index.
This is part 1 of 2 as merge_into still not updated
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2488
Test Plan:
cd build
make -j
make test
cd faiss/python && python setup.py build
cd ../../..
PYTHONPATH="$(ls -d ./build/faiss/python/build/lib*/)" pytest tests/test_*.py
# fbcode
buck test //faiss/tests/:test_index_merge
buck test //faiss/tests/:test_io
Reviewed By: mdouze
Differential Revision: D39726378
Pulled By: AbdelrahmanElmeniawy
fbshipit-source-id: 6739477fddcad3c7a990f3aae9be07c1b2b74fef
