Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3812
Allows factory strings like `IVF3k,Flat` as a shorthand for 3072 centroids.
The main question is whether k or M should be metric (k=1000) or power of 2 (k=1024):
* pro-metric: standard,
* pro-power of 2: in practice we use powers of 2 most often
The suffixes ki and Mi should be used for powers of 2 but this makes the notation more heavy (which is what we wanted to avoid in the first place).
So I picked power of 2.
Reviewed By: mnorris11
Differential Revision: D62019941
fbshipit-source-id: f547962625123ecdfaa406067781c77386017793
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3679
T195237796 Claims we should be able to incldue nbits in the LSH factory string.
Their example is:
```
index = faiss.index_factory(128, 'LSH16rt')
Returns the following error.
faiss/index_factory.cpp:880: could not parse index string LSHrt_16
```
This is my first attempt at modifying the regex to accept an integer for nbits. Can an expert help me understand what the domain of accepted strings should be so I can modify the regex as necessary?
Reviewed By: ramilbakhshyiev
Differential Revision: D60054776
fbshipit-source-id: e47074eb9986b7c1c702832fc0bf758f60f45290
Summary: Code would crash when deallocating the coarse quantizer for a IVFSpectralHash.
Reviewed By: algoriddle
Differential Revision: D40053030
fbshipit-source-id: 6a2987a6983f0e5fc5c5b6296d9000354176af83
Summary:
makes index_factory support IDMap2 not only IDMap and add required tests
adding IDMap2 to index_factory would help users to take advantage of its extra features more than IDMap such as reconstruct the indices.
solves [issue 1864](https://github.com/facebookresearch/faiss/issues/1864)
+fix downcast_index IDMap / IDMap2 order
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2478
Test Plan:
cd build
make -j
cd faiss/python && python setup.py build
cd ../../..
PYTHONPATH="$(ls -d ./build/faiss/python/build/lib*/)" pytest tests/test_*.py
Reviewed By: mdouze
Differential Revision: D39660813
Pulled By: AbdelrahmanElmeniawy
fbshipit-source-id: 4881d325bb3b0eaf9637a544511d18c2084453eb
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2295
Makes a common ancestor for quantizer classes.
As a first application, adds a clone_Quantizer function
Reviewed By: alexanderguzhva
Differential Revision: D35561960
fbshipit-source-id: 896a4f3fc4ab992511cdc0642689a440f170f683
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2147
There was a bug in the OPQ string parsing. This diff adds a test and fixes the error.
Reviewed By: aijanai
Differential Revision: D33020167
fbshipit-source-id: 32e43653849b258a3b6d0cfdc44a6c637433f2c8
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2134
The old parsing was very complex and grew out of hand.
this diff just uses regex parsing.
Reviewed By: wickedfoo
Differential Revision: D32759110
fbshipit-source-id: 243029bba8a7fe70c71323f5edc7e2ce4e669757
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/1972
This fixes a few issues that I ran into + adds tests:
- range_search_max_results with IP search
- a few missing downcasts for VectorTRansforms
- ResultHeap supports max IP search
Reviewed By: wickedfoo
Differential Revision: D29525093
fbshipit-source-id: d4ff0aff1d83af9717ff1aaa2fe3cda7b53019a3
Summary:
## Description
This PR added NSG into the index factory. Here are the supported index strings:
1. `NSG{0}` or `NSG{0},Flat`: Create an IndexNSGFlat with `R = {0}`.
2. `IVF{0}_NSG{1},{2}`: Create an IndexIVF using NSG as a coarse quantizer where `ncentroids = {0}`, `R = {1}` and `{2}` is the second level quantizer.
These two types of indexes may be the most useful ones. Other composite indexes could be supported in the future.
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/1758
Test Plan: buck test //faiss/tests/:test_factory
Reviewed By: beauby
Differential Revision: D27189479
Pulled By: mdouze
fbshipit-source-id: b60000f985c490ef2e7bc561b4e209f9f61c3cc8
Summary: Copy construction of Aligned table was wrong, which crashed cloning of IVFPQ.
Reviewed By: wickedfoo
Differential Revision: D26426400
fbshipit-source-id: 1d43ea6309d0a56eb592f9d6c5b52282f494e653
Summary:
IndexPQ and IndexIVFPQ implementations with AVX shuffle instructions.
The training and computing of the codes does not change wrt. the original PQ versions but the code layout is "packed" so that it can be used efficiently by the SIMD computation kernels.
The main changes are:
- new IndexPQFastScan and IndexIVFPQFastScan objects
- simdib.h for an abstraction above the AVX2 intrinsics
- BlockInvertedLists for invlists that are 32-byte aligned and where codes are not sequential
- pq4_fast_scan.h/.cpp: for packing codes and look-up tables + optmized distance comptuation kernels
- simd_result_hander.h: SIMD version of result collection in heaps / reservoirs
Misc changes:
- added contrib.inspect_tools to access fields in C++ objects
- moved .h and .cpp code for inverted lists to an invlists/ subdirectory, and made a .h/.cpp for InvertedListsIOHook
- added a new inverted lists type with 32-byte aligned codes (for consumption by SIMD)
- moved Windows-specific intrinsics to platfrom_macros.h
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/1542
Test Plan:
```
buck test mode/opt -j 4 //faiss/tests/:test_fast_scan_ivf //faiss/tests/:test_fast_scan
buck test mode/opt //faiss/manifold/...
```
Reviewed By: wickedfoo
Differential Revision: D25175439
Pulled By: mdouze
fbshipit-source-id: ad1a40c0df8c10f4b364bdec7172e43d71b56c34
Changelog:
- changed license: BSD+Patents -> MIT
- propagates exceptions raised in sub-indexes of IndexShards and IndexReplicas
- support for searching several inverted lists in parallel (parallel_mode != 0)
- better support for PQ codes where nbit != 8 or 16
- IVFSpectralHash implementation: spectral hash codes inside an IVF
- 6-bit per component scalar quantizer (4 and 8 bit were already supported)
- combinations of inverted lists: HStackInvertedLists and VStackInvertedLists
- configurable number of threads for OnDiskInvertedLists prefetching (including 0=no prefetch)
- more test and demo code compatible with Python 3 (print with parentheses)
- refactored benchmark code: data loading is now in a single file
* Refactors Makefiles and add configure script.
* Give MKL higher priority in configure script.
* Clean up Linux example makefile.inc.
* Cleanup makefile.inc examples.
* Fix python clean Makefile target.
* Regen swig wrappers.
* Remove useless CUDAFLAGS variable.
* Fix python linking flags.
* Separate compile and link phase in python makefile.
* Add macro to look for swig.
* Add CUDA check in configure script.
* Cleanup make depend targets.
* Cleanup CUDA flags.
* Fix linking flags.
* Fix python GPU linking.
* Remove useless flags from python gpu module linking.
* Add check for cuda libs.
* Cleanup GPU targets.
* Clean up test target.
* Add cpu/gpu targets to python makefile.
* Clean up tutorial Makefile.
* Remove stale OS var from example makefiles.
* Clean up cuda example flags.
Michael Norris
George Wang
Matthijs Douze
Bhavik Sheth
Rujin Cao
Abdelrahman Elmeniawy
CodemodService Bot
Matthijs Douze
Check Deng
Lucas Hosseini
Lucas Hosseini