Summary:
The results returned by `NSG::search` are already sorted. Calling `maxheap_reorder` will make the results unorder.
Fixed https://github.com/facebookresearch/faiss/issues/2081.
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2086
Test Plan: buck test //faiss/tests/:test_index -- test_order
Reviewed By: beauby
Differential Revision: D32593924
Pulled By: mdouze
fbshipit-source-id: 794b94681610657bd2f305f7e3d6cd5d25c6bdba
Summary:
This diff fixes a serious bug in the range search implementation.
During range search in a flat index, (exhaustive_L2sqr_seq and exhaustive_inner_product_seq) when running in multiple threads, the per-thread results are collected into RangeSearchPartialResult structures.
When the computation is finished, they are aggregated into a RangeSearchResult. In the previous version of the code, this loop was nested into a second loop that is used to check for KeyboardInterrupts. Thus, at each iteration, the results were overwritten.
The fix removes the outer loop. It is most likely useless anyways because the sequential code is called only for a small number of queries, for a larger number the BLAS version is used.
Reviewed By: wickedfoo
Differential Revision: D28486415
fbshipit-source-id: 89a52b17f6ca1ef68fc5e758f0e5a44d0df9fe38
Summary:
## Description:
This diff implemented Navigating Spreading-out Graph (NSG) which accepts a KNN graph as input.
Here is the interface of building an NSG graph:
``` c++
void IndexNSG::build(idx_t n, const float *x, idx_t *knn_graph, int GK);
```
where `GK` is the nb of neighbors per node and `knn_graph[i * GK + j]` is the j-th neighbor of node i.
The `add` method is not implemented yet.
The unit tests could be found in `tests/test_nsg.cpp`.
mdouze beauby Maybe I need some advice on how to design the interface and support python.
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/1707
Test Plan: buck test //faiss/tests/:test_index -- TestNSG
Reviewed By: beauby
Differential Revision: D26748498
Pulled By: mdouze
fbshipit-source-id: 3280f705fb1b5f9c8cc5efeba63b904c3b832544
Summary: Checking for invalid parameters (number of nearest neighbors and number of probes where applicable) in the indices and throwing. Along with unit tests.
Reviewed By: wickedfoo
Differential Revision: D26582467
fbshipit-source-id: e345635d2f0f44ddcecc3f3314b2b9113359a787
Summary:
This diff streamlines the code that collects results for brute force distance computations for the L2 / IP and range search / knn search combinations.
It introduces a `ResultHandler` template class that abstracts what happens with the computed distances and ids. In addition to the heap result handler and the range search result handler, it introduces a reservoir result handler that improves the search speed for large k (>=100).
Benchmark results (https://fb.quip.com/y0g1ACLEqJXx#OCaACA2Gm45) show that on small datasets (10k) search is 10-50% faster (improvements are larger for small k). There is room for improvement in the reservoir implementation, whose implementation is quite naive currently, but the diff is already useful in its current form.
Experiments on precomputed db vector norms for L2 distance computations were not very concluding performance-wise, so the implementation is removed from IndexFlatL2.
This diff also removes IndexL2BaseShift, which was never used.
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/1502
Test Plan:
```
buck test //faiss/tests/:test_product_quantizer
buck test //faiss/tests/:test_index -- TestIndexFlat
```
Reviewed By: wickedfoo
Differential Revision: D24705464
Pulled By: mdouze
fbshipit-source-id: 270e10b19f3c89ed7b607ec30549aca0ac5027fe
Summary:
This diff fixes https://github.com/facebookresearch/faiss/issues/1412
There were various inconsistencies in how the shard and replica wrappers updated their internal state as the sub-indices were updated. This makes the two container classes work in the same way with similar synchronization functionality.
Reviewed By: beauby
Differential Revision: D23974186
fbshipit-source-id: c688c0c9124f823e4239aa2ff617b007b4564859
Summary:
`long` is 32 bits on windows and so is the default int type for numpy (eg. the one used for `np.arange`).
This diff explicitly specifies 64-bit ints for all occurrences where it matters.
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/1381
Reviewed By: wickedfoo
Differential Revision: D23371232
Pulled By: mdouze
fbshipit-source-id: 220262cd70ee70379f83de93561a4eae71c94b04
Changelog:
- changed license: BSD+Patents -> MIT
- propagates exceptions raised in sub-indexes of IndexShards and IndexReplicas
- support for searching several inverted lists in parallel (parallel_mode != 0)
- better support for PQ codes where nbit != 8 or 16
- IVFSpectralHash implementation: spectral hash codes inside an IVF
- 6-bit per component scalar quantizer (4 and 8 bit were already supported)
- combinations of inverted lists: HStackInvertedLists and VStackInvertedLists
- configurable number of threads for OnDiskInvertedLists prefetching (including 0=no prefetch)
- more test and demo code compatible with Python 3 (print with parentheses)
- refactored benchmark code: data loading is now in a single file
Facebook sync (Mar 2019)
- MatrixStats object
- option to round coordinates during k-means optimization
- alternative option for search in HNSW
- moved stats and imbalance_factor of IndexIVF to InvertedLists object
- range search for IVFScalarQuantizer
- direct unit8 codec in ScalarQuantizer
- renamed IndexProxy to IndexReplicas and moved to main Faiss
- better support for PQ code assignment with external index
- support for IMI2x16 (4B virtual centroids!)
- support for k = 2048 search on GPU (instead of 1024)
- most CUDA mem alloc failures throw exceptions instead of terminating on an assertion
- support for renaming an ondisk invertedlists
- interrupt computations with ctrl-C in python
Features:
- automatic tracking of C++ references in Python
- non-intel platforms supported -- some functions optimized for ARM
- override nprobe for concurrent searches
- support for floating-point quantizers in binary indexes
Bug fixes:
- no more segfaults in python (I know it's the same as the first feature but it's important!)
- fix GpuIndexIVFFlat issues for float32 with 64 / 128 dims
- fix sharding of flat indexes on GPU with index_cpu_to_gpu_multiple
* Refactors Makefiles and add configure script.
* Give MKL higher priority in configure script.
* Clean up Linux example makefile.inc.
* Cleanup makefile.inc examples.
* Fix python clean Makefile target.
* Regen swig wrappers.
* Remove useless CUDAFLAGS variable.
* Fix python linking flags.
* Separate compile and link phase in python makefile.
* Add macro to look for swig.
* Add CUDA check in configure script.
* Cleanup make depend targets.
* Cleanup CUDA flags.
* Fix linking flags.
* Fix python GPU linking.
* Remove useless flags from python gpu module linking.
* Add check for cuda libs.
* Cleanup GPU targets.
* Clean up test target.
* Add cpu/gpu targets to python makefile.
* Clean up tutorial Makefile.
* Remove stale OS var from example makefiles.
* Clean up cuda example flags.
* moved most FAISS_ASSERT calls to C++ exceptions, and adjusted
memory allocation to avoid mem leaks
* added an IndexIVFScalarQuantizer type that offers an
intermediate compression between IVFFlat and IVFPQ
* support removal of indices in IndexIDMap / IndexFlat combination
* various fixes in GPU code
Check Deng
Matthijs Douze
Dikpal Reddy
Lucas Hosseini
Jeff Johnson
Lucas Hosseini