3 Commits (79d94227b45c12a0259473531b5a1a621db3c02d)

Author	SHA1	Message	Date
Chengqi Deng	26abede812	Non-uniform quantization of vector norms (#2037) ... Summary: This diff implemented non-uniform quantization of vector norms in additive quantizers. index_factory and I/O are supported. index_factory: `XXX_Ncqint{nbits}` where `nbits` is the number of bits to quantize vector norm. For 8 bits code, it is almost the same as 8-bit uniform quantization. It will slightly improve the accuracy if the code size is less than 8 bits. ``` RQ4x8_Nqint8: R@1 0.1116 RQ4x8_Ncqint8: R@1 0.1117 RQ4x8_Nqint4: R@1 0.0901 RQ4x8_Ncqint4: R@1 0.0989 ``` Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2037 Test Plan: buck test //faiss/tests/:test_clustering -- TestClustering1D buck test //faiss/tests/:test_lsq -- test_index_accuracy_cqint buck test //faiss/tests/:test_residual_quantizer -- test_norm_cqint buck test //faiss/tests/:test_residual_quantizer -- test_search_L2 Reviewed By: beauby Differential Revision: D31083476 Pulled By: mdouze fbshipit-source-id: f34c3dafc4eb1c6f44a63e68137158911aa4a2f4	2021-10-11 14:13:16 -07:00
Matthijs Douze	2d380e992b	Add manifold check for size 0 (#1867) ... Summary: Pull Request resolved: https://github.com/facebookresearch/faiss/pull/1867 Merging code for the 1T photodna index seems to fail at https://www.internalfb.com/phabricator/paste/view/P412975011?lines=174 with ``` terminate called after throwing an instance of 'facebook::manifold::blobstore::StorageException' what(): [400] Begin offset and/or length were invalid -- Begin offset must be positive and length must be non-negative. Received: offset = 2642410612, length = 0 Aborted (core dumped) ``` traces back to https://www.internalfb.com/intern/diffusion/FBS/browsefile/master/fbcode/manifold/blobstore/BlobstoreThriftHandler.cpp?lines=671%2C700%2C732 There is a single case where we don't check if the read or write size is 0. So let's try this fix. In the process I realized that the Manifold tests were non functional due to a name collision on common.py. Also fix this in all dependent files. Differential Revision: D28231710 fbshipit-source-id: 700ffa6ca0c82c49e7d1eae9e76549ec5ff16332	2021-05-09 22:30:31 -07:00
Matthijs Douze	7559cf5c5b	add ResidualQuantizer ... Summary: This diff includes: - progressive dimension k-means. - the ResidualQuantizer object - GpuProgressiveDimIndexFactory so that it can be trained on GPU - corresponding tests - reference Python implementation of the same in scripts/matthijs/LCC_encoding Reviewed By: wickedfoo Differential Revision: D27608029 fbshipit-source-id: 9a8cf3310c8439a93641961ca8b042941f0f4249	2021-04-14 13:11:54 -07:00