Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3975
Graph-based indices are often quite bulky in terms of storage becausee the out-degree of the edges is high (32 or more) and edges are just encoded as 32-bit ints.
This diff makes it possible to replace the graph structure of NSG with a compressed version, but only for search (the full graph needs to be present at addition time).
It is easier to do it for NSG than for HNSW for several reasons:
- NSG's graph is not hierarichal -- HNSW is and the edges for the different levels are interleaved
- the NSG graph object is already isolated well (thanks KingLittleQ!)
- NSG cannot be built incrementally so it is easier to convert the graph after all adds are done in one go.
The custom compressed graph is currently only implemented as a test, but could be integrated in the main Faiss as an option to NSG.
Reviewed By: algoriddle
Differential Revision: D64646137
fbshipit-source-id: c10c2a485b44561d32941ce1e7a0e3fe512cf0ac
