mirror of https://github.com/facebookresearch/faiss.git synced 2025-06-03 21:54:02 +08:00

History

Matthijs Douze 838612c9d7 torch.distributed kmeans (#3876 )

Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3876

Demo script for distributed kmeans. It provides a `DatasetAssign` object and shows how to run it with torch.distributed.

Reviewed By: asadoughi, pankajsingh88

Differential Revision: D63013820

fbshipit-source-id: 22c959f3afdc04fd4aa8b9aeed309ea6290b1328

2024-09-20 09:15:27 -07:00

offline_ivf

…

rocksdb_ivf

Fixing headers as per OSS requirement (#3847 )

2024-09-09 22:58:05 -07:00

CMakeLists.txt

…

demo_auto_tune.py

…

demo_client_server_ivf.py

…

demo_distributed_kmeans_torch.py

torch.distributed kmeans (#3876 )

2024-09-20 09:15:27 -07:00

demo_imi_flat.cpp

…

demo_imi_pq.cpp

…

demo_ivfpq_indexing.cpp

…

demo_nndescent.cpp

…

demo_ondisk_ivf.py

…

demo_qinco.py

…

demo_residual_quantizer.cpp

…

demo_sift1M.cpp

…

demo_weighted_kmeans.cpp

…

README.md

…

README.md

Demos for a few Faiss functionalities

demo_auto_tune.py

Demonstrates the auto-tuning functionality of Faiss

demo_ondisk_ivf.py

Shows how to construct a Faiss index that stores the inverted file data on disk, eg. when it does not fit in RAM. The script works on a small dataset (sift1M) for demonstration and proceeds in stages:

0: train on the dataset

1-4: build 4 indexes, each containing 1/4 of the dataset. This can be done in parallel on several machines

5: merge the 4 indexes into one that is written directly to disk (needs not to fit in RAM)

6: load and test the index