Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3703
Now that SVE PR has been merged, we can turn on SVE opt mode in CI
Reviewed By: ramilbakhshyiev
Differential Revision: D60457456
fbshipit-source-id: 053b1f8ac805afba9035095c5df811da05675a81
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3701
Gating ARM SVE behind the plain vanilla linux x64 build like all the other builds.
Reviewed By: mengdilin
Differential Revision: D60425535
fbshipit-source-id: f2e082fbaa6ea1e314ffe0e2e7260c8634cab989
Summary:
related: https://github.com/facebookresearch/faiss/issues/2884
This PR contains below changes:
- Add new optlevel `sve`
- ARM SVE is _extension_ of ARMv8, so it should be treated similar to AVX2 IMO
- Add targets for ARM SVE, `faiss_sve` and `swigfaiss_sve`
- These targets will be built when you give `-DFAISS_OPT_LEVEL=sve` at build time
- Design decision: Don't fix SVE register length.
- The python package of faiss is "fat binary" (for example, the package for avx2 contains `_swigfaiss_avx2.so` and `_swigfaiss.so`)
- SVE is scalable instruction set (= doesn't fix vector length), but actually we can specify the vector length at compile time.
- [with `-msve-vector-length=` option](https://developer.arm.com/documentation/101726/4-0/Coding-for-Scalable-Vector-Extension--SVE-/SVE-Vector-Length-Specific--VLS--programming)
- When this option is specified, the binary can't work correctly on the CPU which has other vector length rather than specified at compile time
- When we use fixed vector length, SVE-supported faiss python package will contain 7 shared libraries like `_swigfaiss.so` , `_swigfaiss_sve.so` , `_swigfaiss_sve128.so` , `_swigfaiss_sve256.so` , `_swigfaiss_sve512.so` , `_swigfaiss_sve1024.so` , and `_swigfaiss_sve2048.so` . The package size will be exploded.
- For these reason, I don't specify the vector length at compile time and `faiss_sve` detects the vector length at run time.
- Add a mechanism of detecting ARM SVE on runtime environment and importing `swigfaiss_sve` dynamically
- Currently it only supports Linux, but there is no SVE environment with non-Linux OS now, as far as I know
NOTE: I plan to make one more PR about add some SVE implementation after this PR merged. This PR only contains adding sve target.
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2886
Reviewed By: ramilbakhshyiev
Differential Revision: D60386983
Pulled By: mengdilin
fbshipit-source-id: 7e66162ee53ce88fbfb6636e7bf705b44e6c3282
Summary:
Add instructions to download arm64 specific conda dependencies and cmake command and run it on CI. This should prepare us to turn on CI with SVE optimization
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3653
Reviewed By: ramilbakhshyiev
Differential Revision: D60043435
Pulled By: mengdilin
fbshipit-source-id: d81bb1c1022681c3da8f98bbf080d5e1d65d6b80
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3688
Looks like our previous changes only modified the cpp API. Not the c_api like the request wanted. This attempts to add faiss_get_version to c_api
Reviewed By: ramilbakhshyiev
Differential Revision: D60207739
fbshipit-source-id: 07184aeae92a154bb3f440279595077f002851f3
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3687
We will write a warning if nbits is not specified while using index factory with LSH. The warning lets users know we will be using default d as nbits.
Reviewed By: ramilbakhshyiev
Differential Revision: D60187935
fbshipit-source-id: 0fa960eeed615d857add77fa131a4cfa1989809d
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3679
T195237796 Claims we should be able to incldue nbits in the LSH factory string.
Their example is:
```
index = faiss.index_factory(128, 'LSH16rt')
Returns the following error.
faiss/index_factory.cpp:880: could not parse index string LSHrt_16
```
This is my first attempt at modifying the regex to accept an integer for nbits. Can an expert help me understand what the domain of accepted strings should be so I can modify the regex as necessary?
Reviewed By: ramilbakhshyiev
Differential Revision: D60054776
fbshipit-source-id: e47074eb9986b7c1c702832fc0bf758f60f45290
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3635
Add a util function to return the version in the c api.
Reviewed By: ramilbakhshyiev, fxdawnn
Differential Revision: D59817407
fbshipit-source-id: ca805f8e04f554d0294ba9da8ec6dc7c31e91fe3
Summary:
With the refactoring diff landed in [D59650573](https://www.internalfb.com/diff/D59650573), we take in the instance type string for aws and register it as the label for the runner. There was a typo in the previous registration (`m7i.xlarge` instead of `m7i.large`). The refactoring diff fixed the typo from the runner registration side but we need to fix it on the CI side as well so CI can find the runners
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3649
Test Plan: CI's AVX512 run on Github Action should succeed instead of stuck in pending
Reviewed By: ramilbakhshyiev
Differential Revision: D59916841
Pulled By: mengdilin
fbshipit-source-id: 5403e1a449ed765f0bf1a00e4a0d81d6c97463b9
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3608
This is a straightforward implementation of QINCo in CPU Faiss, with encoding and decoding capabilities (not training).
For this, we translate a simplified version of some torch classes:
- tensors, restricted to 2D and int32 + float32
- Linear and Embedding layer
Then the QINCoStep and QINCo can just be defined as C++ objects that are copy-constructable.
There is some plumbing required in the wrapping layers to support the integration. Pytroch tensors are converted to numpy for getting / setting them in C++.
Reviewed By: asadoughi
Differential Revision: D59132952
fbshipit-source-id: eea4856507a5b7c5f219efcf8d19fe56944df088
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3611
Using the new dispatcher functions, add search func to flat codes. To test it, make IndexLattice a subclass of FlatCodes and check the resonstruction there.
Reviewed By: asadoughi
Differential Revision: D59367989
fbshipit-source-id: 405dab4358fe34b2e38ac8bcc222b19f58643229
Summary: Add dispatcher function to avoid repeating dispatching code for distance computation and result handlers.
Reviewed By: asadoughi
Differential Revision: D59318865
fbshipit-source-id: 59046ede02f71a0da3b8061289fc70306bf875cb
Summary:
Start beta-testing AVX512 Build on self-hosted github runner with label faiss-aws-m7i.xlarge. This build is non-blocking on the PRs right now (via the parameter `continue-on-error`)
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3602
Reviewed By: ramilbakhshyiev
Differential Revision: D59290555
Pulled By: mengdilin
fbshipit-source-id: 6a82538bd44fecf4913885ac3b1999eb0fa52047
Summary:
The issue was that `uniform_int_distribution` generates numbers in the range `[0, ntotal]` and not `[0, ntotal)`, which was an oversight on my part. This PR also attempts to reduce the tolerance for `copyTo` tests as we have seen those fail intermittently.
cc ramilbakhshyiev mdouze cjnolet
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3552
Reviewed By: junjieqi
Differential Revision: D59097786
Pulled By: ramilbakhshyiev
fbshipit-source-id: 9dac4367e25c6c219b116ed172089a2fa2a39c4f
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3527
**Context**
Design Doc: [Faiss Benchmarking](https://docs.google.com/document/d/1c7zziITa4RD6jZsbG9_yOgyRjWdyueldSPH6QdZzL98/edit)
**In this diff**
1. Be able to reference codec and index from blobstore (bucket & path) outside the experiment
2. To support #1, naming is moved to descriptors.
3. Build index can be written as well.
4. You can run benchmark with train and then refer it in index built and then refer index built in knn search. Index serialization is optional. Although not yet exposed through index descriptor.
5. Benchmark can support index with different datasets sizes
6. Working with varying dataset now support multiple ground truth. There may be small fixes before we could use this.
7. Added targets for bench_fw_range, ivf, codecs and optimize.
**Analysis of ivf result**: D58823037
Reviewed By: algoriddle
Differential Revision: D57236543
fbshipit-source-id: ad03b28bae937a35f8c20f12e0a5b0a27c34ff3b
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3531
**In this diff**
1. I have add bench_fw to bento faiss kernel target
2. First part of notebook is changed to analyze sift1M results
Reviewed By: algoriddle
Differential Revision: D58823037
fbshipit-source-id: a67d4638af4368f0575bd289ce7aff8cf1fcd38b
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3524
Searches with the metric abs(dot(query, database))
This makes it possible to search vectors that are closest to a hyperplane
* adds support for alternative metrics in faiss.knn in python
* checks that it works with HNSW
* simplifies the extra distances interface by removing the template on
Reviewed By: asadoughi
Differential Revision: D58695971
fbshipit-source-id: 2a0ff49c7f7ac2c005d85f141cc5de148081c9c4
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3519
Fixing the conda conflicts because of `_openmp_mutex` build versions. This change pins that version for RAFT conda package builds.
Reviewed By: algoriddle
Differential Revision: D58646659
fbshipit-source-id: 4c1eaa9f08bd354da016b9399a36698007a497d8
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3516
This diff seeks to simplify the steps that install conda packages and environment configuration into a single step at the start of the cmake build action.
Reviewed By: mnorris11
Differential Revision: D58560454
fbshipit-source-id: ee2c6b36865809f31eb335cfb3c2fffdccaa318d
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3512
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3510
GitHub hosted runners some with the build-essentials package pre-installed, self-hosted runners on AWS do not have this package. This made it all steps other than the `all targets` one fall back to the system executables which unintentially worked on GitHub hosted runners but not on the self-hosted ones. This diff fixes it by pulling the line that adds conda bin to path early in the cmake build action.
Reviewed By: asadoughi
Differential Revision: D58513853
fbshipit-source-id: 23e95459e0031c96bd142515db07d1b700d713cf
Summary:
Adding build folder to gitignore, so that they don't show up in the commit tree while building from source
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3492
Reviewed By: junjieqi
Differential Revision: D58171359
Pulled By: asadoughi
fbshipit-source-id: b0efed348769328a3bdbcc13098dcb84cadb6c4f
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3494
This tasks focus on the refine index construction tutorial with different index refinement on fp16/sq8 quantization. The python version was added a while ago.
Reviewed By: junjieqi
Differential Revision: D58161983
fbshipit-source-id: 1c598fe612b5dee3952c5f7398e6802e117f141d
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3487
Sometimes it is not useful to serialize the storage index along with a HNSW index. This diff adds a flag that supports skipping the storage of the index.
Searchign and adding to the index is not possible until a storage index is added back in.
Reviewed By: junjieqi
Differential Revision: D57911060
fbshipit-source-id: 5a4ceee4a8f53f6f746df59af3942b813a99c14f
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3423
Adding small fixes to run experiments from fbcode.
1. Added buck target
2. Full import path of faiss bench_fw modules
3. new dataset path to run tests locally as we can't use an existing directory ./data in fbcode.
Reviewed By: algoriddle, junjieqi
Differential Revision: D57235092
fbshipit-source-id: f78a23199e619b640a19ca37f8b52ff0abdd8298
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3422
Found vec_io failing when running some benchmarking.
There is no such field named big_endian in sys. So, reverting it to original field byteorder
Reviewed By: algoriddle
Differential Revision: D56718607
fbshipit-source-id: 553f1d2d6bc967581142a92282e534f3f164e8f9
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3480
This tutorial summarize the methods to construct different indexs for PQFastScan refinement. It shows how the choice can impact on accuracy.
Reviewed By: junjieqi
Differential Revision: D57799598
fbshipit-source-id: a75c52c60a5217366f3361676da8f03f0c4a9feb
Summary:
Small Raft related modification to StandardGpuResources:
if the stream for a particular device is modified by a user, delete the Raft handle for that device. On any subsequent call to `getRaftHandle(device)`, a new raft handle with the updated stream will be created.
Closes https://github.com/facebookresearch/faiss/issues/3424
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3435
Reviewed By: ramilbakhshyiev
Differential Revision: D57640976
Pulled By: junjieqi
fbshipit-source-id: 41e2898a39250b7e52e920b71e819fc21ca9fc85
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3479
`-Wextra-semi` or `-Wextra-semi-stmt`
If the code compiles, this is safe to land.
Reviewed By: palmje
Differential Revision: D57632759
fbshipit-source-id: 48bc23e87b3f518182085124c4c8e68ddbb3ca8f
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3476
The long tail will still be the RAFT build but we can save on cost if the build has errors by incurring ~10m penalty added by waiting for the basic cmake build to complete. Both GPU and RAFT builds will start together so this will take less time overall to complete.
Reviewed By: algoriddle
Differential Revision: D57707298
fbshipit-source-id: 3589842e9bda9ebca9b25e089e6177fe96b6a0f5
