Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3519
Fixing the conda conflicts because of `_openmp_mutex` build versions. This change pins that version for RAFT conda package builds.
Reviewed By: algoriddle
Differential Revision: D58646659
fbshipit-source-id: 4c1eaa9f08bd354da016b9399a36698007a497d8
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3466
Flattening Conda include directories breaks CUDA 11.4.4 build on Ubuntu 20 / v5 kernel. This change updates the logic to only flatten includes on Ubuntu 22 / v6 kernel (aka as running on GitHub Actions runners).
Reviewed By: algoriddle
Differential Revision: D57602154
fbshipit-source-id: 00c14ca7c64644b8b86483ac6b4d40c6d8f12372
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3459
When building with CUDA 11.4.4, CMake does not properly include files under Conda environment. This workaround flattens the include sub-directories in to the include root. It will unblock us for now while we are looking for a fix through CMakeLists files or figure out if it's a CMake bug and it gets fixed.
Reviewed By: junjieqi
Differential Revision: D57545169
fbshipit-source-id: 9cbdd0866e00e899cc889930a59448da55d873c2
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3440
This change is required to unblock the migration to GitHub Actions. `cuda-toolkit` was only specified in the `libfaiss` package and it was not available in `faiss-gpu` or `faiss-gpu-raft`. This currently works on CircleCI because the runner image has CUDA toolkit of the needed version installed on the system and the build logic falls back to that but breaks on GitHub Actions because their runner images do not come with CUDA toolkit pre-installed.
Reviewed By: junjieqi
Differential Revision: D57371597
fbshipit-source-id: 8bededd53e2528f033fac797b296d74b47f9403e
Summary:
The CMakeLists.txt in faiss/gpu uses the $<LINK_LIBRARY:WHOLE_ARCHIVE expression which requires at least cmake 3.24.
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3305
Reviewed By: mlomeli1
Differential Revision: D56234500
Pulled By: algoriddle
fbshipit-source-id: dfe7df3379c5250dedec7d1988cffa889fc1c393
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3295
In the past, we had build failure due to the latest swig version in conda-forge so we had to specify the version of swig. In this diff, we are going to change it to be the latest version always because the issue has gone.
Reviewed By: algoriddle
Differential Revision: D54975271
fbshipit-source-id: 7ca59fb58390edb0cc5ed52f6fd416f633dd7938
Summary:
Considering that only GPU classic works on Pascal and only if compiled with cuda 11.4, this sets up the conda builds as follows.
tl;dr: Pascal will only be supported by faiss-gpu on cuda 11.
## faiss-gpu
1. build on cuda 11.4.4, supports Pascal, works with pytorch-cuda=11 (including hosts with cuda 12 drivers)
2. build on cuda 12.1.1, does NOT support Pascal, works with pytorch-cuda=12
## faiss-gpu-raft
1. build on cuda 11.8.0, does NOT support Pascal, works with pytorch-cuda=11.8
2. build on cuda 12.1.1, does NOT support Pascal, works with pytorch-cuda=12
The reason faiss-gpu-raft is built with and supports only 11.8 is due to the risk of clobbering otherwise between libraft and pytorch-cuda dependencies.
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3237
Reviewed By: mdouze
Differential Revision: D53354121
Pulled By: algoriddle
fbshipit-source-id: 86108e975168759572f84922bf0ccef55ae75ef8
Summary:
1. Split the GPU code into a separate static library to reduce build times.
2. faiss-gpu now depends on pytorch-cuda to stop clobbering of CUDA libraries when Faiss and Pytorch installed in the same environment
3. Add CUDA 12 (we support both 11.8 and 12.1)
4. Add Python 3.12
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3222
Reviewed By: mlomeli1
Differential Revision: D53215033
Pulled By: algoriddle
fbshipit-source-id: d48c5707af20b7e5ff72c3aa27b0e677259f22f8
Summary:
Raft conda packages are built with conda-forge, which ships a newer version of swig, which broke the build. I'm pinning the version as a zero-effort attempt to fix this.
```
[ 80%] Building CXX object CMakeFiles/swigfaiss.dir/CMakeFiles/swigfaiss.dir/swigfaissPYTHON_wrap.cxx.o
/home/circleci/miniconda/conda-bld/faiss-pkg_1705881814815/work/_build_python_3.9/CMakeFiles/swigfaiss.dir/swigfaissPYTHON_wrap.cxx: In function 'PyObject* swig_ptr(PyObject*)':
/home/circleci/miniconda/conda-bld/faiss-pkg_1705881814815/work/_build_python_3.9/CMakeFiles/swigfaiss.dir/swigfaissPYTHON_wrap.cxx:6573:41: error: 'SWIGTYPE_p_unsigned_long_long' was not declared in this scope; did you mean 'SWIGTYPE_p_unsigned_long'?
6573 | return SWIG_NewPointerObj(data, SWIGTYPE_p_unsigned_long_long, 0);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/circleci/miniconda/conda-bld/faiss-pkg_1705881814815/work/_build_python_3.9/CMakeFiles/swigfaiss.dir/swigfaissPYTHON_wrap.cxx:1136:94: note: in definition of macro 'SWIG_NewPointerObj'
1136 | #define SWIG_NewPointerObj(ptr, type, flags) SWIG_Python_NewPointerObj(NULL, ptr, type, flags)
| ^~~~
/home/circleci/miniconda/conda-bld/faiss-pkg_1705881814815/work/_build_python_3.9/CMakeFiles/swigfaiss.dir/swigfaissPYTHON_wrap.cxx:6580:41: error: 'SWIGTYPE_p_long_long' was not declared in this scope; did you mean 'SWIGTYPE_p_MapLong2Long'?
6580 | return SWIG_NewPointerObj(data, SWIGTYPE_p_long_long, 0);
| ^~~~~~~~~~~~~~~~~~~~
/home/circleci/miniconda/conda-bld/faiss-pkg_1705881814815/work/_build_python_3.9/CMakeFiles/swigfaiss.dir/swigfaissPYTHON_wrap.cxx:1136:94: note: in definition of macro 'SWIG_NewPointerObj'
1136 | #define SWIG_NewPointerObj(ptr, type, flags) SWIG_Python_NewPointerObj(NULL, ptr, type, flags)
| ^~~~
make[3]: *** [CMakeFiles/swigfaiss.dir/build.make:76: CMakeFiles/swigfaiss.dir/CMakeFiles/swigfaiss.dir/swigfaissPYTHON_wrap.cxx.o] Error 1
```
Reviewed By: mlomeli1
Differential Revision: D52962236
fbshipit-source-id: e1a6b4c6ba03c9a0a320b5dbb54a8d618b8f8e74
Summary:
Diff summary courtesy of Metamate:
This diff bumps the version of libraft to fix nightly issues in the faiss-gpu-raft package. The code changes include updating the libraft version in the meta.yaml file to 24.02. This change is necessary to ensure that the package is compatible with the latest version of libraft.
Reviewed By: mlomeli1
Differential Revision: D52736355
fbshipit-source-id: 7ac2427210c06eb2d1c3ab4a3d37d48494fb039e
Summary:
This fixes the build, except for MacOS, where there's a problem with cmake + OpenMP. We can fix it separately.
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2856
Reviewed By: mlomeli1
Differential Revision: D45704458
Pulled By: algoriddle
fbshipit-source-id: 0c09036ae5fa34ab114b857f407a35603986613a
Summary:
RAFT requires cmake 3.23.1, pulling it from conda-forge. We continue to keep the dependency on conda-forge minimal, hence the ordering of the channels and the pinning of sysroot to a specific version.
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2808
Test Plan:
contbuild
Imported from GitHub, without a `Test Plan:` line.
Reviewed By: mlomeli1
Differential Revision: D44746827
Pulled By: algoriddle
fbshipit-source-id: ad576b11b257203bd0cafd57c2c2e7fd8d10ca98
Summary:
1. GPU builds use CircleCI base image, no docker
2. Switched to CUDA 11.4 (used to be 11.3)
3. Merged all build jobs into two parameterized targets: `build_cmake` and `build_conda`.
4. Cleaned up test execution, fixed bug of Python GPU tests not running on PRs
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2803
Reviewed By: mlomeli1
Differential Revision: D44541714
Pulled By: algoriddle
fbshipit-source-id: aa09ae638ecb6ef556d42f27a4bfaddad7355e50
Summary:
# Refactor
- Extract executors and merge all conda builds (that run on diffs) and conda package builds (that run nightly and on release) into two parameterized targets (`build_conda`, `deploy_conda`), except for GPU builds (at least for now)
- Similarly, introduce a `build_cmake` target that can be parameterized with executor, although run it for Linux x86_64 only (for now)
- Keep GPU targets separate (both conda package build and cmake) for now. Introduce "v2" targets that will eventually replace the current GPU build targets (we need to resolve GPU test failures).
- Removed `beauby/faiss-circleci:cpu` docker container, use the miniconda docker for Linux and machine images everywhere else. v2 GPU targets use the latest circleci images (see https://discuss.circleci.com/t/cuda-11-8-gpu-cuda-image-any-plans/47240/3)
# New/changed functionality
- Dropped CUDA 10
- Support for Linux arm64 conda packages
- Workflows have a consistent naming scheme, `OSX arm64 (conda)`, `Linux x86_64 (cmake)` etc.
- No cmake build for Linux or OSX arm64, replaced both with a conda build target only. We can reintroduce arm64 cmake workflows for both if needed (via additional parameterized build_cmake workflows), but it seemed unnecessary to me.
# Next steps
- Make v2 GPU builds work, deprecate v1, get rid of all docker stuff
- Merge GPU builds into cmake/conda build targets
- Possibly further unify package build and conda build targets
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2798
Test Plan: contbuild
Reviewed By: mlomeli1
Differential Revision: D44469783
Pulled By: algoriddle
fbshipit-source-id: 8489942fb7a4e4de1dd2d4466790e550191d15a1
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2791
Removed building for 3.7 (D44373285 - not supported for M1), adding 3.10 with this diff.
The only change that matters is in `conda_build_config.yaml`, the others are about making the configs consistent between CPU and GPU.
Reviewed By: mlomeli1
Differential Revision: D44405573
fbshipit-source-id: ad933e08834593e55a35075c602e5f509a813e73
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2594
Bumping `mkl` to 2021 and installing `mkl-devel` in the build environment to fix the Windows nightly build.
Reviewed By: mlomeli1
Differential Revision: D41534391
fbshipit-source-id: 7c681f530a1efe649cd176135a23ebb0fb44d70f
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2552
The `conda inspect` commands in the `test` section fail without `conda-build` in the `test` environment.
Reviewed By: mlomeli1
Differential Revision: D40793051
fbshipit-source-id: 184418cfa8d0efd6af6b0c806f7bddbeba176732
Summary:
Fixes OSX CI by pinning pytorch version for interop tests. The "real" fix is already landed in pytorch but has not been released yet.
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2482
Reviewed By: alexanderguzhva
Differential Revision: D39891113
Pulled By: beauby
fbshipit-source-id: fa79bf9de1c93e056260ea64613e37625edfecc3
Summary:
CentOS 8 being EOL, some modifications to the OS packages repositories
are needed in order to keep the build working.
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2495
Reviewed By: alexanderguzhva
Differential Revision: D39857306
Pulled By: beauby
fbshipit-source-id: 175f436132c589a9a22d93727bb58e556a43a8f6
Summary:
- Disable problematic tests on OSX.
- Ensure compiler compatibility with CUDA builds.
- Fix path for Python extension libraries.
- Use CentOS for CUDA packaging.
- Update CUDA versions in CI (10.2 and 11.3).
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2121
Reviewed By: mdouze
Differential Revision: D32921117
Pulled By: beauby
fbshipit-source-id: 588c18add8084b8228ff5abc651eaa4567919cc6
Summary:
This should fix the GPU nighties.
The rationale for the cp is that there is a shared file between the CPU and GPU tests.
Ideally, this file should probably moved to contrib at some point.
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/1901
Reviewed By: beauby
Differential Revision: D28680898
Pulled By: mdouze
fbshipit-source-id: b9d0e1969103764ecb6f1e047c9ed4bd4a76aaba
