faiss/.circleci/config.yml

87 lines
2.6 KiB
YAML
Raw Normal View History

version: 2.1
refactor contbuild, add new GPU targets, support Linux arm64 conda packages (#2798) Summary: # Refactor - Extract executors and merge all conda builds (that run on diffs) and conda package builds (that run nightly and on release) into two parameterized targets (`build_conda`, `deploy_conda`), except for GPU builds (at least for now) - Similarly, introduce a `build_cmake` target that can be parameterized with executor, although run it for Linux x86_64 only (for now) - Keep GPU targets separate (both conda package build and cmake) for now. Introduce "v2" targets that will eventually replace the current GPU build targets (we need to resolve GPU test failures). - Removed `beauby/faiss-circleci:cpu` docker container, use the miniconda docker for Linux and machine images everywhere else. v2 GPU targets use the latest circleci images (see https://discuss.circleci.com/t/cuda-11-8-gpu-cuda-image-any-plans/47240/3) # New/changed functionality - Dropped CUDA 10 - Support for Linux arm64 conda packages - Workflows have a consistent naming scheme, `OSX arm64 (conda)`, `Linux x86_64 (cmake)` etc. - No cmake build for Linux or OSX arm64, replaced both with a conda build target only. We can reintroduce arm64 cmake workflows for both if needed (via additional parameterized build_cmake workflows), but it seemed unnecessary to me. # Next steps - Make v2 GPU builds work, deprecate v1, get rid of all docker stuff - Merge GPU builds into cmake/conda build targets - Possibly further unify package build and conda build targets Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2798 Test Plan: contbuild Reviewed By: mlomeli1 Differential Revision: D44469783 Pulled By: algoriddle fbshipit-source-id: 8489942fb7a4e4de1dd2d4466790e550191d15a1
2023-03-29 06:56:27 -07:00
executors:
linux-x86_64-cpu:
docker:
- image: continuumio/miniconda3
resource_class: large
jobs:
refactor contbuild, add new GPU targets, support Linux arm64 conda packages (#2798) Summary: # Refactor - Extract executors and merge all conda builds (that run on diffs) and conda package builds (that run nightly and on release) into two parameterized targets (`build_conda`, `deploy_conda`), except for GPU builds (at least for now) - Similarly, introduce a `build_cmake` target that can be parameterized with executor, although run it for Linux x86_64 only (for now) - Keep GPU targets separate (both conda package build and cmake) for now. Introduce "v2" targets that will eventually replace the current GPU build targets (we need to resolve GPU test failures). - Removed `beauby/faiss-circleci:cpu` docker container, use the miniconda docker for Linux and machine images everywhere else. v2 GPU targets use the latest circleci images (see https://discuss.circleci.com/t/cuda-11-8-gpu-cuda-image-any-plans/47240/3) # New/changed functionality - Dropped CUDA 10 - Support for Linux arm64 conda packages - Workflows have a consistent naming scheme, `OSX arm64 (conda)`, `Linux x86_64 (cmake)` etc. - No cmake build for Linux or OSX arm64, replaced both with a conda build target only. We can reintroduce arm64 cmake workflows for both if needed (via additional parameterized build_cmake workflows), but it seemed unnecessary to me. # Next steps - Make v2 GPU builds work, deprecate v1, get rid of all docker stuff - Merge GPU builds into cmake/conda build targets - Possibly further unify package build and conda build targets Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2798 Test Plan: contbuild Reviewed By: mlomeli1 Differential Revision: D44469783 Pulled By: algoriddle fbshipit-source-id: 8489942fb7a4e4de1dd2d4466790e550191d15a1
2023-03-29 06:56:27 -07:00
build_cmake:
parameters:
refactor contbuild, add new GPU targets, support Linux arm64 conda packages (#2798) Summary: # Refactor - Extract executors and merge all conda builds (that run on diffs) and conda package builds (that run nightly and on release) into two parameterized targets (`build_conda`, `deploy_conda`), except for GPU builds (at least for now) - Similarly, introduce a `build_cmake` target that can be parameterized with executor, although run it for Linux x86_64 only (for now) - Keep GPU targets separate (both conda package build and cmake) for now. Introduce "v2" targets that will eventually replace the current GPU build targets (we need to resolve GPU test failures). - Removed `beauby/faiss-circleci:cpu` docker container, use the miniconda docker for Linux and machine images everywhere else. v2 GPU targets use the latest circleci images (see https://discuss.circleci.com/t/cuda-11-8-gpu-cuda-image-any-plans/47240/3) # New/changed functionality - Dropped CUDA 10 - Support for Linux arm64 conda packages - Workflows have a consistent naming scheme, `OSX arm64 (conda)`, `Linux x86_64 (cmake)` etc. - No cmake build for Linux or OSX arm64, replaced both with a conda build target only. We can reintroduce arm64 cmake workflows for both if needed (via additional parameterized build_cmake workflows), but it seemed unnecessary to me. # Next steps - Make v2 GPU builds work, deprecate v1, get rid of all docker stuff - Merge GPU builds into cmake/conda build targets - Possibly further unify package build and conda build targets Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2798 Test Plan: contbuild Reviewed By: mlomeli1 Differential Revision: D44469783 Pulled By: algoriddle fbshipit-source-id: 8489942fb7a4e4de1dd2d4466790e550191d15a1
2023-03-29 06:56:27 -07:00
exec:
type: executor
opt_level:
type: string
default: generic
refactor contbuild, add new GPU targets, support Linux arm64 conda packages (#2798) Summary: # Refactor - Extract executors and merge all conda builds (that run on diffs) and conda package builds (that run nightly and on release) into two parameterized targets (`build_conda`, `deploy_conda`), except for GPU builds (at least for now) - Similarly, introduce a `build_cmake` target that can be parameterized with executor, although run it for Linux x86_64 only (for now) - Keep GPU targets separate (both conda package build and cmake) for now. Introduce "v2" targets that will eventually replace the current GPU build targets (we need to resolve GPU test failures). - Removed `beauby/faiss-circleci:cpu` docker container, use the miniconda docker for Linux and machine images everywhere else. v2 GPU targets use the latest circleci images (see https://discuss.circleci.com/t/cuda-11-8-gpu-cuda-image-any-plans/47240/3) # New/changed functionality - Dropped CUDA 10 - Support for Linux arm64 conda packages - Workflows have a consistent naming scheme, `OSX arm64 (conda)`, `Linux x86_64 (cmake)` etc. - No cmake build for Linux or OSX arm64, replaced both with a conda build target only. We can reintroduce arm64 cmake workflows for both if needed (via additional parameterized build_cmake workflows), but it seemed unnecessary to me. # Next steps - Make v2 GPU builds work, deprecate v1, get rid of all docker stuff - Merge GPU builds into cmake/conda build targets - Possibly further unify package build and conda build targets Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2798 Test Plan: contbuild Reviewed By: mlomeli1 Differential Revision: D44469783 Pulled By: algoriddle fbshipit-source-id: 8489942fb7a4e4de1dd2d4466790e550191d15a1
2023-03-29 06:56:27 -07:00
executor: << parameters.exec >>
environment:
OMP_NUM_THREADS: 10
refactor contbuild, add new GPU targets, support Linux arm64 conda packages (#2798) Summary: # Refactor - Extract executors and merge all conda builds (that run on diffs) and conda package builds (that run nightly and on release) into two parameterized targets (`build_conda`, `deploy_conda`), except for GPU builds (at least for now) - Similarly, introduce a `build_cmake` target that can be parameterized with executor, although run it for Linux x86_64 only (for now) - Keep GPU targets separate (both conda package build and cmake) for now. Introduce "v2" targets that will eventually replace the current GPU build targets (we need to resolve GPU test failures). - Removed `beauby/faiss-circleci:cpu` docker container, use the miniconda docker for Linux and machine images everywhere else. v2 GPU targets use the latest circleci images (see https://discuss.circleci.com/t/cuda-11-8-gpu-cuda-image-any-plans/47240/3) # New/changed functionality - Dropped CUDA 10 - Support for Linux arm64 conda packages - Workflows have a consistent naming scheme, `OSX arm64 (conda)`, `Linux x86_64 (cmake)` etc. - No cmake build for Linux or OSX arm64, replaced both with a conda build target only. We can reintroduce arm64 cmake workflows for both if needed (via additional parameterized build_cmake workflows), but it seemed unnecessary to me. # Next steps - Make v2 GPU builds work, deprecate v1, get rid of all docker stuff - Merge GPU builds into cmake/conda build targets - Possibly further unify package build and conda build targets Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2798 Test Plan: contbuild Reviewed By: mlomeli1 Differential Revision: D44469783 Pulled By: algoriddle fbshipit-source-id: 8489942fb7a4e4de1dd2d4466790e550191d15a1
2023-03-29 06:56:27 -07:00
MKL_THREADING_LAYER: GNU
steps:
- checkout
- run:
refactor contbuild, add new GPU targets, support Linux arm64 conda packages (#2798) Summary: # Refactor - Extract executors and merge all conda builds (that run on diffs) and conda package builds (that run nightly and on release) into two parameterized targets (`build_conda`, `deploy_conda`), except for GPU builds (at least for now) - Similarly, introduce a `build_cmake` target that can be parameterized with executor, although run it for Linux x86_64 only (for now) - Keep GPU targets separate (both conda package build and cmake) for now. Introduce "v2" targets that will eventually replace the current GPU build targets (we need to resolve GPU test failures). - Removed `beauby/faiss-circleci:cpu` docker container, use the miniconda docker for Linux and machine images everywhere else. v2 GPU targets use the latest circleci images (see https://discuss.circleci.com/t/cuda-11-8-gpu-cuda-image-any-plans/47240/3) # New/changed functionality - Dropped CUDA 10 - Support for Linux arm64 conda packages - Workflows have a consistent naming scheme, `OSX arm64 (conda)`, `Linux x86_64 (cmake)` etc. - No cmake build for Linux or OSX arm64, replaced both with a conda build target only. We can reintroduce arm64 cmake workflows for both if needed (via additional parameterized build_cmake workflows), but it seemed unnecessary to me. # Next steps - Make v2 GPU builds work, deprecate v1, get rid of all docker stuff - Merge GPU builds into cmake/conda build targets - Possibly further unify package build and conda build targets Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2798 Test Plan: contbuild Reviewed By: mlomeli1 Differential Revision: D44469783 Pulled By: algoriddle fbshipit-source-id: 8489942fb7a4e4de1dd2d4466790e550191d15a1
2023-03-29 06:56:27 -07:00
name: Install conda
command: |
refactor contbuild, add new GPU targets, support Linux arm64 conda packages (#2798) Summary: # Refactor - Extract executors and merge all conda builds (that run on diffs) and conda package builds (that run nightly and on release) into two parameterized targets (`build_conda`, `deploy_conda`), except for GPU builds (at least for now) - Similarly, introduce a `build_cmake` target that can be parameterized with executor, although run it for Linux x86_64 only (for now) - Keep GPU targets separate (both conda package build and cmake) for now. Introduce "v2" targets that will eventually replace the current GPU build targets (we need to resolve GPU test failures). - Removed `beauby/faiss-circleci:cpu` docker container, use the miniconda docker for Linux and machine images everywhere else. v2 GPU targets use the latest circleci images (see https://discuss.circleci.com/t/cuda-11-8-gpu-cuda-image-any-plans/47240/3) # New/changed functionality - Dropped CUDA 10 - Support for Linux arm64 conda packages - Workflows have a consistent naming scheme, `OSX arm64 (conda)`, `Linux x86_64 (cmake)` etc. - No cmake build for Linux or OSX arm64, replaced both with a conda build target only. We can reintroduce arm64 cmake workflows for both if needed (via additional parameterized build_cmake workflows), but it seemed unnecessary to me. # Next steps - Make v2 GPU builds work, deprecate v1, get rid of all docker stuff - Merge GPU builds into cmake/conda build targets - Possibly further unify package build and conda build targets Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2798 Test Plan: contbuild Reviewed By: mlomeli1 Differential Revision: D44469783 Pulled By: algoriddle fbshipit-source-id: 8489942fb7a4e4de1dd2d4466790e550191d15a1
2023-03-29 06:56:27 -07:00
if [ -n "${CONDA_ARCH}" ]
then
curl https://repo.anaconda.com/miniconda/Miniconda3-latest-${CONDA_ARCH}.sh --output miniconda.sh
bash miniconda.sh -b -p $HOME/miniconda
~/miniconda/bin/conda init
fi
- run:
refactor contbuild, add new GPU targets, support Linux arm64 conda packages (#2798) Summary: # Refactor - Extract executors and merge all conda builds (that run on diffs) and conda package builds (that run nightly and on release) into two parameterized targets (`build_conda`, `deploy_conda`), except for GPU builds (at least for now) - Similarly, introduce a `build_cmake` target that can be parameterized with executor, although run it for Linux x86_64 only (for now) - Keep GPU targets separate (both conda package build and cmake) for now. Introduce "v2" targets that will eventually replace the current GPU build targets (we need to resolve GPU test failures). - Removed `beauby/faiss-circleci:cpu` docker container, use the miniconda docker for Linux and machine images everywhere else. v2 GPU targets use the latest circleci images (see https://discuss.circleci.com/t/cuda-11-8-gpu-cuda-image-any-plans/47240/3) # New/changed functionality - Dropped CUDA 10 - Support for Linux arm64 conda packages - Workflows have a consistent naming scheme, `OSX arm64 (conda)`, `Linux x86_64 (cmake)` etc. - No cmake build for Linux or OSX arm64, replaced both with a conda build target only. We can reintroduce arm64 cmake workflows for both if needed (via additional parameterized build_cmake workflows), but it seemed unnecessary to me. # Next steps - Make v2 GPU builds work, deprecate v1, get rid of all docker stuff - Merge GPU builds into cmake/conda build targets - Possibly further unify package build and conda build targets Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2798 Test Plan: contbuild Reviewed By: mlomeli1 Differential Revision: D44469783 Pulled By: algoriddle fbshipit-source-id: 8489942fb7a4e4de1dd2d4466790e550191d15a1
2023-03-29 06:56:27 -07:00
name: Set up environment
command: |
conda config --set solver libmamba
refactor contbuild, add new GPU targets, support Linux arm64 conda packages (#2798) Summary: # Refactor - Extract executors and merge all conda builds (that run on diffs) and conda package builds (that run nightly and on release) into two parameterized targets (`build_conda`, `deploy_conda`), except for GPU builds (at least for now) - Similarly, introduce a `build_cmake` target that can be parameterized with executor, although run it for Linux x86_64 only (for now) - Keep GPU targets separate (both conda package build and cmake) for now. Introduce "v2" targets that will eventually replace the current GPU build targets (we need to resolve GPU test failures). - Removed `beauby/faiss-circleci:cpu` docker container, use the miniconda docker for Linux and machine images everywhere else. v2 GPU targets use the latest circleci images (see https://discuss.circleci.com/t/cuda-11-8-gpu-cuda-image-any-plans/47240/3) # New/changed functionality - Dropped CUDA 10 - Support for Linux arm64 conda packages - Workflows have a consistent naming scheme, `OSX arm64 (conda)`, `Linux x86_64 (cmake)` etc. - No cmake build for Linux or OSX arm64, replaced both with a conda build target only. We can reintroduce arm64 cmake workflows for both if needed (via additional parameterized build_cmake workflows), but it seemed unnecessary to me. # Next steps - Make v2 GPU builds work, deprecate v1, get rid of all docker stuff - Merge GPU builds into cmake/conda build targets - Possibly further unify package build and conda build targets Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2798 Test Plan: contbuild Reviewed By: mlomeli1 Differential Revision: D44469783 Pulled By: algoriddle fbshipit-source-id: 8489942fb7a4e4de1dd2d4466790e550191d15a1
2023-03-29 06:56:27 -07:00
conda update -y -q conda
- run:
name: Install env using main channel
command: |
conda install -y -q python=3.11 cmake make swig mkl=2023 mkl-devel=2023 numpy scipy pytest gxx_linux-64=11.2 sysroot_linux-64
- run:
name: Build all targets
refactor contbuild, add new GPU targets, support Linux arm64 conda packages (#2798) Summary: # Refactor - Extract executors and merge all conda builds (that run on diffs) and conda package builds (that run nightly and on release) into two parameterized targets (`build_conda`, `deploy_conda`), except for GPU builds (at least for now) - Similarly, introduce a `build_cmake` target that can be parameterized with executor, although run it for Linux x86_64 only (for now) - Keep GPU targets separate (both conda package build and cmake) for now. Introduce "v2" targets that will eventually replace the current GPU build targets (we need to resolve GPU test failures). - Removed `beauby/faiss-circleci:cpu` docker container, use the miniconda docker for Linux and machine images everywhere else. v2 GPU targets use the latest circleci images (see https://discuss.circleci.com/t/cuda-11-8-gpu-cuda-image-any-plans/47240/3) # New/changed functionality - Dropped CUDA 10 - Support for Linux arm64 conda packages - Workflows have a consistent naming scheme, `OSX arm64 (conda)`, `Linux x86_64 (cmake)` etc. - No cmake build for Linux or OSX arm64, replaced both with a conda build target only. We can reintroduce arm64 cmake workflows for both if needed (via additional parameterized build_cmake workflows), but it seemed unnecessary to me. # Next steps - Make v2 GPU builds work, deprecate v1, get rid of all docker stuff - Merge GPU builds into cmake/conda build targets - Possibly further unify package build and conda build targets Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2798 Test Plan: contbuild Reviewed By: mlomeli1 Differential Revision: D44469783 Pulled By: algoriddle fbshipit-source-id: 8489942fb7a4e4de1dd2d4466790e550191d15a1
2023-03-29 06:56:27 -07:00
no_output_timeout: 30m
command: |
eval "$(conda shell.bash hook)"
conda activate
cmake -B build \
-DBUILD_TESTING=ON \
-DBUILD_SHARED_LIBS=ON \
-DFAISS_ENABLE_GPU=OFF \
-DFAISS_ENABLE_RAFT=OFF \
-DFAISS_OPT_LEVEL=<< parameters.opt_level >> \
-DFAISS_ENABLE_C_API=ON \
-DPYTHON_EXECUTABLE=$(which python) \
-DCMAKE_BUILD_TYPE=Release \
-DBLA_VENDOR=Intel10_64_dyn \
-DCMAKE_CUDA_FLAGS="-gencode arch=compute_75,code=sm_75" \
.
make -k -C build -j$(nproc)
- run:
name: C++ tests
command: |
export GTEST_OUTPUT="xml:$(realpath .)/test-results/googletest/"
make -C build test
- run:
name: Install Python extension
command: |
cd build/faiss/python
python setup.py install
- run:
name: Python tests (CPU only)
command: |
conda install -y -q pytorch -c pytorch
pytest --junitxml=test-results/pytest/results.xml tests/test_*.py
pytest --junitxml=test-results/pytest/results-torch.xml tests/torch_*.py
- store_test_results:
path: test-results
workflows:
version: 2
build:
jobs:
- build_cmake:
name: Linux x86_64 AVX512 (cmake)
exec: linux-x86_64-cpu
opt_level: "avx512"