88 lines
2.4 KiB
Markdown
88 lines
2.4 KiB
Markdown
|
# Torchserve Deployment
|
||
|
|
|
||
|
|
In order to serve an `MMClassification` model with [`TorchServe`](https://pytorch.org/serve/), you can follow the steps:
|
||
|
|
|
||
|
|
## 1. Convert model from MMClassification to TorchServe
|
||
|
|
|
||
|
|
```shell
|
||
|
|
python tools/torchserve/mmcls2torchserve.py ${CONFIG_FILE} ${CHECKPOINT_FILE} \
|
||
|
|
--output-folder ${MODEL_STORE} \
|
||
|
|
--model-name ${MODEL_NAME}
|
||
|
|
```
|
||
|
|
|
||
|
|
```{note}
|
||
|
|
${MODEL_STORE} needs to be an absolute path to a folder.
|
||
|
|
```
|
||
|
|
|
||
|
|
Example:
|
||
|
|
|
||
|
|
```shell
|
||
|
|
python tools/torchserve/mmcls2torchserve.py \
|
||
|
|
configs/resnet/resnet18_8xb32_in1k.py \
|
||
|
|
checkpoints/resnet18_8xb32_in1k_20210831-fbbb1da6.pth \
|
||
|
|
--output-folder ./checkpoints \
|
||
|
|
--model-name resnet18_in1k
|
||
|
|
```
|
||
|
|
|
||
|
|
## 2. Build `mmcls-serve` docker image
|
||
|
|
|
||
|
|
```shell
|
||
|
|
docker build -t mmcls-serve:latest docker/serve/
|
||
|
|
```
|
||
|
|
|
||
|
|
## 3. Run `mmcls-serve`
|
||
|
|
|
||
|
|
Check the official docs for [running TorchServe with docker](https://github.com/pytorch/serve/blob/master/docker/README.md#running-torchserve-in-a-production-docker-environment).
|
||
|
|
|
||
|
|
In order to run in GPU, you need to install [nvidia-docker](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html). You can omit the `--gpus` argument in order to run in GPU.
|
||
|
|
|
||
|
|
Example:
|
||
|
|
|
||
|
|
```shell
|
||
|
|
docker run --rm \
|
||
|
|
--cpus 8 \
|
||
|
|
--gpus device=0 \
|
||
|
|
-p8080:8080 -p8081:8081 -p8082:8082 \
|
||
|
|
--mount type=bind,source=`realpath ./checkpoints`,target=/home/model-server/model-store \
|
||
|
|
mmcls-serve:latest
|
||
|
|
```
|
||
|
|
|
||
|
|
```{note}
|
||
|
|
`realpath ./checkpoints` points to the absolute path of "./checkpoints", and you can replace it with the absolute path where you store torchserve models.
|
||
|
|
```
|
||
|
|
|
||
|
|
[Read the docs](https://github.com/pytorch/serve/blob/master/docs/rest_api.md) about the Inference (8080), Management (8081) and Metrics (8082) APis
|
||
|
|
|
||
|
|
## 4. Test deployment
|
||
|
|
|
||
|
|
```shell
|
||
|
|
curl http://127.0.0.1:8080/predictions/${MODEL_NAME} -T demo/demo.JPEG
|
||
|
|
```
|
||
|
|
|
||
|
|
You should obtain a response similar to:
|
||
|
|
|
||
|
|
```json
|
||
|
|
{
|
||
|
|
"pred_label": 58,
|
||
|
|
"pred_score": 0.38102269172668457,
|
||
|
|
"pred_class": "water snake"
|
||
|
|
}
|
||
|
|
```
|
||
|
|
|
||
|
|
And you can use `test_torchserver.py` to compare result of TorchServe and PyTorch, and visualize them.
|
||
|
|
|
||
|
|
```shell
|
||
|
|
python tools/torchserve/test_torchserver.py ${IMAGE_FILE} ${CONFIG_FILE} ${CHECKPOINT_FILE} ${MODEL_NAME}
|
||
|
|
[--inference-addr ${INFERENCE_ADDR}] [--device ${DEVICE}]
|
||
|
|
```
|
||
|
|
|
||
|
|
Example:
|
||
|
|
|
||
|
|
```shell
|
||
|
|
python tools/torchserve/test_torchserver.py \
|
||
|
|
demo/demo.JPEG \
|
||
|
|
configs/resnet/resnet18_8xb32_in1k.py \
|
||
|
|
checkpoints/resnet18_8xb32_in1k_20210831-fbbb1da6.pth \
|
||
|
|
resnet18_in1k
|
||
|
|
```
|