YOLOv5 Study: Speed vs Batch-Size

[glenn-jocher]

Batch Size study here. Larger batches contribute to improved per-image inference speeds. Study run on Colab Pro+ with A100 40GB GPU. I used val.py --data coco.yaml --task study, and just updated the code a bit to run the study over batch size instead of image size.

YOLOv5 🚀 v6.0-255-gca0a007 torch 1.10.0+cu111 CUDA:0 (Tesla A100-SXM-40GB)

Google sheet with complete results.
https://docs.google.com/spreadsheets/d/1Nm3jofjdgKja0AZHV8Jk_m8TgcF7jenCSA06DuEG2C0

Inference Time vs Batch Size

FPS vs Batch Size

Speed gains vs batch-size 1

Interesting takeaway is that smaller models benefit disproportionately from large batch sizes. These are the FPS gains vs batch-size 1 from the study:

Batch Size	YOLOv5n	YOLOv5s	YOLOv5m	YOLOv5l	YOLOv5x
1	1.0	1.0	1.0	1.0	1.0
2	1.8	1.8	1.9	1.9	1.9
4	3.4	3.4	3.5	3.5	3.8
8	7.0	7.0	7.0	6.4	5.6
16	13.1	10.5	8.5	7.6	6.0
32	17.1	11.7	9.2	7.8	6.4
64	20.0	14.1	10.5	8.4	6.8
128	23.1	15.2	10.7	8.4	6.7

[MrRace]

The results above is for Pytorch model format?

[glenn-jocher]

Yes

[MrRace]

@glenn-jocher I try the yolov5s with different batch size in T4 GPU, however it seems no gains in inference time on average of each image . I use cmd like python3 val.py --weights models/yolov5s.pt --device 0 --batch-size 4 --half

model	batch size	cost time(ms)/image
pytorch-fp32	1	7.1
pytorch-fp32	2	7.0
pytorch-fp32	4	6.9
pytorch-fp32	8	6.0
pytorch-fp32	16	5.5
pytorch-fp16	1	6.3
pytorch-fp16	2	5.1
pytorch-fp16	4	4.9
pytorch-fp16	8	3.6
pytorch-fp16	16	4.2
onnx-fp32	1	15.2
onnx-fp32	2	14.7
onnx-fp32	4	18.4
onnx-fp32	8	20.6
onnx-fp32	16	20.9
onnx-fp16	1	11.9
onnx-fp16	2	8.3
onnx-fp16	4	7.9
onnx-fp16	8	10.1
onnx-fp16	16	11.5
tensorrt-fp16	1	4.4
tensorrt-fp16	2	3.0
tensorrt-fp16	4	2.8
tensorrt-fp16	8	2.9
tensorrt-fp16	16	3.1

Have you try the batch size inference in T4 GPU? @glenn-jocher

[glenn-jocher]

@MrRace commands to reproduce are indicated above. Tests run on A100. Results will vary by hardware, software, drivers etc.

[glenn-jocher]

@MrRace also for best statistics you need to run on coco.yaml, not coco128.yaml.