Conveyor belt, as an important transportation equipment in coal mine, is often damaged, which affects the safety and efficiency. An improved YOLOv4 detection model is proposed to solve the problem of missing detection and false detection when the existing model detects objects with small size and low contrast with the background. In order to solve the problem of small size, firstly, the DDS unit is designed to replace the Res unit in the backbone network. By connecting features of different levels across layers, complete and rich multi-scale features can be obtained, and small-size damage detection can be completed. Secondly, the gradient harmonized mechanism is introduced into the classification loss function, and the weight of small-size damage is dynamically adjusted to make it fully trained. Aiming at the low contrast between damage and background, firstly, the coordinate attention mechanism is embedded in the deep network layer of the backbone network to enhance the model's attention to damage characteristics and reduce the interference of background noise. Secondly, the accurate decoupled head is designed to improve detection accuracy by solving the contradiction between classification and location requirements for features. Experimental results demonstrate that the mean average precision of this model is increased by 3.92% compared with YOLOv4, and the detection accuracy of small-size crack damage and low-contrast wear damage is improved by 4.32% and 4.24%, respectively, which effectively solved the problems of missed detection and false detection.