Aiming at the problem of low detection accuracy caused by irregular crack shape and complex background in pavement crack detection scene, an end-to-end pavement crack detection method based on transformer, CrackFormerNet, was proposed. First, in the feature extraction stage, a multi-scale feature fusion mechanism is introduced, and a Multi-Scale Transformer feature extraction network is designed to fuse the feature maps of different downsampling magnifications in the Swin Transformer process to extract crack texture features with rich details. Second, a joint regression loss function based on CIoU Loss and L1 Loss is proposed to measure the distance between the predicted box and the label, and more accurately evaluate the detection effect of the predicted box. At the same time, in order to deal with the problem of slow convergence of the transformer model, the model convergence is accelerated by using the Pre-LN Transformer structure in the encoder-decoder stage and using layer normalization inside the residual block. The experimental results show that the MAP of the method in this paper reaches 84.2%. Compared with the mainstream benchmark methods, the proposed method gives significantly improved detection accuracy in the pavement crack detection task. Compared with DETR detection method, the is compressed the proposed method in model convergence round by 18.4%, while is improved in detection accuracy by 3.6%, which proves the effectiveness of this method.