Online detection and analysis of polarizer quality can be completed by image processing technology. The existing methods based on deep learning can ensure the accuracy of classification, but the detection speed is slow and the model occupies a large amount of memory, thus difficult to meet the real-time requirements of online detection system. In order to solve these problems, a real-time polarizer defect detection algorithm based on deep learning was proposed. First, a new parallel module was designed to build the network. The module mixes different sizes of convolution kernels, and can extract more defect features compared with traditional convolution layers; The standard convolution in the module can be replaced by depthwise separable convolution, which can greatly reduce the number of parameters and the multiply-accumulate operations(MACCs). Second, asymmetric convolution instead of depthwise separable convolution in parallel module was used to obtain parallel asymmetric convolution module, which can further reduce the number of parameters of network. Finally, the global average pooling layer was used to replace the fully connected layer, which can significantly reduce the number of parameters of the network. The experimental results show that the test time per image is 108 ms, the accuracy of the model on the testing dataset is 99.4%, and the size of the model is 0.583 MB, which can meet the real-time requirements of industry.