Aiming at the complex structure of deep network models, inspired by the fact that the construction of Gabor filters does not require any learning process and has nothing to do with training data, and that radial basis function (RBF) kernel pooling can extract nonlinear second-order features, a lightweight convolutional network approach combining Gabor filters and RBF kernel pooling was proposed. First, Gabor convolution is performed on the face image to obtain the feature map; then the hyperbolic tangent function tanh is used to activate the feature map with the hope of enhancing the discription ability of the feature; finally, the multi-scale pyramid strategy is applied to divide the feature map into multiple regions, RBF kernel pooling is conducted on each region, and the kernel pooling features of all regions are concatenated to obtain the face feature representation. The influence of multiple parameters on the recognition performance was discussed, and the difference and performance of covariance pooling and kernel pooling were compared. Extensive experiments were carried out on three single-sample face recognition datasets and one video face verification dataset. The results demonstrate that the face features learned by our method have excellent discriminative power and strong robustness to the factors such as illumination, occlusion, and age.