【Purposes】Two-stream convolutional networks primarily achieve high recognition accuracy by fusing spatial and temporal features of videos. Traditional two-stream convolutional networks extract temporal features using optical flow or temporal shift modules. The disadvantage of using optical flow is the heavy workload required, while the disadvantage of using temporal shift modules is that they can damage the original video's spatial and channel features.【Methods】This paper proposes a two-stream network called the two-stream feature enhancement network (TFEN) to address these issues. To solve the problem of feature damage caused by temporal shift, this paper proposes a spatial enhancement-temporal shift module (SE-TSM) and a channel enhancement-temporal shift module (CE-TSM) to enhance features after each time shift, improving damaged features. For the issue with optical flow, this paper proposes a fusion-based frame difference network for extracting temporal features, which is less time-consuming and easier to operate than optical flow. To address the weak motion information in frame differences, this paper introduces a Sports Improvement Module (SIM) to enhance motion features and improve performance.【Findings】The network in this article achieved 96.1% and 75.7% accuracy on the public video datasets UCF101 and HMDB51, respectively, which is superior to current mainstream networks. At the same time, it has the advantage of low complexity and high performance compared to 3D networks and transformer networks.