Deep convolutional neural networks are powerful models that have been widely used in various medical image processing tasks. However, when processing time series data with a growth relationship, CNN may not be effective due to its inability to extract the complete spatial growth relationship between sequence images. The Transformer-based model overcomes this problem through its self-attention mechanism. Therefore, this paper proposes a Dual stream enhanced lung cancer growth evolution predictive network under time series (DSGNet). DSGNet makes full use of the advantages of CNN and Transformer, extracting the static features of tumors through CNN-based branches, and strengthening the extracted feature representations in a multi-scale manner. The Transformer-based branch obtains sequential dependencies between tumor sequence images, with its core component being the multi-head self-attention layer proposed in this paper. This branch maps lesion sequence images into a feature map sequence and then inputs the sequence into a deep network with multi-head self-attention, from which the complete inter-tumor growth relationship is extracted. This paper evaluates the proposed algorithm on the lung cancer NLST dataset and a dataset from cooperative hospitals. The experimental results show that DSGNet achieves a Precision of 92.45% and a Dice coefficient of 82.78% in predicting tumor growth. Compared with other tumor prediction algorithms, DSGNet proposed in this paper has been improved to a certain extent in all aspects and has been proven to be applicable to clinical research in many ways.