For traditional threshold denoising method, the effect of the fault signal of gearbox extracted from the background noise under soft background and soft threshold is not ideal, and the threshold processing method of the real and imaginary parts causes local phase distortion.On the basis of the translation invariance of double-tree complex wavelet transform, a method of noise reduction of double-tree complex wavelet transform and high-order cumulant is proposed.The algorithm uses the fourth-order cumulant processing method for each dual-tree complex wavelet coefficient after decomposition and single reconstruction, and carries out signal-noise separation according to the statistical characteristic of signal and noise.Because the number of wavelet decomposition layers directly affects the denoising effect of signal, the particle swarm optimization algorithm is used to select the number of decomposed layers of wavelet.The simulation and experimental signal processing show that this method can effectively suppress the noise interference and improve the signal-to-noise ratio at different signal and noise levels compared with the soft and hard threshold processing methods of double-tree complex wavelet transform, and satisfy the feature extraction need for subsequent processing of collected experiment vibration signal.