In this paper, some important algorithms in the field of target detection and tracking are optimized. Firstly, Gaussian modeling is performed in the color space for the dynamic background, and the priority is set for ranking. Then introduce adaptive Gaussian component number mixing, adaptively change the weights, and adaptively change the number of mixed Gaussian components according to the pixel color change in the scene to improve the convergence speed of the complex scene. Finally, Kalman filtering and mean drift algorithm are combined to ensure the robustness of target tracking in complex scenes. The single-frame detection time, accuracy, and average tracking error of the algorithm designed in this paper are examined on the dataset, and it is found that the time consumed by the algorithm in this paper in the three scenarios is 242ms, 323ms, and 274ms, respectively, with the highest accuracy of 96.9% and the average tracking error of only 1.5 pixels. The optimization algorithm designed in this paper is able to adapt to the slight disturbance of the background scene and overcome the influence of noise and ambient lighting, which is a target detection and tracking algorithm with good robustness.