With the development of transportation systems, there is an increasing demand for real-time understanding of traffic scenes using image segmentation algorithms. Therefore, this paper carries out an in-depth study on how image segmentation algorithms for complex traffic scenes can meet the detection requirements of real-time while maintaining accuracy. The article first proposes a lightweight semantic segmentation method based on IDEL_DeepLabV3+, which lightens the IDE_DeepLabV3+ network and optimizes the loss function to improve the positive and negative sample imbalance problem. Then an improved image multi-texture detection method based on Faster RCNN is proposed to improve the detection performance of complex traffic scenes. Finally, the performance of the algorithm designed in this paper is tested through experiments. The performance of the deformable convolution, attention mechanism and feature pyramid improved model is tested and verified, the AP value of the deformable convolution is increased from 41.36 to 47.26, the mAP value of the overall model of the scSE attention mechanism is increased by 0.84%, and the final AP value of the weighted bi-directional feature pyramid network reaches 45.4. The improved DeepLabv3+ network achieves a high AP value of 75.03% in terms of the evaluation index mIOU by 75.03% is better than the original network’s 72.26%, so it can be said that we experimentally verified that our improved method enhances the segmentation accuracy of DeepLabv3+ network. The experimental results show that the proposed method in this paper improves the image segmentation accuracy while guaranteeing the segmentation speed, which effectively improves the segmentation effect.