The article takes the defect detection and recognition of railroad track as the main research point, and extracts, preprocesses and corrects the railroad track surface image by introducing image segmentation algorithm. Gabor function, K-means clustering method and conditional iterative pattern algorithm are embedded in the original Markov random field model to construct the improved two layer graph model for railroad track defect segmentation. The recall, precision, mean average precision, and loss function of the improved Markov defect segmentation model are significantly better than those of the original model, and the mean average precision of the defect segmentation model is increased to 95.7% after the Gabor function, K-means clustering method, and conditional iterative pattern algorithm are applied. The improved Markov defect segmentation model fused with clustering features in this paper can better meet the classification and identification of railroad track defects.