In today’s digital era, user interface (UI) design is crucial for enhancing user experience and strengthening user engagement. The study uses heatmap analysis, K-means clustering algorithm and random forest regression algorithm to comprehensively analyze the characteristics of user behavior in UI pages. The predicted results of user behavior in UI pages are visualized and analyzed through heatmaps. Cluster classes are divided according to user behavioral characteristics to generate user profiles with the same behavior. Combine Random Forest and Logistic regression algorithm to get the key indexes of UI optimization design and predict their impact on user behavior experience. The research results show that the MAE and SMAPE values of Random Forest regression algorithm on user behavior prediction are 133.55 and 8.18%, respectively, with an R² of 0.96, and the accuracy rate of behavior prediction is more than 80%, which shows a good performance of user behavior prediction. The clustering algorithm divides the user behavioral characteristics into 6 clusters based on their behavioral characteristics, including cluster class 1 (browsing and exploring class), which accounts for 11.5% of the number of investigators. The weight of the top 8 of the importance of UI optimization design obtained by the random forest regression analysis algorithm is 70.26%. And the user behavior experience can be improved by 5.377~9.925 times when each element is improved by one unit.