Behavior of different types of English learners tends to follow different patterns and characteristics, and the analysis of behavioral data is one of the directions for improving English learning and teaching. This study designs a set of behavioral analysis methods based on machine learning for English teaching and learners. The learners’ behaviors are firstly operated with feature extraction and quantification, and the behavioral data are clustered by using the systematic clustering method (HCM) to improve the SOM model. 1DCNN is used to process the learning time-series data and enhance the data mining and performance prediction ability by BiLSTM and attention mechanism, respectively. This paper distinguishes five categories of English learners, such as excellent, diligent, average, procrastinating and negative, and filters out the factors that are highly correlated with English performance, such as the download of learning resources and the number of times of teaching viewing. Comparison experiments show that the ACC of this paper’s achievement prediction model = 0.53, which is better than other comparison methods. Therefore, the idea of this paper based on machine learning methods to analyze the behavior of English teaching and learners has feasibility.