Objective and comprehensive teaching effectiveness assessment is a strong guarantee for the quality of English teaching in colleges and universities. This paper establishes an index system for evaluating the teaching effectiveness of English in colleges and universities from the levels of students, teachers and classroom teaching. The particle swarm algorithm is used to optimize the convolutional neural network, the SGD formula is used to improve the calculation accuracy, the Adam optimizer is improved to improve the model operation efficiency, and the optimization algorithm of convolutional neural network, PSO-CNN, is proposed. The PSO-CNN algorithm is introduced into the system and the logic design is carried out to realize the evaluation of the teaching effect of English teaching and to build up the English teaching effect evaluation system based on the intelligent algorithm. Evaluation system based on intelligent algorithm. The performance of the system is examined and analyzed with the help of PCA method, which shows that the cumulative contribution rate of the first six indicators, such as learning acquisition and teaching ability, reaches 91.08%. In the mean square error of model training, the PSO-CNN algorithm applied in this paper’s system has a lower mean square error than other algorithms after 35 iterations, and has better evaluation accuracy. In the application practice of English majors in a higher education institution in B city, after applying the system of this paper to evaluate the effectiveness of English teaching, the English test scores of the regular and experimental classes with improved teaching were significantly improved.